Beyond Multi-Touch: Voice, Gaze, Facial Expression Recognition

Apple's patent applications provide a glimpse into what might lie ahead in our future. For example, we saw an explosion of multi-touch patents in the year before the iPhone's release. And while many of these concepts may never come to market, they provide an interesting look into the direction of Apple's research.

One of the most recent patent applications from Apple is entitled Multitouch Data Fusion and is authored by Wayne Westerman and John Elias (formerly of Fingerworks). Westerman and Elias have been prolific publishers of multi-touch patent applications and likely helped establish the multi-touch technology behind Apple's iPhone. While many have since hoped to see some sort of advanced multi-touch interface for Apple's Macs, only limited multi-touch support has been included into Apple's notebooks.

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)

The patent application gives examples of how each of these could be used in conjunction with multi-touch to provide a better user experience. A few highlights are provided here:

051541 insert 200

Voice - Some tasks are described as being better suited for either voice recognition vs multi-touch. For example, if a user's task is to resize, rotate and change the color of an object on the screen, multi-touch would be best suited for the resize/rotation tasks but changing of color (or inserting text) may be better suited to a voice command.

Finger identification - the article suggests that using built in cameras (such as the iSight) with a swing mirror could provide an over the keyboard view of multi-touch gestures. This video information could be used to better distinguish which fingers are being used in which position. In conjunction with the touch input, this could be used to create more specific and accurate gestures.

053148 gazerec 400

Gaze - tracking where the user is looking could help pick windows or objects on the screen. Rather than resorting to moving a mouse pointer to the proper window, a user could simply direct his gaze at the particular window and then invoke a touch gesture.

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior. The example given is if a user is incorrectly trying to scroll a window using 3 fingers, instead of 2, the computer may be able to detect the frustration and either accept the faulty input or prompt the user.

Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.

Popular Stories

iPhone 17 Pro Blue Feature Tighter Crop

iPhone 17 Pro Launching in Three Months With These 12 New Features

Saturday June 14, 2025 5:45 pm PDT by
The iPhone 17 Pro and iPhone 17 Pro Max are three months away, and there are plenty of rumors about the devices. Below, we recap key changes rumored for the iPhone 17 Pro models as of June 2025:Aluminum frame: iPhone 17 Pro models are rumored to have an aluminum frame, whereas the iPhone 15 Pro and iPhone 16 Pro models have a titanium frame, and the iPhone X through iPhone 14 Pro have a...
apple watch ultra 2 new black

Apple Watch Ultra 3 Finally Coming After Two-Year Hiatus

Monday June 16, 2025 8:45 am PDT by
Apple will finally deliver the Apple Watch Ultra 3 sometime this year, according to analyst Jeff Pu of GF Securities Hong Kong (via @jukanlosreve). The analyst expects both the Apple Watch Series 11 and Apple Watch Ultra 3 to arrive this year (likely alongside the new iPhone 17 lineup, if previous launches are anything to go by), according to his latest product roadmap shared with...
Logitech Logo Feature

Logitech Announces Two New Accessories for WWDC

Friday June 13, 2025 7:22 am PDT by
Alongside WWDC this week, Logitech announced notable new accessories for the iPad and Apple Vision Pro. The Logitech Muse is a spatially-tracked stylus developed for use with the Apple Vision Pro. Introduced during the WWDC 2025 keynote address, Muse is intended to support the next generation of spatial computing workflows enabled by visionOS 26. The device incorporates six degrees of...
iPadOS 26 App Windowing

Apple Explains Why iPads Don't Just Run macOS

Friday June 13, 2025 7:46 am PDT by
iPadOS 26 allows iPads to function much more like Macs, with a new app windowing system, a swipe-down menu bar at the top of the screen, and more. However, Apple has stopped short of allowing iPads to run macOS, and it has now explained why. In an interview this week with Swiss tech journalist Rafael Zeier, Apple's software engineering chief Craig Federighi said that iPadOS 26's new Mac-like ...
iphone 16 pro models 1

17 Reasons to Wait for the iPhone 17

Thursday June 12, 2025 8:58 am PDT by
Apple's iPhone development roadmap runs several years into the future and the company is continually working with suppliers on several successive iPhone models simultaneously, which is why we often get rumored features months ahead of launch. The iPhone 17 series is no different, and we already have a good idea of what to expect from Apple's 2025 smartphone lineup. If you skipped the iPhone...
apple watch ultra snow

6 Features Coming to the Apple Watch Ultra 3

Tuesday February 25, 2025 9:00 am PST by
The Apple Watch Ultra 3 is expected to launch later this year, arriving two years after the previous model with a series of improvements. While no noticeable design changes are expected for the third generation since the company tends to stick with the same Apple Watch design through three generations before changing it, there are a series of internal upgrades on the way. By the time the ...
terminal macos tahoe

Apple's Terminal App Gets Colorful Redesign in macOS Tahoe

Monday June 16, 2025 4:12 am PDT by
Apple's Terminal app is getting a visual refresh in macOS Tahoe, and it's the first notable design update since the command-line tool debuted. The updated Terminal will support 24-bit color and Powerline fonts, according to Apple's State of the Platforms presentation at WWDC25. The app will also adopt the new Liquid Glass aesthetic with redesigned themes that align with macOS 26's broader...