Beyond Multi-Touch: Voice, Gaze, Facial Expression Recognition

Apple's patent applications provide a glimpse into what might lie ahead in our future. For example, we saw an explosion of multi-touch patents in the year before the iPhone's release. And while many of these concepts may never come to market, they provide an interesting look into the direction of Apple's research.

One of the most recent patent applications from Apple is entitled Multitouch Data Fusion and is authored by Wayne Westerman and John Elias (formerly of Fingerworks). Westerman and Elias have been prolific publishers of multi-touch patent applications and likely helped establish the multi-touch technology behind Apple's iPhone. While many have since hoped to see some sort of advanced multi-touch interface for Apple's Macs, only limited multi-touch support has been included into Apple's notebooks.

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)

The patent application gives examples of how each of these could be used in conjunction with multi-touch to provide a better user experience. A few highlights are provided here:

051541 insert 200

Voice - Some tasks are described as being better suited for either voice recognition vs multi-touch. For example, if a user's task is to resize, rotate and change the color of an object on the screen, multi-touch would be best suited for the resize/rotation tasks but changing of color (or inserting text) may be better suited to a voice command.

Finger identification - the article suggests that using built in cameras (such as the iSight) with a swing mirror could provide an over the keyboard view of multi-touch gestures. This video information could be used to better distinguish which fingers are being used in which position. In conjunction with the touch input, this could be used to create more specific and accurate gestures.

053148 gazerec 400

Gaze - tracking where the user is looking could help pick windows or objects on the screen. Rather than resorting to moving a mouse pointer to the proper window, a user could simply direct his gaze at the particular window and then invoke a touch gesture.

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior. The example given is if a user is incorrectly trying to scroll a window using 3 fingers, instead of 2, the computer may be able to detect the frustration and either accept the faulty input or prompt the user.

Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.

Popular Stories

iOS 18

iOS 18.4 Coming Next Week With These New Features for Your iPhone

Friday February 14, 2025 6:18 am PST by
The first iOS 18.4 beta for iPhones should be just around the corner, and the update is expected to include many new features and changes. Bloomberg's Mark Gurman expects the iOS 18.4 beta to be released by next week. Below, we outline what to expect from iOS 18.4 so far. Apple Intelligence for Siri Siri is expected to get several enhancements powered by Apple Intelligence on iOS...
apple launch feb 2025 alt

What to Expect From the 'Apple Launch' Next Week

Thursday February 13, 2025 11:48 am PST by
Apple has yet to announce any new devices this year, but that could change starting next week. Apple CEO Tim Cook today said to "get ready" for a "launch" on Wednesday, February 19. "Get ready to meet the newest member of the family," said Cook, in a social media post. The post includes an #AppleLaunch hashtag, along with a short video featuring an animated Apple logo inside of a circle....
apple launch feb 2025

Tim Cook Teases an 'Apple Launch' Next Wednesday

Thursday February 13, 2025 8:07 am PST by
In a social media post today, Apple CEO Tim Cook teased an upcoming "launch" of some kind scheduled for Wednesday, February 19. "Get ready to meet the newest member of the family," he said, with an #AppleLaunch hashtag. The post includes a short video with an animated Apple logo inside a circle. Cook did not provide an exact time for the launch, or share any other specific details, so...
iPhone 17 Pro Render Front Page Tech

iPhone 17 Pro With All-New Camera Bar Design Allegedly Revealed

Thursday February 13, 2025 5:49 pm PST by
Apple's next-generation iPhone 17 Pro will feature three rear cameras arranged in a familiar triangular layout, but the cameras will be housed in an all-new rectangular camera bar with rounded corners, according to YouTube channel Front Page Tech. iPhone 17 Pro camera design render created by Asher for Front Page Tech In a video uploaded today, Front Page Tech host Jon Prosser said the camera ...
iPhone SE 4 Thumb 1

'New' iPhone SE Product Listing Appears on French Website

Wednesday February 12, 2025 6:49 am PST by
As the wait continues for Apple's long-rumored, fourth-generation iPhone SE, French electronics retailer Boulanger has prematurely published a product listing for a "new" model of the iPhone SE. The placeholder page says the device is "coming soon," but it offers no further information, and the price shown is obviously not real. The listing was spotted by a reader of the French technology...
M4 Mac mini Apple Video

Apple's Refurbished Mac Mini Pricing Has a Problem

Thursday February 13, 2025 6:20 am PST by
Apple this week began selling refurbished Mac mini models with the M4 chip for the first time, but this has led to a pricing conundrum. In the United States, Apple is offering a refurbished Mac mini with the base M4 chip, 256GB of storage, 16GB of RAM, and Gigabit Ethernet for $509, down from $599 new. This is the standard 15% discount that Apple offers on refurbished Macs. The issue is...
iPhone 16 Apple Store

iPhone 17 in New Sizes This Year: What to Know

Thursday February 13, 2025 2:45 am PST by
Last year, Apple tweaked iPhone 16 Pro screen sizes to make them bigger than 2023's iPhone 15 Pro models, and this year we are also expecting a change in the size of the displays in the iPhone 17 lineup. Here's what we know. Standard iPhone 17 Apple could introduce a new display size for the standard iPhone 17 model in 2025. The iPhone 17 could measure in at 6.3 inches, up from 6.1 inches,...
iphone air concept weis

iPhone 17 Air Could Look Like This in Real Life

Friday February 14, 2025 3:41 am PST by
There have been several alleged leaked details of the iPhone 17 Air, Apple's rumored new slim iPhone, but images have been limited to grainy shots taken in component factories. However, this hyper-realistic concept created by WEIS Studio gives us the best idea yet of what Apple's thin device might actually look like. The concept design is inspired by recent leaks indicating that the device...