Beyond Multi-Touch: Voice, Gaze, Facial Expression Recognition

Apple's patent applications provide a glimpse into what might lie ahead in our future. For example, we saw an explosion of multi-touch patents in the year before the iPhone's release. And while many of these concepts may never come to market, they provide an interesting look into the direction of Apple's research.

One of the most recent patent applications from Apple is entitled Multitouch Data Fusion and is authored by Wayne Westerman and John Elias (formerly of Fingerworks). Westerman and Elias have been prolific publishers of multi-touch patent applications and likely helped establish the multi-touch technology behind Apple's iPhone. While many have since hoped to see some sort of advanced multi-touch interface for Apple's Macs, only limited multi-touch support has been included into Apple's notebooks.

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)

The patent application gives examples of how each of these could be used in conjunction with multi-touch to provide a better user experience. A few highlights are provided here:

051541 insert 200

Voice - Some tasks are described as being better suited for either voice recognition vs multi-touch. For example, if a user's task is to resize, rotate and change the color of an object on the screen, multi-touch would be best suited for the resize/rotation tasks but changing of color (or inserting text) may be better suited to a voice command.

Finger identification - the article suggests that using built in cameras (such as the iSight) with a swing mirror could provide an over the keyboard view of multi-touch gestures. This video information could be used to better distinguish which fingers are being used in which position. In conjunction with the touch input, this could be used to create more specific and accurate gestures.

053148 gazerec 400

Gaze - tracking where the user is looking could help pick windows or objects on the screen. Rather than resorting to moving a mouse pointer to the proper window, a user could simply direct his gaze at the particular window and then invoke a touch gesture.

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior. The example given is if a user is incorrectly trying to scroll a window using 3 fingers, instead of 2, the computer may be able to detect the frustration and either accept the faulty input or prompt the user.

Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.

Popular Stories

apple card 1

Apple Ending Apple Card Partnership With Goldman Sachs

Tuesday November 28, 2023 3:09 pm PST by
Apple is ending its credit card partnership with Goldman Sachs, according to The Wall Street Journal. Apple plans to stop working with Goldman Sachs in the next 12 to 15 months, and it is not yet clear if Apple has established a new partnership for the Apple Card. Apple and Goldman Sachs will dissolve their entire consumer partnership, including the Apple Card and the Apple Savings account....
iOS 17

Everything New in iOS 17.2 Beta 4

Tuesday November 28, 2023 12:18 pm PST by
Apple is wrapping up development on iOS 17.2, with the update expected to come out in December. While we're getting to the end of the beta testing period, Apple is still tweaking features and adding new functionality. We've rounded up everything new in the fourth beta of iOS 17.2. Default Notification Sound Under Sounds & Haptics, there's a new "Default Alerts" section that allows you to ...
iOS 17

Apple Releases iOS 17.1.2 With Security Fixes

Thursday November 30, 2023 10:12 am PST by
Apple today released iOS 17.1.2 and iPadOS 17.1.2, small updates to the iOS 17 and iPadOS 17 operating systems that Apple introduced in September. iOS 17.1.2 and iPadOS 17.1.2 come a few weeks after the release of iOS 17.1.1, another bug fix update. iOS 17.1.2 and iPadOS 17.1.2 can be downloaded on eligible iPhones and iPads over-the-air by going to Settings > General > Software Update....
Apple 5G Modem Feature Triad

Apple to Discontinue Custom 5G Modem Development, Claim Reports

Wednesday November 29, 2023 4:19 am PST by
Apple is discontinuing in-house modem development after several unsuccessful attempts to perfect its own custom 5G modem chip, according to unconfirmed reports coming out of Asia. According to the operator of news aggregator account "yeux1122" on the Naver blog, supply chain sources related to Apple's 5G modem departments claim that the company's attempts to develop its own modem have...
All New CarPlay Five New Features Article 2

What to Expect From All-New CarPlay, Still Listed as Coming 'Late 2023'

Tuesday November 28, 2023 7:44 am PST by
At WWDC in June 2022, Apple previewed the next generation of CarPlay, promising deeper integration with vehicle functions like A/C and FM radio, support for multiple displays across the dashboard, increased personalization, and more. Apple's website still says the first vehicles with support for the next-generation CarPlay experience will be announced in "late 2023," but it has not shared...
Apple Logo

Apple Discontinued These 5 Products This Year

Monday November 27, 2023 7:03 am PST by
As the end of 2023 nears, now is a good opportunity to look back at some of the devices and accessories that Apple discontinued throughout the year. Apple products discontinued in 2023 include the iPhone 13 mini, 13-inch MacBook Pro, MagSafe Battery Pack, MagSafe Duo Charger, and leather accessories. Also check out our lists of Apple products discontinued in 2022 and 2021. iPhone Mini ...
iOS 17

iOS 17.1.2 Update for iPhone Likely to Be Released This Week

Monday November 27, 2023 8:24 am PST by
Apple will likely release iOS 17.1.2 this week, based on mounting evidence of the software in our website's analytics logs in recent days. As a minor update, iOS 17.1.2 should be focused on bug fixes, but it's unclear exactly which issues might be addressed. Some users have continued to experience Wi-Fi issues on iOS 17.1.1, so perhaps iOS 17.1.2 will include the same fix for Wi-Fi...
iOS 17

28 New Things Your iPhone Can Do in December's iOS 17.2 Update

Friday December 1, 2023 2:57 am PST by
Apple made the first beta of iOS 17.2 available to developers in October. Since then we've seen three more betas, and with each iteration Apple continues to add more new features and changes, many of which users have been anticipating for quite a while. Below, we've listed 28 new things that are coming to your iPhone when the finalized version is publicly released this December. 1. Help...