Beyond Multi-Touch: Voice, Gaze, Facial Expression Recognition

Apple's patent applications provide a glimpse into what might lie ahead in our future. For example, we saw an explosion of multi-touch patents in the year before the iPhone's release. And while many of these concepts may never come to market, they provide an interesting look into the direction of Apple's research.

One of the most recent patent applications from Apple is entitled Multitouch Data Fusion and is authored by Wayne Westerman and John Elias (formerly of Fingerworks). Westerman and Elias have been prolific publishers of multi-touch patent applications and likely helped establish the multi-touch technology behind Apple's iPhone. While many have since hoped to see some sort of advanced multi-touch interface for Apple's Macs, only limited multi-touch support has been included into Apple's notebooks.

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)

The patent application gives examples of how each of these could be used in conjunction with multi-touch to provide a better user experience. A few highlights are provided here:

Voice - Some tasks are described as being better suited for either voice recognition vs multi-touch. For example, if a user's task is to resize, rotate and change the color of an object on the screen, multi-touch would be best suited for the resize/rotation tasks but changing of color (or inserting text) may be better suited to a voice command.

Finger identification - the article suggests that using built in cameras (such as the iSight) with a swing mirror could provide an over the keyboard view of multi-touch gestures. This video information could be used to better distinguish which fingers are being used in which position. In conjunction with the touch input, this could be used to create more specific and accurate gestures.


Gaze - tracking where the user is looking could help pick windows or objects on the screen. Rather than resorting to moving a mouse pointer to the proper window, a user could simply direct his gaze at the particular window and then invoke a touch gesture.

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior. The example given is if a user is incorrectly trying to scroll a window using 3 fingers, instead of 2, the computer may be able to detect the frustration and either accept the faulty input or prompt the user.

Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.

Top Stories

Apple-Acquired Dark Sky Officially Shuts Down Android App

Saturday August 1, 2020 3:43 pm PDT by
Apple in March purchased weather app Dark Sky, and at that time, Dark Sky's developers said that the app's Android version would be discontinued on July 1, 2020. However, instead of shuttering the app on that date, the app's developers announced that the discontinuation would be delayed for another month. Now that it's August, Android users are no longer able to access the app, and...

Apple May Launch This Year's 'iPhone 12' Lineup in Two Stages, With 6.1-inch Models Debuting First

Monday August 3, 2020 3:14 am PDT by
Apple last week confirmed that its "‌iPhone‌ 12" launch will be delayed this year due to the ongoing global health crisis and restrictions on travel. Apple last year started selling iPhones in late September, but this year, Apple projects supply will be "available a few weeks later," suggesting a release sometime in October. We're expecting a total of four OLED iPhones in 5.4, 6.1, and...

Top Stories: Try the 5.4-Inch iPhone 12 Display Size, Blockbuster Earnings, Tim Cook at Antitrust Hearing

Saturday August 1, 2020 6:00 am PDT by
Another busy week of Apple news and rumors has wrapped up, with a lot of focus on Tim Cook's appearance at a Congressional antitrust hearing and a blockbuster earnings report. Subscribe to the MacRumors YouTube channel for more videos. We continued to hear rumors about the upcoming iPhone 12 lineup, including a rare admission from Apple that the lineup will launch "a few weeks later" than...

Just How Small Will the 5.4-Inch iPhone 12 Screen Be? Try It Out for Yourself

Tuesday July 28, 2020 12:57 pm PDT by
As rumors of the iPhone 12 have continued to build over the past few months, the one model that has the most excitement around it is the smallest 5.4" model. The iPhone 12 is believed to be coming in 5.4", 6.7", and 6.1" sizes. Dummy models have shown how much smaller the 5.4" is compared to the rest of the iPhone lineup. The upcoming 5.4" iPhone falls in-between the size of the original...

Unreleased iPod Touch with Mac Pro Glossy Black Finish Shared Online

Sunday August 2, 2020 11:32 am PDT by
Twitter user @DongleBookPro has today posted images of what seems to be a first-generation iPod Touch prototype with a 2013 Mac Pro-style glossy black finish. The Twitter user claims that the iPod Touch prototype pictured has "the same coating as the 2013 Mac Pro." Had the finish been selected for the final product, it also would have been similar to the metallic glossy black finish that...

Apple Confirms This Year's iPhone 12 Models Will Be a Little Bit Late

Thursday July 30, 2020 2:34 pm PDT by
During today's earnings call covering the third fiscal quarter of 2020 (second calendar quarter) Apple CFO Luca Maestri confirmed that Apple is expecting to release this year's iPhones later than usual. Maestri said that Apple last year started selling iPhones in late September, but this year, Apple projects supply will be "available a few weeks later." Multiple rumors have suggested that ...

Battery Likely for Upcoming Apple Watch Series 6 Filed in Certification Listings

Saturday August 1, 2020 5:46 am PDT by
A battery likely for the upcoming Apple Watch Series 6 has been filed at the Korea Testing and Research Institute and discovered by a Twitter user @yabhishekhd. Certification for a 1.17Wh battery with a capacity of 303.8mAh was issued on June 23 by the KTR, a Korean regulatory body that approves and tests new hardware ahead of public sale. The battery seems to be destined for a future...

Apple Watch Series 6 to Feature Blood Oxygen Monitoring Sensor

Friday July 31, 2020 1:56 am PDT by
The Apple Watch Series 6 will add blood oxygen monitoring to its features list when it's launched later this year, according to a new report from DigiTimes. Apple Watch 6 will feature biosensors that can monitor sleeping conditions, detect blood oxygen and measure pulse rates, heartbeats and atrial fibrillation, and will also incorporate MEMS-based accelerometer and gyroscope, all allowing the ...

Apple Marks Return of NHL With New 'Hockey Tape' Ad Shot on iPhone 11 Pro

Saturday August 1, 2020 2:33 am PDT by
Apple today marked the return of NHL hockey with a new "Shot on iPhone" ad on its YouTube channel in Canada. Titled "Hockey Tape," the 30-second video features Vegas Golden Knights players Marc-André Fleury and Mark Stone having some on-ice fun with the iPhone 11 Pro, which they attach to the boards, a hockey stick, and a skate with hockey tape. "See the game like never before with Ultra ...

Emails Reveal Why Steve Jobs and Phil Schiller Blocked In-App Purchase of Kindle Books

Friday July 31, 2020 6:25 am PDT by
Internal Apple emails, made public by the House Judiciary Committee's antitrust inquiry, have revealed information about why Apple blocked in-app purchases of Kindle books on iOS devices, reports The Verge. Two sets of emails between Steve Jobs, Phil Schiller, Eddy Cue, and various other senior Apple executives, disclose the exact thinking behind how Apple approached Kindle on iOS. The...