Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.


This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.


Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.


Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Related Roundups: iOS 14, iPadOS 14, macOS 11 Big Sur

Top Rated Comments

(View all)
Avatar
13 weeks ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
Avatar
13 weeks ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
Avatar
13 weeks ago


Seems a lot like what Xbox was able to do with the Kinect 10 years ago

That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Avatar
13 weeks ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
Avatar
13 weeks ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
Avatar
13 weeks ago


The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.

I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Top Stories

iOS 14 Widgets Offer iPhone Users Creative Home Screen Ideas

Sunday September 20, 2020 8:43 pm PDT by
In iOS 14, Apple introduced ‌the concept of Home Screen‌ widgets, which provide information from apps at a glance. Widgets can be pinned to the Home Screen in various spots and sizes, allowing for many different layouts. Despite the relative lack of 3rd party widgets at launch, iOS users around the...

iPhone 12 Lineup Rumored to Be Named 'iPhone 12 mini,' 'iPhone 12,' 'iPhone 12 Pro,' and 'iPhone 12 Pro Max'

Monday September 21, 2020 5:24 am PDT by
Leaker known as "L0vetodream" has today shared the alleged naming for the upcoming iPhone 12 lineup on Twitter. The tweet proposes that the upcoming iPhone 12 models will be titled "iPhone 12 mini," "iPhone 12," "iPhone 12 Pro," and "iPhone 12 Pro Max." The names likely correspond to the three expected sizes of iPhone 12, with the 5.4-inch model being the iPhone 12 mini, the 6.7-inch model ...

Hands-On With the New Apple Watch Series 6 and Apple Watch SE

Friday September 18, 2020 1:19 pm PDT by
Today's the official launch date for the Apple Watch Series 6 and the Apple Watch SE, both of which Apple announced on Tuesday. We picked up a couple of the new models and thought we'd give them a quick look for MacRumors readers thinking of ordering a new watch. Apple Watch Series 6 & Apple Watch SE Hands-On! When it comes to design, both the $399 Series 6 and the $279 SE look just like...

iOS 14 Picture in Picture No Longer Working With YouTube's Mobile Website in Safari [Without Premium]

Friday September 18, 2020 12:21 pm PDT by
Apple in iOS 14 added Picture in Picture to the iPhone, a feature designed to let you watch a video in a small screen on your device while you continue to do other things on the phone. When Picture in Picture was working with YouTube The YouTube app doesn't support Picture in Picture, but up until yesterday there was a functional workaround that allowed videos from YouTube.com to be watched...

When Will the iPhone 12 Launch? Here's What We Know

Wednesday September 16, 2020 6:12 am PDT by
Yesterday's "Time Flies" Apple event saw the release of the Apple Watch Series 6, Apple Watch SE, iPad 8, and iPad Air 4, but no new iPhone models. Rumors before the event strongly alleged that it would not see the unveiling of new iPhones, with many reports pointing to an October launch. The lack of new iPhone models yesterday seems to confirm that the iPhone 12 lineup will not appear...

Here's How You Can Download iOS 14 and iPadOS 14 Around the World [It's Out]

Wednesday September 16, 2020 2:36 am PDT by
Apple's official public release of iOS 14 and iPadOS 14 dropped on Wednesday, September 16, just a day after the company released the Golden Master to third-party developers. Also set to be made available to the general public for the first time are watchOS 7 and tvOS 14. Getting Started With iOS 14 Video Click image to watch iOS 14 Getting Started While that's left a lot of developers...

AirPods Studio Rumored to Come With U1 Chip, Ultra-Wideband Said to Be Vital to Future Apple Ecosystem

Sunday September 20, 2020 6:17 am PDT by
Proven leaker known as "L0vetodream" has today shared a range of information about the ultra-wideband U1 chip in Apple's upcoming AirTags item trackers and AirPods Studio headphones. The first of a series of tweets shared today simply stated that AirPods Studio will contain an ultra-wideband U1 chip. It seems likely that the U1 chip would be used in AirPods Studio to track the location of...

Kuo: Apple to Accelerate Adoption of Mini-LED Displays in iPad and Mac Notebook Lineups

Sunday September 20, 2020 10:00 pm PDT by
Increased competition among Apple's suppliers for mini-LED display chips will accelerate the company's adoption of the advanced technology in its iPad and MacBook lineups, according to a new research note from analyst Ming-Chi Kuo seen by MacRumors. Kuo says that while Epistar had been predicted to be the exclusive supplier of mini-LED chips for Apple products in 2021, Sanan Optoelectronics...

Top Stories: Apple Event Recap, Apple Watch Series 6, Redesigned iPad Air, and More

Saturday September 19, 2020 6:00 am PDT by
This week's news was obviously dominated by Apple's media event and the launch of iOS 14, but there was a lot to digest, so check out our summary below for the high-level view of the past week. With the exception of the massively redesigned iPad Air, all of the new hardware introduced this week is starting to appear on store shelves and on customers' doorsteps, while all of the new software...

Apple Updates AirPods 2 and AirPods Pro Firmware to Version 3A283

Monday September 14, 2020 11:24 am PDT by
Apple today released new 3A283 firmware updates for the second-generation AirPods and the AirPods Pro. The second-generation AirPods are being updated from the 2D15 firmware they were previously running, while the AirPods Pros are being updated from the 2D27 firmware they had installed previously. Apple does not provide details on what's included in refreshed firmware so we don't know what's ...