Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Related Roundup: WWDC 2022

Top Rated Comments

smuzzy Avatar
26 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
26 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
26 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
26 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
26 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
26 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Popular Stories

2022 back to school apple

Apple Launches 2022 Back to School Offer: Up to $150 Gift Card With Mac or iPad

Friday June 24, 2022 5:08 am PDT by
Apple today launched its annual "Back to School" promotion for college/university students in the United States and Canada. This year's promotion offers a free Apple gift card with the purchase of an eligible Mac or iPad, rather than free AirPods like last year. Apple is also offering students 20% off AppleCare+ plans during the promotion. Apple is offering a $150 gift card with the purchase ...
apple ar headset concept 2

Apple CEO Tim Cook Hints at AR/VR Headset: 'Stay Tuned and You'll See What We Have to Offer'

Wednesday June 22, 2022 6:34 pm PDT by
CEO Tim Cook this week did an interview with China Daily, where he once again commented on on the future of augmented reality and hinted at Apple's work on an AR/VR headset. Render via designer Ian Zelbo Cook said that Apple is excited about the opportunities available with augmented reality, which is not too far off from prior comments that he's made, but he went on to say that people should ...
airpods pro 2 1

AirPods Pro 2 Said to Feature Upgraded H1 Chip, Find My, Heart Rate Detection, USB-C and More

Friday June 24, 2022 9:48 am PDT by
The next-generation AirPods Pro could come with a long list of new features that include heart rate detection, the ability to function as a hearing aid, and a USB-C port according to a report from 52Audio. The site claims that it has received new information on the AirPods Pro 2, and it has used that information to provide some renders on what the earbuds might look like. Design wise, there...
customize wallpaper setting ios 16

iOS 16 Beta 2 Tidbits: New Wallpaper Colors, Backup Over LTE, SMS Filtering and More

Wednesday June 22, 2022 2:16 pm PDT by
Apple today seeded the second betas of iOS 16 and iPadOS 16 to developers for testing purposes, and the new betas introduce new features and refine some of the changes that Apple made with the first iOS 16 release. Subscribe to the MacRumors YouTube channel for more videos. Lock Screen Photo Wallpaper Customization When customizing a photo on the Lock Screen, there are two new DuoTone and...
home app ios 16

Apple Confirms iPad Will No Longer Be Supported as a Home Hub in iOS 16

Wednesday June 22, 2022 12:38 pm PDT by
The iPad will no longer be able to be used as a home hub following the launch of iOS 16, iPadOS 16, macOS Ventura, and the HomePod 16 software this fall, Apple confirmed today. As discovered in iOS 16 code by MacRumors contributor Steve Moser, Apple says that the iPad will no longer be supported as a home hub. This information will be displayed in the Home app after updating to iOS 16.A home ...
apple ar headset concept 1

Apple Rumored to Announce 'Game-Changer' AR/VR Headset in January 2023

Friday June 24, 2022 2:52 am PDT by
Apple is "likely" to announce its long-rumored mixed-reality headset as soon as January 2023, Apple analyst Ming-Chi Kuo has reiterated. Concept render based on purported leaked information by Ian Zelbo In a detailed post on Medium, Kuo explained that Apple's headset will be a "game-changer" for the augmented-reality and virtual-reality market. Describing some of the headset's...