Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Top Rated Comments

smuzzy Avatar
12 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
12 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
12 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
12 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
12 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
12 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Top Stories

tracking disabled ios 14 5

Analytics Suggest 96% of Users Leave App Tracking Disabled in iOS 14.5

Friday May 7, 2021 1:51 am PDT by
An early look at an ongoing analysis of Apple's App Tracking Transparency suggests that the vast majority of iPhone users are leaving app tracking disabled since the feature went live on April 26 with the release of iOS 14.5. According to the latest data from analytics firm Flurry, just 4% of iPhone users in the U.S. have actively chosen to opt into app tracking after updating their device...
macbook colors 3d black bezels

Prosser: Next MacBook Air Could Come in Colors Similar to iMac

Friday May 7, 2021 6:55 am PDT by
According to Apple leaker Jon Prosser, Apple's upcoming release of the MacBook Air will feature various colors, similar to the colors in the newly released 24-inch iMac. In the latest video of his YouTube channel Front Page Tech, Prosser says the same source who accurately provided him information on the first Apple silicon iMac coming in colors has told him that he recently saw a prototype...
snapchat dark mode

Snapchat Rolls Out Dark Mode on iOS

Wednesday May 5, 2021 1:17 am PDT by
Nearly two years following the release of iOS and iPadOS 13, which included native, built-in, and systemwide dark mode, Snapchat, one of the world's most prominent social media networks, has finally rolled out a dark mode theme for iOS users. Snapchat began testing a dark mode theme of its app design late last year with a small group of iOS users. Now, Snapchat says that as of this week, it...
tile amazon sidewalk integration

Tile to Leverage Amazon Echo and Ring Devices to Better Compete With AirTags

Friday May 7, 2021 2:07 pm PDT by
Amazon today announced that it is teaming up with Tile to add Amazon Sidewalk integration to Tile's Bluetooth trackers. Amazon Sidewalk, for those unfamiliar, is a network of Amazon Bluetooth devices that's designed to improve the connectivity of devices like the Ring and Amazon Echo. Tile will now be joining Amazon Sidewalk, and through this integration, Amazon Echo and Ring devices will be ...
tile sticker e1570533758981

Tile CEO: 'We Welcome Competition From Apple, But We Think It Needs to Be Fair'

Tuesday May 4, 2021 9:51 am PDT by
Just after Apple announced its AirTags, Tile CEO CJ Prober relayed his concerns about competing with Apple in the tracking space, and said that Tile would ask Congress to investigate Apple's business practices specific to Find My and item trackers. Prober this week did an interview with Bloomberg, where he further expanded on Tile's complaints about Apple and why he feels that Tile is...
airtag 1

AirTag Anti-Stalking Measures 'Just Aren't Sufficient' Says Washington Post Report

Wednesday May 5, 2021 6:03 pm PDT by
The safeguards that Apple built into AirTags to prevent them from being used to track someone "just aren't sufficient," The Washington Post's Geoffrey Fowler said today in a report investigating how AirTags can be used for covert stalking. Fowler planted an AirTag on himself and teamed up with a colleague to be pretend stalked, and he came to the conclusion that the AirTags are a "new means...
signal instagram ads3

Signal Shares the Instagram Ads Facebook Doesn't Want You to See

Wednesday May 5, 2021 1:29 am PDT by
Encrypted messaging app Signal has had a series of Instagram ads blocked from the social media platform, after it attempted to show users how much data the Facebook-owned company collects about them and how it's used to push targeted ads. In a blog post, Signal described how it generated the ads to show users why they were seeing them, simply by declaring upfront the information that the...
fortnite apple logo 2

Epic CEO Tim Sweeney Admits App Store's 30% Cut Is Similar to Consoles, Would Have Accepted Special Deal With Apple

Tuesday May 4, 2021 1:54 pm PDT by
Apple's legal battle with Epic Games is continuing on, and during the second day of the trial, Epic Games' CEO Tim Sweeney continued his testimony against Apple. Sweeney was grilled by Apple's lawyers, and made several points seemingly favorable to Apple. In addition to mentioning how he prefers Apple's iPhone and values Apple's privacy policies that he's aiming to dismantle, Sweeney...
iphone 12 preorder purple

Apple Begins Transition to Randomized Serial Numbers With Purple iPhone 12

Wednesday May 5, 2021 9:17 am PDT by
MacRumors previously reported about Apple's plan to switch to randomized serial numbers for future products starting in early 2021, and this transition has now started with the new purple iPhone 12 model in multiple countries. With assistance from Aaron Zollo, host of the YouTube channel ZolloTech, we can confirm that the purple iPhone 12 released last month has a new 10-character serial...
precision finding developer mode

AirTag Precision Finding Interface Includes Hidden 'Developer Mode'

Thursday May 6, 2021 1:32 am PDT by
A frustrated AirTag owner has inadvertently discovered the existence of a hidden "developer mode" in the on-screen interface that Find My displays when the Precision Finding feature is activated to help locate one of Apple's item trackers. Precision Finding is a feature that provides users with specific on-screen directions for finding a nearby AirTag. iPhones with a U1 chip, which includes ...