Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Related Roundups: iOS 14, iPadOS 14, macOS Big Sur

Top Rated Comments

smuzzy Avatar
16 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
16 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
16 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
16 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
16 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
16 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Top Stories

m1x mac mini screen feature

High-End 'M1X' Mac Mini With New Design and Additional Ports Expected to Launch in the 'Next Several Months'

Sunday August 22, 2021 5:59 am PDT by
Apple can be expected to launch an updated high-end Mac mini with a new design and a faster "M1X" Apple silicon processor in the "next several months," Bloomberg's Mark Gurman reports. In the latest publication of his Power On newsletter, Gurman writes that a new high-end Mac mini, which has previously been reported to feature a new design with additional ports, can be expected to replace...
mac scanner permission error

Apple Says Fix Planned for 'You Do Not Have Permission to Open the Application' Error When Using a Scanner on Mac

Saturday August 14, 2021 6:15 am PDT by
In a newly published support document on its website, Apple has acknowledged an error that some users may receive when they try to use a scanner with a Mac in the Image Capture app, Preview app, or the Printers & Scanners section of System Preferences. A screenshot of the error message from the HP Support Community When attempting to use a scanner with a Mac, Apple said users might get an...
macbookpro13large

macOS Big Sur Update Bricking Some Older MacBook Pro Models

Sunday November 15, 2020 5:33 am PST by
A large number of late 2013 and mid 2014 13-inch MacBook Pro owners are reporting that the macOS Big Sur update is bricking their machines. A MacRumors forum thread contains a significant number of users reporting the issue, and similar problems are being reported across Reddit and the Apple Support Communities, suggesting the problem is widespread. Users are reporting that during the...
original iphone

Phil Schiller Says iPhone Was 'Earth-Shattering' Ten Years Ago and Remains 'Unmatched' Today

Monday January 9, 2017 7:15 am PST by
To commemorate the tenth anniversary of the iPhone, Apple marketing chief Phil Schiller sat down with tech journalist Steven Levy for a wide-ranging interview about the smartphone's past, present, and future. The report first reflects upon the iPhone's lack of support for third-party apps in its first year. The argument inside Apple was split between whether the iPhone should be a closed...
m1 imac orange

New iMac Tidbits: Headphone Jack on Side, Ethernet Port on Power Adapter, Spatial Audio and WiFi 6 Support, No SD Card Slot

Wednesday April 21, 2021 6:38 am PDT by
Apple yesterday announced a completely redesigned 24-inch iMac with the M1 Apple silicon chip. The new iMac, the first major redesign of the Mac desktop computer since 2012, has several changes compared to the previous generation. In the aftermath of the event, a few new features and tidbits may have slipped under the radar, so we’ve compiled this list of some of the less-talked-about...
omg lightning cable comparison

Security Researcher Develops Lightning Cable With Hidden Chip to Steal Passwords

Thursday September 2, 2021 6:59 am PDT by
A normal-looking Lightning cable that can used to steal data like passwords and send it to a hacker has been developed, Vice reports. The "OMG Cable" compared to Apple's Lightning to USB cable. The "OMG Cable" works exactly like a normal Lightning to USB cable and can log keystrokes from connected Mac keyboards, iPads, and iPhones, and then send this data to a bad actor who could be over a...
Top Stories 75 Thumbnail

Top Stories: Last-Minute iPhone 13 Rumors, Apple Announces App Store Changes, and More

Saturday September 4, 2021 6:00 am PDT by
The finish line is in sight! Apple's annual iPhone event is likely just a week or so away and all eyes will be on the company as it unveils the next version of its most popular product line. With any luck, we'll also see the next-generation Apple Watch and perhaps even some new AirPods. Other news this week saw Apple making some more changes to its App Store policies in response to a...
General YouTube Feature 1

YouTube Premium and Music Surpass 50 Million Subscribers

Friday September 3, 2021 2:19 am PDT by
YouTube says it has passed 50 million subscribers for its Premium and Music subscriptions, making it the "fastest growing music subscription" service in the world, according to YouTube's global head of music, Lyor Cohen. YouTube says that it has more than 50 million paying subscribers collectively across YouTube Premium and YouTube Music. The Google-owned service says it attributes this...
iPhone 13 Dummy Thumbnail 2

Full iPhone 13 Feature Breakdown: Everything Rumors Say We Can Expect

Tuesday August 31, 2021 7:50 am PDT by
With the launch of Apple's iPhone 13 lineup believed to be just a few weeks away, we have compiled all of the coherent rumors from our coverage over the past year to build a full picture of the features and upgrades coming to the company's new smartphones. For clarity, only explicit improvements, upgrades, and new features compared to the iPhone 12 lineup are listed. It is worth noting that...
apple wallet drivers license

Apple Announces First U.S. States That Will Let You Add Your Driver's License to Your iPhone

Wednesday September 1, 2021 6:15 am PDT by
Apple today announced the first U.S. states that will be rolling out the ability for residents to add their driver's license or state ID to the Wallet app on the iPhone and Apple Watch. Arizona and Georgia will be the first states to support the feature, with Connecticut, Iowa, Kentucky, Maryland, Oklahoma, and Utah to follow, according to Apple. Apple said select TSA security checkpoints in ...