Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Popular Stories

ios 17 iphone 15 pro status bar sos crop feature2

iPhone SOS: Verizon Experiences Major Outage Across the U.S. [Update: Fixed]

Monday September 30, 2024 9:03 am PDT by
Verizon is currently experiencing a major outage that is affecting many customers across the U.S., including iPhone users. Affected users may be unable to send or receive phone calls, send or receive text messages, or use cellular data. As a result of the network being down, many affected iPhone users are seeing "SOS" displayed in their device's status bar. In a support document, Apple says...
15 New Things Your iPhone Can Do in iOS 18

15 New Things Your iPhone Can Do in iOS 18.1

Friday September 27, 2024 6:14 am PDT by
Apple is set to release iOS 18.1 in October, bringing the first set of Apple Intelligence features to iPhone 15 Pro and iPhone 16 models. This update marks a significant step forward in Apple's AI integration, offering a new Siri contextually-aware experience and a range of additional capabilities powered by on-device machine learning and large language models. There are a couple of handy new...
m3 mbp space black

What to Expect From an Apple Event in October: iPad Mini 7, Redesigned Mac Mini, and More

Friday September 27, 2024 11:47 am PDT by
Apple will likely hold another event in October this year to announce new Macs and iPads. If so, it would be the fourth time in the last five years that Apple has held an event in October. Last year, Apple held a virtual event on Monday, October 30 to announce new MacBook Pro and iMac models with the M3 series of chips. Subscribe to the MacRumors YouTube channel for more videos. Below, we...
airpods pro 2 gradient

AirPods Pro 3 Expected Next Year: Here's What We Know

Tuesday October 1, 2024 5:47 am PDT by
Despite being released over two years ago, Apple's AirPods Pro 2 continue to dominate the wireless earbud market. However, with the AirPods Pro 3 expected to launch sometime in 2025, anyone thinking of buying Apple's premium earbuds may be wondering if the next generation is worth holding out for. Apart from their audio and noise-canceling performance, which are generally regarded as...
iphone 16 pro colors 1

iPhone 16 Pro Max Charging Speed Test Proves 45W Rumor Was Wrong

Monday September 30, 2024 8:16 am PDT by
While a Chinese regulatory filing showed that all iPhone 16 models are rated for up to 45W charging speeds, tests have since shown that the devices do not actually charge this fast. However, there are still improvements. ChargerLAB last week tested the iPhone 16 Pro Max with a variety of Apple and third-party chargers, and it found that the device achieved maximum sustained charging speeds...
iPhone SE 4 Thumb 2

Apple's Next New iPhone to Debut in the Spring: What to Expect

Tuesday October 1, 2024 3:14 am PDT by
Apple's budget-friendly iPhone SE is set for a major overhaul with a fourth generation model expected to launch in spring 2025. The upcoming model will mark a significant departure from its predecessors, adopting several features from higher-end iPhones while maintaining its position as the most affordable new model in Apple's lineup. According to recent reports, the iPhone SE 4 will sport a ...
apple silicon mac lineup wwdc 2022 feature purple

MacBook Pro, iMac, and Redesigned Mac Mini With M4 Chips on Track to Launch 'This Year'

Tuesday October 1, 2024 1:57 pm PDT by
Apple plans to release new MacBook Pro, iMac, and Mac mini models with the M4 series of chips "this year," according to Bloomberg's Mark Gurman. Gurman initially said these Macs would likely be announced during a virtual event this October, but he has been more vague about the timing lately, with wording such as "in the coming weeks" and now merely "this year." In any case, it is clear that...
iPad iOS 16 WP Display Feature eric edit

Apple May Launch First iPad-Like Smart Home Accessory Next Year

Monday September 30, 2024 2:55 am PDT by
Apple could release an iPad-like smart home accessory based on its homeOS platform as early as next year, according to Bloomberg's Mark Gurman. Writing in his latest Power On newsletter, Gurman reports that the display will run Apple apps like Calendar, Notes, and Home, and will feature an interface "optimized for controlling home appliances and quickly seeing information." Apple's...

Top Rated Comments

smuzzy Avatar
56 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
56 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
56 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
56 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
56 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
56 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)