Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Related Roundup: WWDC 2024

Top Rated Comments

smuzzy Avatar
49 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
49 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
49 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
49 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
49 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
49 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Popular Stories

iPhone 16 Camera Lozenge 2 Perspective Gray

Five Key Upgrades Coming to iPhone 16

Friday March 15, 2024 1:45 pm PDT by
The iPhone is Apple's top-selling product, and it gets an update every year. In 2024, we're expecting the iPhone 16 and iPhone 16 Pro lineup, with an arguably more interesting feature set than we got with the iPhone 15 and iPhone 15 Pro. Subscribe to the MacRumors YouTube channel for more videos. Capture Button All four iPhone 16 models are set to get a whole new button, which will be...
When To Expect New iPads Feature 1

Apple to Announce New iPads on March 26, Rumors Claim

Monday March 18, 2024 4:02 am PDT by
Apple is widely expected to release new iPad Air and OLED iPad Pro models in the next few weeks. According to new rumors coming out of Asia, the company will announce its new iPads on Tuesday, March 26. Chinese leaker Instant Digital on Weibo this morning 日发布%23">claimed that the date will see some sort of announcement from Apple related to new iPads, but stopped short of calling it an...
airpods 3 orange

Two New AirPods 4 Models Expected to Launch in September or October

Sunday March 17, 2024 7:56 am PDT by
Apple suppliers will begin production of two new fourth-generation AirPods models in May, according to Bloomberg's Mark Gurman. Based on this production timeframe, he expects the headphones to be released in September or October. Gurman expects both fourth-generation AirPods models to feature a new design with better fit, improved sound quality, and an updated charging case with a USB-C...
iphone se 4 modified flag edges

iPhone SE 4 Expected to Depreciate Heavily

Tuesday March 12, 2024 9:04 am PDT by
Resale value trends suggest the iPhone SE 4 may not hold its value as well as Apple's flagship models, according to SellCell. According to the report, Apple's iPhone SE models have historically depreciated much more rapidly than the company's more premium offerings. The third-generation iPhone SE, which launched in March 2022, experienced a significant drop in resale value, losing 42.6%...
General iOS 17 Feature Orange Purple

iOS 17.4.1 Update for iPhone is Imminent

Monday March 18, 2024 5:27 am PDT by
iOS 17.4.1 and iPadOS 17.4.1 should be released within the next few days, with a build number of 21E235, according to a source with a proven track record. MacRumors previously reported that Apple was internally testing iOS 17.4.1. As a minor update for the iPhone, it will likely address software bugs and/or security vulnerabilities. It is unclear if the update will include any other changes. ...
iOS 17 Passkey With Apple ACCOUNT Feature

'Apple ID' Expected to Change to 'Apple Account' Starting With iOS 18

Sunday March 17, 2024 7:13 am PDT by
MacRumors was first to report that Apple was planning to rebrand "Apple ID" to "Apple Account" across its software platforms and websites like iCloud.com as early as this year, and now Bloomberg's Mark Gurman has corroborated this change. A mockup of the new Apple Account branding In his Power On newsletter today, Gurman said the new "Apple Account" branding will start to be used later this...
M3 iPad Feature 3

New iPads Likely to Begin Shipping in April

Monday March 18, 2024 9:52 am PDT by
Apple's new iPad Pro models with OLED displays will likely begin shipping to customers in April, according to information shared today by Ross Young, CEO of display industry research firm Display Supply Chain Consultants. Bloomberg's Mark Gurman also said the new iPad Pro models might not ship until "deeper" into April in his Power On newsletter on Sunday:I've repeatedly said that new...