Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Related Roundups: iOS 14, iPadOS 14, macOS Big Sur

Top Rated Comments

smuzzy Avatar
22 weeks ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
22 weeks ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
22 weeks ago


Seems a lot like what Xbox was able to do with the Kinect 10 years ago

That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
22 weeks ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
22 weeks ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
22 weeks ago


The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.

I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Top Stories

Apple Watc black friday 20 sale feature

Apple Black Friday 2020: Best Apple Watch Deals [Updated]

Wednesday November 25, 2020 4:01 pm PST by
Black Friday sales have begun on a variety of products, including the Apple Watch. There are quite a few deals across the Apple Watch lineup this year, including one of the lowest price we've ever seen the Apple Watch Series 3. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the...
AirPods Pro black friday 20 sale feature 2

Black Friday 2020: AirPods Pro Reach Lowest Price Ever [Updated]

Wednesday November 25, 2020 3:22 pm PST by
Black Friday has kicked off this week, and one of the first major sales for the AirPods Pro is available right now on Walmart. You can find this deal below, along with a few other solid discounts on the regular AirPods. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site...
windows 10

Developer Successfully Virtualizes Windows for Arm on M1 Mac

Friday November 27, 2020 7:16 am PST by
Developer Alexander Graf has successfully virtualized the Arm version of Windows on an M1 Mac, proving that the M1 chip is capable of running Microsoft's operating system (via The 8-Bit). Currently, Macs with the M1 chip do not support Windows and there is no Boot Camp feature as there is on Intel Macs, but support for Windows is a feature that many users would like to see. Using the...
General black friday 20 sale feature

Thanksgiving Day Deals Still Available on AirPods, M1 Macs, Apple Watch Series 6, iPads

Thursday November 26, 2020 10:21 am PST by
Black Friday deals seem to start earlier and earlier every year, so there were already a wide variety of discounts available for Apple products on Thanksgiving Day. Many of the deals remain available even after Thanksgiving, but act fast, as inventory quickly fluctuates. Thanksgiving Day Deals on Apple Products — Still Available:AirPods with a wireless charging case remain available for...
iPhone black friday 20 sale feature

Apple Black Friday 2020: Best iPhone Deals

Friday November 27, 2020 12:56 pm PST by
Black Friday is halfway done, but there are still a few deals to shop for on iPhones at carriers like AT&T, Verizon, and T-Mobile/Sprint. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. AT&T Starting with AT&T, you'll find up to $700 off any iPhone 12 when...
13 16 inch macbook pro air trio

Reliable Leaker Suggests Redesigned MacBooks in 2021 Will Include Both Apple Silicon and Intel Models

Wednesday November 25, 2020 9:15 am PST by
Reliable leaker known as "L0vetodream" has today suggested on Twitter that redesigned MacBooks coming in the second half of 2021 will include models with both Apple Silicon chips and Intel processors. The brief Tweet came in response to a MacRumors article from earlier today, which outlined a report from Ming-Chi Kuo claiming that Apple plans to release redesigned MacBook models with Apple ...
iphone trade in store

UK Environmental Committee Says Apple Contributing to 'Throwaway Culture' of 'Short-Lived Products'

Thursday November 26, 2020 7:07 am PST by
Technology companies like Apple are contributing to e-waste by making their products difficult to repair, and charging expensive repair fees, according to a lengthy report published today by the UK Parliament's Environmental Audit Committee. "We were told that Apple glues and solders parts together on their laptops, which makes repairing them very difficult," the Committee wrote in a summary ...
m1 chip macbook air pro

Kuo: Redesigned MacBooks With Apple Silicon to Launch in Second Half of 2021

Tuesday November 24, 2020 7:53 pm PST by
Apple plans to release additional MacBook models with Apple Silicon in the second half of 2021, according to analyst Ming-Chi Kuo, as part of the company's two-year transition away from Intel processors across its Mac lineup. In a research note today, obtained by MacRumors, Kuo said that these MacBook models will feature a new design. Kuo did not specify which models these will be, but he...
mac mini macbook pro macbook air

Apple M1 Hands-On Comparison: MacBook Air vs. MacBook Pro vs. Mac Mini

Monday November 23, 2020 3:40 pm PST by
Apple's M1 Macs are out in the wild now, but ahead of the holidays, you might still be trying to figure out which one to pick up, either for yourself or as a gift for someone else. We've got all three of the new Macs available, so we thought we'd give MacRumors readers a hands-on overview of each machine in our latest YouTube video. Subscribe to the MacRumors YouTube channel for more videos. ...
iPad Pro 5G and Mini LED feature

Rumored 2021 High-End iPad Pro May Feature 5G With mmWave Support

Thursday November 26, 2020 2:14 am PST by
Apple's rumored high-end iPad Pro models to be released next year will be 5G-enabled with mmWave support, according to sources cited by industry publication DigiTimes. Multiple rumors have suggested Apple is planning to release a high-end 12.9-inch iPad Pro with a mini-LED display next year, with the possibility that there will be an 11-inch mini-LED model too, but details beyond that have...