Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Top Rated Comments

smuzzy Avatar
34 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
34 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
34 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
34 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
34 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
34 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Popular Stories

dewey airtag

Report Highlights Danger of Using AirTags for Tracking Dogs

Monday January 30, 2023 1:45 pm PST by
AirTags may be a convenient way for tracking dogs that might get off leash or otherwise lost, but there are dangers associated with the practice, as outlined by a report from The Wall Street Journal. At 1.26 inches in diameter, AirTags are able to fit easily on a dog's collar, but that size also makes the tracking devices small enough to swallow, at least for a medium to large-sized dog, and ...
General iOS 16 Feature Yellow

Five New iOS Features Coming to Your iPhone Later This Year

Tuesday January 31, 2023 11:58 am PST by
Apple has previously announced several upcoming iOS features that are expected to be added to the iPhone this year. Some of the features could be introduced with iOS 16.4, which should enter beta testing soon, while others will arrive later in the year. Below, we have recapped five new iOS features that are expected to launch in 2023, such as an Apple Pay Later financing option for purchases ...
Apple Silicon Teal Feature

The Next Big Apple Silicon Device May Not Be a Mac or iPad

Wednesday February 1, 2023 3:57 am PST by
Apple's next device with an Apple silicon chip may not be a Mac or an iPad, but rather an advanced external display, according to recent reports. The display, which is rumored to arrive this year, is expected to sit somewhere between the $1,599 Studio Display and the $4,999 Pro Display XDR – but more exact information about the device's positioning and price point is as yet unknown. While ...
Multi Display CarPlay 1

Apple Launching All-New CarPlay Experience Later This Year With These 5 Features

Sunday January 29, 2023 10:15 am PST by
In June 2022, Apple previewed the next generation of CarPlay, promising deeper integration with vehicle functions like A/C and FM radio, support for multiple displays across the dashboard, personalization options, and more. Apple says the first vehicles with support for the next-generation CarPlay experience will be announced in late 2023, with committed automakers including Acura, Audi,...
MKBHD HomePod 2 White Ring Stain

New HomePod Can Still Stain Some Wooden Surfaces

Tuesday January 31, 2023 8:29 am PST by
When the original HomePod launched in 2018, it was discovered that the speaker can leave white rings on some wooden surfaces. Now, well-known YouTuber Marques Brownlee has confirmed that the issue persists to a lesser extent with the new HomePod. In a side-by-side test, he showed that the white second-generation HomePod left a white ring on the wooden surface that he placed the speaker on,...
HomePod 2 White and Midnight Feature Purple Blue

Apple Explains Why HomePod Was Released Again, Wi-Fi 4 Limitation, and More

Thursday February 2, 2023 7:57 am PST by
Apple's VP of hardware engineering Matthew Costello and product marketing employee Alice Chan recently spoke with Men's Journal and TechCrunch about the new second-generation HomePod in wide-ranging interviews about the smart speaker. Apple discontinued the original full-size HomePod in March 2021 after multiple reports indicated that sales of the speaker were lackluster, but Chan told Men's ...