Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Popular Stories

New Things Your iPhone Can Do in iOS 18

18 New Things Your iPhone Can Do in iOS 18.2

Wednesday November 27, 2024 5:05 am PST by
Apple is set to release iOS 18.2 in early December, bringing the second round of Apple Intelligence features to iPhone 15 Pro and iPhone 16 models. This update brings several major advancements to Apple's AI integration, including completely new image generation tools and a range of Visual Intelligence-based enhancements. There are a handful of new non-AI related feature controls incoming as...
iphone 16 pro models 1

12 Reasons to Wait for Next Year's iPhone 17

Friday November 29, 2024 5:17 am PST by
Apple's iPhone development roadmap runs several years into the future and the company is continually working with suppliers on several successive iPhone models simultaneously, which is why we sometimes get rumored feature leaks so far ahead of launch. The iPhone 17 series is no different – already we have some idea of what to expect from Apple's 2025 smartphone lineup. If you plan to skip...
iPhone 17 Pro Dual Tone Rectangle Feature 1

iPhone 17 Pro Already Rumored to Have These 8 New Features

Wednesday November 27, 2024 12:19 pm PST by
While the iPhone 17 Pro and iPhone 17 Pro Max are not expected to launch for 10 more months, there are already plenty of rumors about the devices. An imaginative iPhone 17 Pro concept based on rumors Below, we recap key changes rumored for the iPhone 17 Pro models so far: Aluminum frame: iPhone 17 Pro models are rumored to have an aluminum frame, whereas the iPhone 15 Pro and iPhone 16 Pro ...
General Black Friday Deals 24 Blue Snowmen

The Best 50+ Black Friday Apple Deals Still Available on AirPods, iPads, MacBooks, and More

Thursday November 28, 2024 1:01 pm PST by
Black Friday 2024 is over, but you can still find great deals on numerous Apple devices this weekend. Right now, this includes big savings on AirPods, Apple Watch, MacBook Air, iPad, and more. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. Specifically, in...
maxresdefault

The MacRumors Show: iPhone 17 Designs Revealed!

Friday November 29, 2024 9:34 am PST by
On this week's episode of The MacRumors Show, we discuss the recently leaked design of the iPhone 17 "Air" and iPhone 17 Pro. Subscribe to The MacRumors Show YouTube channel for more videos Earlier this week, a report from The Information's Wayne Ma revealed that the iPhone 17 Air will have a thickness of between 5mm and 6mm, which would make it the thinnest iPhone ever. In comparison, iPhone ...
airpods pro 2 gradient

AirPods Pro 3 Expected Next Year: Here's What We Know

Thursday November 28, 2024 3:30 am PST by
Despite being released over two years ago, Apple's AirPods Pro 2 continue to dominate the wireless earbud market. However, with the AirPods Pro 3 expected to launch sometime in 2025, anyone thinking of buying Apple's premium earbuds may be wondering if the next generation is worth holding out for. Apart from their audio and noise-canceling performance, which are generally regarded as...
new streaming black friday

Cyber Monday Streaming Deals Include Big Savings on Disney+, Hulu, Paramount+, and Peacock

Friday November 29, 2024 5:38 am PST by
Black Friday is over, but you can still find great savings on streaming memberships this weekend as we head into Cyber Monday. Some of the biggest services have great discounts for new and select returning members, including Disney+, Hulu, Paramount+, and Peacock. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a...
Apple AI Command Center Concept Mock 3

Apple Expected to Launch This All-New Device Next Year

Wednesday November 27, 2024 1:05 pm PST by
Apple is expected to kick off 2025 by launching an all-new smart home hub, also referred to as a "command center," as early as March. The hub is expected to feature around a six-inch display that can be attached to a tabletop base with a speaker, or mounted on a wall. The device is said to run a new "homeOS" operating system with a customizable widget-focused home screen, and it is expected...

Top Rated Comments

smuzzy Avatar
58 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
58 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
58 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
58 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
58 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
58 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)