Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Top Rated Comments

smuzzy Avatar
32 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
32 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
32 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
32 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
32 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
32 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Popular Stories

Emergency SOS via Satellite iPhone YT

Apple's iPhone 14 Emergency SOS via Satellite Feature Saves Stranded Man in Alaska

Thursday December 1, 2022 4:37 pm PST by
With the launch of iOS 16.1, Apple rolled out a Emergency SOS via Satellite, which is designed to allow iPhone 14 owners to contact emergency services using satellite connectivity when no cellular or WiFi connection is available. The feature was put to the test in Alaska today, when a man became stranded in a rural area. In the early hours of the morning on December 1, Alaska State Troopers ...
General iOS 16 Feature Yellow

iOS 16.2 for iPhone Launching This Month With These 8 New Features

Thursday December 1, 2022 8:44 am PST by
Apple plans to publicly release iOS 16.2 for the iPhone in mid-December, according to Bloomberg's Mark Gurman. The update remains in beta testing for now, with at least eight new features and changes already uncovered so far. iOS 16.2 introduces a number of new features, including Apple's new whiteboard app Freeform, two new Lock Screen widgets for Sleep and Medications, the ability to hide...
iOS 16

Apple Releases iOS 16.1.2 With Carrier Improvements and Crash Detection Optimizations

Wednesday November 30, 2022 10:09 am PST by
Apple today released iOS 16.1.2, another minor bug fix update that comes one week after the release of iOS 16.1.1 and three weeks after the launch of iOS 16.1, an update that added support for iCloud Shared Photo Library, Matter, Live Activities, and more. The iOS 16.1.2 update can be downloaded on eligible iPhones over-the-air by going to Settings > General > Software Update. According...
iPad 10 Battery Pull Tabs

iPad 10 Teardown Reveals Why Device Isn't Compatible With Apple Pencil 2

Thursday December 1, 2022 10:48 am PST by
Do-it-yourself repair website iFixit today shared a video teardown of Apple's new 10th-generation iPad, providing a closer look inside the tablet and revealing why the device lacks support for the second-generation Apple Pencil. The teardown reveals the internal layout of the iPad, including its two-cell 7,606 mAh battery, logic board with the A14 Bionic chip, and more. As suspected, the...
iOS 16

When Will iOS 16.2 Be Released?

Friday December 2, 2022 2:13 pm PST by
Apple in late October began testing iOS 16.2 and iPadOS 16.2 updates, providing betas to both developers and public beta testers. As of now, we've had four total betas, with the fourth beta having been released earlier this week. iOS 16.2 and iPadOS 16.2 are expected before the end of the year, and we thought we'd try to narrow down the launch timeline. With only four betas released since...
14 vs 16 inch mbp m2 pro and max feature 1

Major RAM Upgrade Coming to Next-Generation MacBook Pro

Friday December 2, 2022 2:03 am PST by
The next-generation MacBook Pro models could feature faster RAM, according to a recent report from a reliable source. MacRumors Forums member "Amethyst," who accurately revealed details about the Mac Studio and Studio Display before those products were announced, recently provided information about Apple's upcoming 14- and 16-inch MacBook Pro models. The new machines are expected to feature...
iOS 16

Apple Still Has These 5 Things to Release Heading Into 2023

Thursday December 1, 2022 7:12 am PST by
The calendar has turned to December and that means Apple has only one month left to fulfill its promises of releasing an Apple Music Classical app and expanding its self-service repair program to Europe before the end of 2022. Delays are always possible, of course, so the plans could be pushed back to 2023. In any case, we have put together a list of five things that Apple still has to release...
iOS 16

Apple Seeds Fourth Betas of iOS 16.2 and iPadOS 16.2 [Update: Public Beta Available]

Thursday December 1, 2022 10:16 am PST by
Apple today seeded the fourth betas of upcoming iOS 16.2 and iPadOS 16.2 updates to developers for testing purposes, with the betas coming two weeks after Apple seeded the third betas of iOS 16.2 and iPadOS 16.2. Registered developers are able to download the iOS 16‌.2 and iPadOS 16.2 profiles from the Apple Developer Center, and once installed, the beta is available over the air. iOS...
iPhone Measure Height

Newer iPhones Allow You to Measure Someone's Height Instantly — Here's How

Saturday December 3, 2022 10:23 am PST by
iPhone 12 Pro and Pro Max, iPhone 13 Pro and Pro Max, and iPhone 14 Pro and Pro Max models feature a LiDAR Scanner next to the rear camera that can be used to measure a person's height instantly in Apple's preinstalled Measure app. To measure a person's height, simply open the Measure app, point your iPhone at the person you want to measure, and make sure they are visible on the screen from...
apple ar headset concept 2

Apple Now Calling AR/VR Headset Operating System 'xrOS'

Thursday December 1, 2022 12:57 pm PST by
Apple has decided to call the software that will run on its upcoming AR/VR headset "xrOS," an update from the original "RealityOS or "rOS" naming the company was planning on, according to Bloomberg. Render created by Ian Zelbo based on rumored information The name change comes as Apple begins to prepare for the launch of the headset, which is expected at some point in 2023. The headset will...