Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.


This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.


Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.


Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Related Roundups: iOS 14, iPadOS 14, macOS 11 Big Sur

Top Rated Comments

(View all)
Avatar
12 weeks ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
Avatar
12 weeks ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
Avatar
12 weeks ago


Seems a lot like what Xbox was able to do with the Kinect 10 years ago

That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Avatar
12 weeks ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
Avatar
12 weeks ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
Avatar
12 weeks ago


The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.

I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Top Stories

Here's How You Can Download iOS 14 and iPadOS 14 Around the World [It's Out]

Wednesday September 16, 2020 2:36 am PDT by
Apple's official public release of iOS 14 and iPadOS 14 dropped on Wednesday, September 16, just a day after the company released the Golden Master to third-party developers. Also set to be made available to the general public for the first time are watchOS 7 and tvOS 14. Getting Started With iOS 14 Video Click image to watch iOS 14 Getting Started While that's left a lot of developers...

Apple Updates AirPods 2 and AirPods Pro Firmware to Version 3A283

Monday September 14, 2020 11:24 am PDT by
Apple today released new 3A283 firmware updates for the second-generation AirPods and the AirPods Pro. The second-generation AirPods are being updated from the 2D15 firmware they were previously running, while the AirPods Pros are being updated from the 2D27 firmware they had installed previously. Apple does not provide details on what's included in refreshed firmware so we don't know what's ...

New AirPods Pro Firmware Introduces Spatial Audio Support and Automatic Switching

Monday September 14, 2020 12:22 pm PDT by
The new 3A283 firmware that Apple released for the AirPods Pro today appears to introduce support for Spatial Audio, a new feature coming to the higher-end AirPods with iOS 14. Multiple reports on Twitter and the MacRumors forums indicate that the firmware update adds a "Spatial Audio" option to the Control Center, which can be activated to enable the feature. Note that using Spatial Audio...

Spotify Says Apple One Bundle Will Cause 'Irreparable Harm to Developer Community'

Tuesday September 15, 2020 12:26 pm PDT by
Apple today announced Apple One, a series of new subscription bundles that provide access to various Apple services at a combined monthly price. In response, Spotify sent out statements (via Peter Kafka) to the press decrying Apple's anti-competitive behavior and calling on "competition authorities" to stop Apple before it is able to cause "irreparable harm" to developers.Once again, Apple...

Bloomberg: Apple Watch Series 6 With SpO2 Tracking and All-Screen iPad Air Coming at 'Time Flies' Event, Apple Silicon Macs by November

Monday September 14, 2020 5:20 am PDT by
Apple is set to host a virtual event on Tuesday at 10 a.m. Pacific Time, and ahead of time, Bloomberg's Mark Gurman has outlined his expectations for new product announcements both tomorrow and in the months to follow. Gurman, who has a very strong track record, reiterated that this Tuesday's event will be focused on the introduction of new Apple Watch Series 6 and iPad Air models, adding...

iOS 14, iPadOS 14, watchOS 7, and tvOS 14 Will Be Released September 16

Tuesday September 15, 2020 11:06 am PDT by
Apple today announced that iOS 14, iPadOS 14, watchOS 7, and tvOS 14 will be officially released on Wednesday, September 16. Apple has seeded the final Golden Master beta versions of each update to developers in advance. ‌iOS 14‌ and iPadOS 14 introduce dozens of new features, including a redesigned Home Screen that supports widgets on the iPhone and iPad for the first time. For more...

When Will the iPhone 12 Launch? Here's What We Know

Wednesday September 16, 2020 6:12 am PDT by
Yesterday's "Time Flies" Apple event saw the release of the Apple Watch Series 6, Apple Watch SE, iPad 8, and iPad Air 4, but no new iPhone models. Rumors before the event strongly alleged that it would not see the unveiling of new iPhones, with many reports pointing to an October launch. The lack of new iPhone models yesterday seems to confirm that the iPhone 12 lineup will not appear...

Full Transcript of Apple's 'Time Flies' Event With Apple Watch and iPad Updates

Tuesday September 15, 2020 8:46 am PDT by
Apple's virtual "Time Flies" event kicks off today at 10:00 a.m. Pacific Time, with Apple expected to debut new Apple Watch and iPad models, but it appears we may not see the iPhone 12 until next month. While we're not expecting to see new iPhones today, Apple's software updates for its various platforms are likely nearly ready for launch, so we may be hearing more about them today. Check...

Apple Releases iOS 14 and iPadOS 14 With Home Screen Redesign, App Library, Compact UI, Translate App, Scribble Support, App Clips, and More

Wednesday September 16, 2020 12:48 pm PDT by
Apple has released iOS 14 and iPadOS 14, the newest operating system updates designed for the iPhone and iPad. As with all of Apple's software updates, iOS 14 and iPadOS 14 can be downloaded for free. iOS 14 is available on the iPhone 6s and later, while iPadOS 14 is available on the iPad Air 2 and later. The updates are available on all eligible devices over-the-air in the Settings app. To ...

Everything Apple Announced at Today's iPad and Apple Watch Event in Just Seven Minutes

Tuesday September 15, 2020 2:19 pm PDT by
Apple today held its annual September event, but this year was a bit different because no new iPhones were announced. The "Time Flies" event focused on the Apple Watch Series 6 and the iPad Air, and saw the debut of a new Apple One services bundle. Subscribe to the MacRumors YouTube channel for more videos. It took Apple an hour to introduce the new devices, but we've recapped the event in...