Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Related Roundup: WWDC 2022

Top Rated Comments

smuzzy Avatar
21 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
21 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
21 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
21 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
21 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
21 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)

Popular Stories

maxresdefault

Review: M1 Max MacBook Pro After Three Months

Wednesday January 19, 2022 11:30 am PST by
It's now been a few months since the M1 Pro and M1 Max MacBook Pro models launched in October, and MacRumors video editor Dan Barbera has been using one of the new machines since they debuted. Over on the MacRumors YouTube channel, Dan has shared a three month review of his MacBook Pro to see how it has held up over time and how it's changed his workflow. Subscribe to the MacRumors YouTube ...
iphone se 2020 top

New iPhone SE Likely to Launch in April Based on Production Timeframe

Wednesday January 19, 2022 6:44 am PST by
Apple suppliers will begin producing display panels for the third-generation iPhone SE this month, with final assembly of the device likely to start in March, according to information shared by display industry consultant Ross Young. Based on this production timeframe, Young believes the third-generation iPhone SE is likely to launch in the second half of April, or perhaps in early May at...
AirPods 3 New Firmware Feature

Apple Updates AirPods 3 Firmware to Version 4C170

Tuesday January 18, 2022 11:46 am PST by
Apple today released a new 4C170 firmware update for the AirPods 3, an update from the prior 4C165 that was made available in December. Apple does not offer details on what's included in new firmware updates for the AirPods‌, so we don't know what improvements or bug fixes the new firmware brings. There is no standard way to upgrade the ‌AirPods‌‌ software, but firmware is...
iPad Air Feature 2 green

New Apple Products Filed in Regulatory Database, Likely Including New iPhone SE and iPad Air

Tuesday January 18, 2022 6:11 am PST by
Apple today filed unreleased iPhone and iPad models in the Eurasian Economic Commission database, as spotted by French blog Consomac. The filings likely represent the rumored third-generation iPhone SE, fifth-generation iPad Air, and potentially more. The unreleased iPhone models have the identifiers A2595, A2783, and A2784, while the unreleased iPad models have the identifiers A2588, A2589, ...
iphone 5g mmwave

U.S. Airlines Warn of 'Catastrophic' Crisis With Impending 5G Rollout, AT&T and Verizon Agree to Delay Around Airports

Tuesday January 18, 2022 10:35 am PST by
Verizon and AT&T's upcoming rollout of new C-Band 5G technology could cause chaos and lead to widespread delays of passenger and cargo flights, major U.S. airlines said on Monday in a letter sent to the White House National Economic Council, the FAA, and the FCC (via Reuters). "Unless our major hubs are cleared to fly, the vast majority of the traveling and shipping public will essentially...
microsoft office icons

Microsoft Releases Office for Mac Update With Full Apple Silicon Support in Excel

Tuesday January 18, 2022 4:15 am PST by
Microsoft has released a new version of its Office for Mac productivity suite that includes an updated Excel app with 100% native support for Apple silicon machines. According to the release notes accompanying version 16.57, Excel will now run natively on Macs powered by Apple's M1-series processors without having to use the Rosetta 2 translation layer, which means anyone using a Mac with an ...
appleeducation

Apple's US Education Store Now Requires Institution Verification to Buy Discounted Products

Wednesday January 19, 2022 2:22 am PST by
Apple is now requiring that customers in the United States verify that they're active students, teachers, or staff members at an educational institution in order to access education discounts on products. Previously, little verification was needed for customers to purchase products through Apple's education store in the United States. Apple's education stores offer models of the iPad and Mac ...
iPhone 14 Mock pill and hole thumb

ProMotion Now Expected to Remain Exclusive to iPhone 14 Pro Models, Not Expand to Entire Lineup

Sunday January 16, 2022 8:56 am PST by
Continuing the tradition set with the iPhone 13 Pro, only the highest-end iPhone 14 models will feature Apple's ProMotion display technology, according to a respected display analyst. Ross Young, who on multiple occasions has detailed accurate information about Apple's future products, said in a tweet that ProMotion will not be expanded to the entire iPhone 14 lineup and will remain...
Beyond iPhone 13 Better Blue Face ID

Rumored iPhone 14 Pro Feature Delayed Until Next Year

Wednesday January 19, 2022 3:00 am PST by
Apple's long-rumored under-display Face ID technology will not be coming to the iPhone 14 Pro, according to recent reports. Based on the latest rumors, it now looks unlikely that the 6.1-inch iPhone 14 Pro and 6.7-inch iPhone 14 Pro Max will feature Face ID technology under the display as several reports claimed over the past year. Display industry consultant Ross Young, who often reveals ...