Apple Patent Details Visual-Based AR Navigation Device

by

Apple has been granted an augmented reality navigation patent stemming from its acquisition of AR startup Flyby Media earlier this year (via AppleInsider).

The patent was published today by the U.S. Patent and Trademark Office under the title "Visual-based inertial navigation", and describes a system that allows a consumer device to position itself in three-dimensional space using data from cameras and sensors.

AR patent flyby
The system combines images from an onboard camera with measurements gleaned from a gyroscope and accelerometers as well as other sensors, to build a picture of the device's real-time position in physical space.

The patent notes that visual-based inertial navigation systems can achieve positional awareness down to the centimeter scale without the need for GPS or cellular network signals. However, the technology is unsuitable for implementation in typical mobile devices because of the processing demands involved in variable real-time location tracking.

To overcome the limitation, Apple's invention uses something called a sliding window inverse filter (SWF) that minimizes computational load by using predictive coding to map the orientation of objects relative to the device.

AR patent flyby 2
The system could be used in a navigational AR device that overlays an output image with location-based information. One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles. Another describes the use of depth sensors to generate a 3D map of a given environment.

Whether or not Apple uses the patent in an upcoming product is obviously unknown at this time, but the company has been relatively open about its interest in innovating in the virtual reality and AR space. Apple is said to have a large team experimenting with headsets and other technologies and is believed to have been working in the area since at least early 2015.

The patent was filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Michigan professor Stergios Roumeliotis, as its inventors.

Tag: patent

Top Rated Comments

(View all)
Avatar
55 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Score: 3 Votes (Like | Disagree)
Avatar
55 months ago
It would be nice if this could be used to help guide blind people too....
Score: 2 Votes (Like | Disagree)
Avatar
55 months ago

One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles.

This is something that I would find quite useful. I'm the type of shopper that wants to get in the store and get out as quickly as I can. As long as I already know where everything I need/want is in the store, like my weekly grocery shopping task, I'm good. I don't particularly like having to hunt for something in a store that I don't normally frequent. If I had the ability to map out a specific items location in an unfamiliar store and then walk right to it when I get that, that would be quite useful to me.
Score: 1 Votes (Like | Disagree)
Avatar
55 months ago

It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.

Came to the comments to make a similar point (this seems obviously developed with intention to use in the Car; side benefit may be application on personal devices too) but you've beaten me to it - and with a more detailed post :)
Score: 1 Votes (Like | Disagree)
Avatar
55 months ago
While this might be useful in a Car windshield.... It may prove a little dangerous if not limited to minimal relevant use!

Microsoft have gone on record saying that they will be launching Hololens for Windows 10 next year and it will be integrated into the Windows Desktop or some new 3d AR version of it.

It makes sense for Apple to be developing some Operating System level AR functionality where you can run multiple AR applications at the same time. So you can be running a virtual TV on your wall while you play a virtual game on the table in front of you and look out the window and see the names of stars and planets in the sky provided by your astronomy app for example.

For this is be viable Apple need to update all the Mac range with GPUs that are powerful enough to support AR.

If Apple aren't working on this they will miss the next big thing. This will hit before any Apple Car. This could be the next Mac, iPod, iPhone, iPad type product. iSee?
Score: 1 Votes (Like | Disagree)

Top Stories

First iPhone 12 Mini Hands-On Video Surfaces [Update: Video Pulled]

Wednesday October 28, 2020 1:21 pm PDT by
Apple's iPhone 12 mini and the iPhone 12 Pro Max aren't set to be available until November 13, but a Romanian YouTuber got his hands on the iPhone 12 mini and showed it off today, offering a size comparison between the iPhone 12 and the 12 mini along with going over some of the device's features. The iPhone 12 mini is identical to the iPhone 12 in design and functionality, but it has a...

Apple References Unreleased 2020 16-Inch MacBook Pro in Boot Camp Update

Monday October 26, 2020 8:42 am PDT by
Last week, Apple released an update for Boot Camp, its utility for running Windows on a Mac. While this update would typically be unremarkable, several of our readers noticed that the release notes reference an unreleased 2020 model of the 16-inch MacBook Pro. While this could easily be a mistake, the 16-inch MacBook Pro is nearly a year old, so it is certainly a worthy candidate for a...

After Mocking Apple, Samsung May Remove Power Adapter From Galaxy S21 Box

Tuesday October 27, 2020 4:29 pm PDT by
Samsung's Galaxy S21, coming in 2021, may not include a power adapter or headphones in the box, according to reports from Korean media sites highlighted by SamMobile. Rumors earlier this year also said that Samsung was considering removing these accessories from future smartphone models, but that didn't stop Samsung from mocking Apple for selling the iPhone 12 models without a power adapter...

Report: Apple Silicon iMac Featuring Desktop Class 'A14T' Chip Coming First Half of 2021

Tuesday October 27, 2020 4:14 am PDT by
The first iMac powered by Apple Silicon is set to arrive in the first half of next year and will feature a desktop class "A14T" chip, according to Chinese-language newspaper The China Times. Codenamed "Mt. Jade," Apple's first custom-made desktop processor will be twinned with its first self-developed GPU, codenamed "Lifuka," both of which are being produced using TSMC's 5-nanometer process, ...

iPhone 12 Ceramic Shield Still 'Scratches at Level 6 With Deeper Grooves at Level 7' in Mohs Hardness Test

Wednesday October 28, 2020 7:10 am PDT by
iPhone 12 and iPhone 12 Pro models feature a new Ceramic Shield front cover that is "tougher than any smartphone glass," according to Apple, but the displays on the devices still have similar scratch resistance as previous iPhones based on a new test. Zack Nelson today shared his much-anticipated iPhone 12 Pro durability test on his YouTube channel JerryRigEverything, and based on the Mohs...

2020 iPad Air vs. iPad Pro: Hands-On Comparison

Tuesday October 27, 2020 3:03 pm PDT by
Apple announced the new 2020 fourth-generation iPad Air in September, but the new tablets just started shipping out to customers last Friday. We picked one up and thought we'd do a hands-on comparison with the iPad Pro, which was last updated in March, because both tablets are about as powerful and share many similarities. Subscribe to the MacRumors YouTube channel for more videos. Design and ...

MagSafe Charger Only Charges at Full 15W Speeds With Apple's 20W Power Adapter [Updated]

Monday October 26, 2020 3:38 pm PDT by
Alongside the iPhone 12 and 12 Pro models, Apple introduced a new MagSafe charger that attaches to the magnetic ring in the back of the devices, providing up to 15W of charging power, which is double the speed of the 7.5W Qi-based wireless charging maximum. Apple does not provide a power adapter with the $39 MagSafe charger, requiring users to supply their own USB-C compatible option. Apple...

Apple Files Mystery 'Personal Computer' With Placeholder 'B2002' Name in Bluetooth Product Database

Tuesday October 27, 2020 12:36 pm PDT by
Last week, a listing appeared in the Bluetooth product database for an Apple product with a placeholder name "B2002" and a model number of "TBD." MacRumors was alerted to the listing by health and fitness tech website MyHealthyApple. The product is filed under the "personal computer" category, which Apple has used for previous Mac and iPad listings in the database, so it is hard to pinpoint...

Apple Releases First macOS Big Sur 11.0.1 Beta to Developers [Update: Public Beta Available]

Wednesday October 28, 2020 10:15 am PDT by
Apple today seeded a new macOS Big Sur 11.0.1 beta to developers for testing purposes, with the new beta replacing the existing macOS Big Sur 11 beta 10 update that was released two weeks ago. The macOS Big Sur beta can be downloaded through the Apple Developer Center and once the appropriate profile is installed, subsequent betas will be available through the Software Update mechanism in...

iPhone 12 Models Might Support Reverse Charging of Future Apple Accessories According to FCC Filing

Tuesday October 27, 2020 6:25 pm PDT by
iPhone 12 models could have an inactive wireless charging feature for accessories, according to an FCC filing discovered by VentureBeat's Jeremy Horwitz. In the filing, Apple said that 2020 iPhones support a wireless charging function that will seemingly be enabled for at least one future Apple accessory:In addition to being able to be charged by a desktop WPT charger (puck), 2020 iPhones...