Apple Patent Details Visual-Based AR Navigation Device

by

Apple has been granted an augmented reality navigation patent stemming from its acquisition of AR startup Flyby Media earlier this year (via AppleInsider).

The patent was published today by the U.S. Patent and Trademark Office under the title "Visual-based inertial navigation", and describes a system that allows a consumer device to position itself in three-dimensional space using data from cameras and sensors.

AR patent flyby
The system combines images from an onboard camera with measurements gleaned from a gyroscope and accelerometers as well as other sensors, to build a picture of the device's real-time position in physical space.

The patent notes that visual-based inertial navigation systems can achieve positional awareness down to the centimeter scale without the need for GPS or cellular network signals. However, the technology is unsuitable for implementation in typical mobile devices because of the processing demands involved in variable real-time location tracking.

To overcome the limitation, Apple's invention uses something called a sliding window inverse filter (SWF) that minimizes computational load by using predictive coding to map the orientation of objects relative to the device.

AR patent flyby 2
The system could be used in a navigational AR device that overlays an output image with location-based information. One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles. Another describes the use of depth sensors to generate a 3D map of a given environment.

Whether or not Apple uses the patent in an upcoming product is obviously unknown at this time, but the company has been relatively open about its interest in innovating in the virtual reality and AR space. Apple is said to have a large team experimenting with headsets and other technologies and is believed to have been working in the area since at least early 2015.

The patent was filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Michigan professor Stergios Roumeliotis, as its inventors.

Tag: patent

Top Rated Comments

(View all)
Avatar
52 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Score: 3 Votes (Like | Disagree)
Avatar
52 months ago
It would be nice if this could be used to help guide blind people too....
Score: 2 Votes (Like | Disagree)
Avatar
52 months ago

One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles.

This is something that I would find quite useful. I'm the type of shopper that wants to get in the store and get out as quickly as I can. As long as I already know where everything I need/want is in the store, like my weekly grocery shopping task, I'm good. I don't particularly like having to hunt for something in a store that I don't normally frequent. If I had the ability to map out a specific items location in an unfamiliar store and then walk right to it when I get that, that would be quite useful to me.
Score: 1 Votes (Like | Disagree)
Avatar
52 months ago

It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.

Came to the comments to make a similar point (this seems obviously developed with intention to use in the Car; side benefit may be application on personal devices too) but you've beaten me to it - and with a more detailed post :)
Score: 1 Votes (Like | Disagree)
Avatar
52 months ago
While this might be useful in a Car windshield.... It may prove a little dangerous if not limited to minimal relevant use!

Microsoft have gone on record saying that they will be launching Hololens for Windows 10 next year and it will be integrated into the Windows Desktop or some new 3d AR version of it.

It makes sense for Apple to be developing some Operating System level AR functionality where you can run multiple AR applications at the same time. So you can be running a virtual TV on your wall while you play a virtual game on the table in front of you and look out the window and see the names of stars and planets in the sky provided by your astronomy app for example.

For this is be viable Apple need to update all the Mac range with GPUs that are powerful enough to support AR.

If Apple aren't working on this they will miss the next big thing. This will hit before any Apple Car. This could be the next Mac, iPod, iPhone, iPad type product. iSee?
Score: 1 Votes (Like | Disagree)

Top Stories

Supposed iPhone 12 Display Unit Leaks

Thursday August 6, 2020 8:13 am PDT by
An image supposedly of an iPhone 12 display unit has been shared online by leaker "Twitter user Mr. White". Compared to images of an iPhone 11 Pro display piece, this new unit has a reoriented display connector, reaching up from the bottom of the display, rather than from the left-hand side on iPhone 11 Pro. This may be due to the logic board moving to the other side of the device. A...

8 Third-Party Home Screen Widgets That You Can Try Out Now on iOS 14

Wednesday August 5, 2020 12:56 pm PDT by
One of the biggest new features of iOS 14 is Home Screen widgets, which provide information from apps at a glance. The widgets can be pinned to the Home Screen in various spots and sizes, allowing for many different layouts. When the iOS 14 beta was first released in June, widgets were limited to Apple's own apps like Calendar and Weather, but several third-party developers have begun to test ...

Apple Seeds iOS 14 and iPadOS 14 Public Beta 4 to Testers

Thursday August 6, 2020 10:05 am PDT by
Apple today seeded new public betas of upcoming iOS 14 and iPadOS 14 updates to its public beta testing group. Today's software releases, which Apple labels as fourth betas to keep them in line with developer betas, are actually the third betas that Apple has provided and they come two weeks after the prior beta releases. Public beta testers who have signed up for Apple's beta testing...

Apple Takes Legal Action Against Small Company With Pear Logo

Saturday August 8, 2020 11:09 am PDT by
Apple is taking legal action against the developers of the app "Prepear" due to its logo, according to iPhone in Canada. Prepear is an app that helps users discover recipes, plan meals, make lists, and arrange grocery deliveries. The app is a spinoff of "Super Healthy Kids," and the founders claim that they are facing litigation from Apple. Apple reportedly takes issue with Prepear's logo, ...

Google's $349 Pixel 4a vs. Apple's $399 iPhone SE

Wednesday August 5, 2020 1:45 pm PDT by
Google this week launched its newest smartphone, the $349 Pixel 4a, a low-cost device that's designed to compete with other affordable devices like Apple's iPhone SE. We picked up one of the new Pixel 4a smartphones and thought we'd check it out to see how it measures up to the iPhone SE, given that the two devices have such similar price points. Subscribe to the MacRumors YouTube channel ...

2020 iMac Benchmarks Surface Online [Updated]

Thursday August 6, 2020 7:16 am PDT by
Benchmarks from the new 2020 iMac have today been shared online by Mac Otakara. The Geekbench benchmarks are from the newly-released 27-inch iMac with 3.0GHz Intel Core i5 processor and Radeon Pro 5300 graphics, compared to multiple specs of the previous 2019 iMac. The lowest spec 27-inch i5 iMac from 2020 performs about 20 percent better in multicore than the lowest spec 27-inch i5...

New 27-Inch iMac's Storage Affixed to Logic Board, 4TB and 8TB Configurations Have Expansion Connector

Friday August 7, 2020 7:46 am PDT by
Following a report from German blog iFun.de that claimed the new 27-inch iMac's flash storage is soldered to the logic board, MacRumors has obtained additional information in an internal document for Apple technicians. In the document, Apple says that the flash storage is indeed affixed to the logic board and cannot be removed. However, for the 4TB and 8TB configurations, Apple says that a...

Microsoft's xCloud and Xbox Game Pass Not Coming to iOS Due to Apple's Restrictions [Updated]

Thursday August 6, 2020 1:06 pm PDT by
Microsoft's "Project xCloud" streaming game service that pairs with its Xbox Game Pass won't be available on iPhone and iPad when it launches this September, and Apple's App Store restrictions are to blame. Xbox Game Pass and its accompanying xCloud streaming feature offer access to hundreds of games that can be streamed to mobile devices, and Apple says that because it can't review each...

Apple Announces New 27-Inch iMac With 10th-Gen Processors, Up to 128GB RAM, 1080p Webcam, True Tone, and More

Tuesday August 4, 2020 8:07 am PDT by
Apple today announced a new 27-inch iMac with faster 10th-generation Intel Core processor options, next-generation AMD graphics, up to 128GB of RAM, a higher-resolution 1080p front-facing FaceTime camera, a True Tone display with a nano-texture glass option, a T2 chip, higher fidelity speakers, studio-quality microphones, and more. A breakdown of the new 27-inch iMac's features and specs:10th...

Samsung Launches Galaxy Note 20, Galaxy Z Fold 2, and Galaxy Buds to Compete With Apple's iPhones and AirPods Pro

Wednesday August 5, 2020 10:07 am PDT by
Samsung today held a virtual Galaxy Unpacked event where it unveiled its next-generation smartphones that will compete with Apple's 2020 iPhone lineup, set to come out in the fall. Samsung announced the launch of the Galaxy Note 20 and the Galaxy Note 20 Ultra, the two newest devices in the Note lineup, and, more notably, the Galaxy Z Fold 2, Samsung's latest foldable smartphone. The...