Apple has been granted an augmented reality navigation patent stemming from its acquisition of AR startup Flyby Media earlier this year (via AppleInsider).

The patent was published today by the U.S. Patent and Trademark Office under the title "Visual-based inertial navigation", and describes a system that allows a consumer device to position itself in three-dimensional space using data from cameras and sensors.

AR patent flyby
The system combines images from an onboard camera with measurements gleaned from a gyroscope and accelerometers as well as other sensors, to build a picture of the device's real-time position in physical space.

The patent notes that visual-based inertial navigation systems can achieve positional awareness down to the centimeter scale without the need for GPS or cellular network signals. However, the technology is unsuitable for implementation in typical mobile devices because of the processing demands involved in variable real-time location tracking.

To overcome the limitation, Apple's invention uses something called a sliding window inverse filter (SWF) that minimizes computational load by using predictive coding to map the orientation of objects relative to the device.

AR patent flyby 2
The system could be used in a navigational AR device that overlays an output image with location-based information. One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles. Another describes the use of depth sensors to generate a 3D map of a given environment.

Whether or not Apple uses the patent in an upcoming product is obviously unknown at this time, but the company has been relatively open about its interest in innovating in the virtual reality and AR space. Apple is said to have a large team experimenting with headsets and other technologies and is believed to have been working in the area since at least early 2015.

The patent was filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Michigan professor Stergios Roumeliotis, as its inventors.

Tag: Patent

Top Rated Comments

Thunderstrike76 Avatar
98 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Score: 3 Votes (Like | Disagree)
iamnotme Avatar
98 months ago
It would be nice if this could be used to help guide blind people too....
Score: 2 Votes (Like | Disagree)
SandboxGeneral Avatar
98 months ago
One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles.
This is something that I would find quite useful. I'm the type of shopper that wants to get in the store and get out as quickly as I can. As long as I already know where everything I need/want is in the store, like my weekly grocery shopping task, I'm good. I don't particularly like having to hunt for something in a store that I don't normally frequent. If I had the ability to map out a specific items location in an unfamiliar store and then walk right to it when I get that, that would be quite useful to me.
Score: 1 Votes (Like | Disagree)
orbital~debris Avatar
98 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Came to the comments to make a similar point (this seems obviously developed with intention to use in the Car; side benefit may be application on personal devices too) but you've beaten me to it - and with a more detailed post :)
Score: 1 Votes (Like | Disagree)
anthonylambert Avatar
98 months ago
While this might be useful in a Car windshield.... It may prove a little dangerous if not limited to minimal relevant use!

Microsoft have gone on record saying that they will be launching Hololens for Windows 10 next year and it will be integrated into the Windows Desktop or some new 3d AR version of it.

It makes sense for Apple to be developing some Operating System level AR functionality where you can run multiple AR applications at the same time. So you can be running a virtual TV on your wall while you play a virtual game on the table in front of you and look out the window and see the names of stars and planets in the sky provided by your astronomy app for example.

For this is be viable Apple need to update all the Mac range with GPUs that are powerful enough to support AR.

If Apple aren't working on this they will miss the next big thing. This will hit before any Apple Car. This could be the next Mac, iPod, iPhone, iPad type product. iSee?
Score: 1 Votes (Like | Disagree)

Popular Stories

iOS 17

10 New Things Your iPhone Can Do in Next Week's iOS 17.4 Update

Friday March 1, 2024 1:30 am PST by
Apple will this month release iOS 17.4, its biggest iPhone software update of the year so far, featuring a number of features and changes that users have been anticipating for quite a while. Below, we've listed 10 new things that your iPhone will be able to do after you've installed the update, which is projected to arrive by March 7. When the day arrives, be sure to check Settings ➝...
Apple Maps vs Google Maps Feature

Apple Maps vs. Google Maps: Which Is Better?

Friday March 1, 2024 7:10 am PST by
Apple Maps has been providing navigational guidance to Apple users for almost 13 and a half years now, and much has changed about the app in that time. However, according to data from Canalys, the overwhelming majority of iPhones in the U.S. still have Google Maps downloaded as an alternative to Apple Maps, which comes preinstalled on all iPhones. We want to hear from MacRumors readers. Which do...
apple tv plus banner

Apple TV+ Gains Over 50 Movies for a Limited Time

Friday March 1, 2024 6:29 am PST by
Apple TV+ today gained over 50 movies, adding to its back catalog of content for a limited time. The collection includes a large number of popular and classic titles. Subscribers can access the movies in a "Great Movies on Apple TV+" section in the Apple TV app. Some titles are also available in 3D. Movies in the collection include: 21 Jump Street 300 American Sniper Argo ...
airpods pro 2 pink

Apple Releases New Beta Firmware for AirPods Pro 2

Thursday February 29, 2024 11:41 am PST by
Apple today introduced a new beta firmware update for the AirPods Pro 2, both the USB-C and Lightning versions. The new firmware is version 6E188, up from the prior 6B34 firmware released in December. Apple does not often provide details or notes on what features might be included in the refreshed firmware, so it is unclear what's new. Note that this software is limited to developers at the...
iOS 18 Mock Feature Baubles

Will Your iPhone Support iOS 18? Here's What the Latest Rumor Says

Thursday February 29, 2024 7:29 am PST by
iOS 18 and iPadOS 18 are still over three months away from being unveiled, but we may already know which devices will be compatible with the updates. iOS 18 will be compatible with the same iPhone models as iOS 17, meaning that the iPhone XS, iPhone XS Max, and iPhone XR from 2018 will remain supported for at least another year, according to information shared this week by a source with a...
maxresdefault

Apple Spent More Than $10 Billion on Apple Car Before Canceling Project

Wednesday February 28, 2024 5:13 pm PST by
Apple spent more than $10 billion working on the Apple Car over the last decade, according to a report from The New York Times that details the issues the project faced during development. Apple first launched the project in 2014 and let it flounder for more than a decade before calling it off earlier this week. Subscribe to the MacRumors YouTube channel for more videos. Money was spent on...