Apple has been granted an augmented reality navigation patent stemming from its acquisition of AR startup Flyby Media earlier this year (via AppleInsider).

The patent was published today by the U.S. Patent and Trademark Office under the title "Visual-based inertial navigation", and describes a system that allows a consumer device to position itself in three-dimensional space using data from cameras and sensors.

AR patent flyby
The system combines images from an onboard camera with measurements gleaned from a gyroscope and accelerometers as well as other sensors, to build a picture of the device's real-time position in physical space.

The patent notes that visual-based inertial navigation systems can achieve positional awareness down to the centimeter scale without the need for GPS or cellular network signals. However, the technology is unsuitable for implementation in typical mobile devices because of the processing demands involved in variable real-time location tracking.

To overcome the limitation, Apple's invention uses something called a sliding window inverse filter (SWF) that minimizes computational load by using predictive coding to map the orientation of objects relative to the device.

AR patent flyby 2
The system could be used in a navigational AR device that overlays an output image with location-based information. One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles. Another describes the use of depth sensors to generate a 3D map of a given environment.

Whether or not Apple uses the patent in an upcoming product is obviously unknown at this time, but the company has been relatively open about its interest in innovating in the virtual reality and AR space. Apple is said to have a large team experimenting with headsets and other technologies and is believed to have been working in the area since at least early 2015.

The patent was filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Michigan professor Stergios Roumeliotis, as its inventors.

Tag: patent

Top Rated Comments

Thunderstrike76 Avatar
66 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Score: 3 Votes (Like | Disagree)
iamnotme Avatar
66 months ago
It would be nice if this could be used to help guide blind people too....
Score: 2 Votes (Like | Disagree)
SandboxGeneral Avatar
66 months ago
One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles.
This is something that I would find quite useful. I'm the type of shopper that wants to get in the store and get out as quickly as I can. As long as I already know where everything I need/want is in the store, like my weekly grocery shopping task, I'm good. I don't particularly like having to hunt for something in a store that I don't normally frequent. If I had the ability to map out a specific items location in an unfamiliar store and then walk right to it when I get that, that would be quite useful to me.
Score: 1 Votes (Like | Disagree)
orbital~debris Avatar
66 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Came to the comments to make a similar point (this seems obviously developed with intention to use in the Car; side benefit may be application on personal devices too) but you've beaten me to it - and with a more detailed post :)
Score: 1 Votes (Like | Disagree)
anthonylambert Avatar
66 months ago
While this might be useful in a Car windshield.... It may prove a little dangerous if not limited to minimal relevant use!

Microsoft have gone on record saying that they will be launching Hololens for Windows 10 next year and it will be integrated into the Windows Desktop or some new 3d AR version of it.

It makes sense for Apple to be developing some Operating System level AR functionality where you can run multiple AR applications at the same time. So you can be running a virtual TV on your wall while you play a virtual game on the table in front of you and look out the window and see the names of stars and planets in the sky provided by your astronomy app for example.

For this is be viable Apple need to update all the Mac range with GPUs that are powerful enough to support AR.

If Apple aren't working on this they will miss the next big thing. This will hit before any Apple Car. This could be the next Mac, iPod, iPhone, iPad type product. iSee?
Score: 1 Votes (Like | Disagree)

Top Stories

iphone 12 colors 2021

iPhone 12 Colors: Deciding on The Right Color

Thursday November 5, 2020 8:35 am PST by
The iPhone 12 and iPhone 12 Pro arrived last October in a range of color options, with entirely new hues available on both devices, as well as some popular classics. The 12 and 12 Pro have different color choices, so if you have your heart set on a particular shade, you might not be able to get your preferred model in that color. iPhone 12 mini and iPhone 12 The iPhone 12 mini and iPhone 12...
iPhone 13 Dummy Thumbnail 2

Full iPhone 13 Feature Breakdown: Everything Rumors Say We Can Expect

Tuesday August 31, 2021 7:50 am PDT by
With the launch of Apple's iPhone 13 lineup believed to be just a few weeks away, we have compiled all of the coherent rumors from our coverage over the past year to build a full picture of the features and upgrades coming to the company's new smartphones. For clarity, only explicit improvements, upgrades, and new features compared to the iPhone 12 lineup are listed. It is worth noting that...
original iphone

Phil Schiller Says iPhone Was 'Earth-Shattering' Ten Years Ago and Remains 'Unmatched' Today

Monday January 9, 2017 7:15 am PST by
To commemorate the tenth anniversary of the iPhone, Apple marketing chief Phil Schiller sat down with tech journalist Steven Levy for a wide-ranging interview about the smartphone's past, present, and future. The report first reflects upon the iPhone's lack of support for third-party apps in its first year. The argument inside Apple was split between whether the iPhone should be a closed...
macbook air deals

Deals: Amazon Drops Price of 256GB M1 MacBook Air to New Low of $849.99 ($149 Off)

Friday August 27, 2021 6:16 am PDT by
Amazon today introduced new low prices on the M1 MacBook Air for both 256GB and 512GB storage options. To start, you can get the 256GB model for $849.99, down from an original price of $999.00. Note: MacRumors is an affiliate partner with Amazon. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. Only Silver and Gold are...
iCloud General Feature

iCloud+'s New Custom Email Domain Feature Now Available in Beta

Wednesday August 25, 2021 7:48 am PDT by
Starting with iOS 15, iPadOS 15, and macOS Monterey, users with a paid iCloud+ storage plan can personalize their iCloud email address with a custom domain name, such as johnny@appleseed.com, and the feature is now available in beta. iCloud+ subscribers interested in setting up a custom email domain can visit the beta.icloud.com website, select "Account Settings" under their name, and select ...
iPhone 13 Dummy Thumbnail 2

Kuo: iPhone 13 to Feature LEO Satellite Communications to Make Calls and Texts Without Cellular Coverage

Sunday August 29, 2021 7:39 am PDT by
The iPhone 13 will feature low earth orbit (LEO) satellite communication connectivity to allow users to make calls and send messages in areas without 4G or 5G coverage, according to the reliable analyst Ming-Chi Kuo. In a note to investors, seen by MacRumors, Kuo explained that the iPhone 13 lineup will feature hardware that is able to connect to LEO satellites. If enabled with the relevant...
maxresdefault

New MacBook Pro Models Coming at WWDC, Suggests Leaker

Monday May 24, 2021 1:27 pm PDT by
New MacBook Pro models are coming at WWDC, according to leaker Jon Prosser who has a mixed track record when it comes to predicting Apple's plans. Subscribe to the MacRumors YouTube channel for more videos. Prosser provided no additional information, but there are new 14 and 16-inch MacBook Pro models in the works. The new MacBook Pros will feature the most radical redesign to the MacBook Pro ...
omg lightning cable comparison

Security Researcher Develops Lightning Cable With Hidden Chip to Steal Passwords

Thursday September 2, 2021 6:59 am PDT by
A normal-looking Lightning cable that can used to steal data like passwords and send it to a hacker has been developed, Vice reports. The "OMG Cable" compared to Apple's Lightning to USB cable. The "OMG Cable" works exactly like a normal Lightning to USB cable and can log keystrokes from connected Mac keyboards, iPads, and iPhones, and then send this data to a bad actor who could be over a...
ted lasso notchless phone

No, That Notchless iPhone Spotted in 'Ted Lasso' Isn't the iPhone 13

Tuesday August 31, 2021 2:15 am PDT by
Recent sightings of a notchless iPhone in highly popular Apple TV+ comedy "Ted Lasso" have led to sensational headlines suggesting this is a canny bit of product placement on Apple's part and that the iPhone 13 will be notchless. In actuality – and this could go without saying – the phone in question is very likely just showing a poorly superimposed display added in post-production. Notchless ...
calculatorapp

iOS 11 Bug: Typing 1+2+3 Quickly in the Calculator App Won't Get You 6

Tuesday October 24, 2017 2:03 pm PDT by
A bug in the built-in Calculator app in iOS 11 is getting some major attention this week, despite the fact that it's been around since iOS 11 was in beta testing. At issue is a calculator animation that causes some symbols to be ignored when calculations are entered in rapid succession. You can try it for yourself: Type 1+2+3 and then the equals sign into the Calculator app quickly. Due to...