Apple has been granted an augmented reality navigation patent stemming from its acquisition of AR startup Flyby Media earlier this year (via AppleInsider).

The patent was published today by the U.S. Patent and Trademark Office under the title "Visual-based inertial navigation", and describes a system that allows a consumer device to position itself in three-dimensional space using data from cameras and sensors.

AR patent flyby
The system combines images from an onboard camera with measurements gleaned from a gyroscope and accelerometers as well as other sensors, to build a picture of the device's real-time position in physical space.

The patent notes that visual-based inertial navigation systems can achieve positional awareness down to the centimeter scale without the need for GPS or cellular network signals. However, the technology is unsuitable for implementation in typical mobile devices because of the processing demands involved in variable real-time location tracking.

To overcome the limitation, Apple's invention uses something called a sliding window inverse filter (SWF) that minimizes computational load by using predictive coding to map the orientation of objects relative to the device.

AR patent flyby 2
The system could be used in a navigational AR device that overlays an output image with location-based information. One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles. Another describes the use of depth sensors to generate a 3D map of a given environment.

Whether or not Apple uses the patent in an upcoming product is obviously unknown at this time, but the company has been relatively open about its interest in innovating in the virtual reality and AR space. Apple is said to have a large team experimenting with headsets and other technologies and is believed to have been working in the area since at least early 2015.

The patent was filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Michigan professor Stergios Roumeliotis, as its inventors.

Tag: patent

Top Rated Comments

Thunderstrike76 Avatar
62 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Score: 3 Votes (Like | Disagree)
iamnotme Avatar
62 months ago
It would be nice if this could be used to help guide blind people too....
Score: 2 Votes (Like | Disagree)
SandboxGeneral Avatar
62 months ago
One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles.
This is something that I would find quite useful. I'm the type of shopper that wants to get in the store and get out as quickly as I can. As long as I already know where everything I need/want is in the store, like my weekly grocery shopping task, I'm good. I don't particularly like having to hunt for something in a store that I don't normally frequent. If I had the ability to map out a specific items location in an unfamiliar store and then walk right to it when I get that, that would be quite useful to me.
Score: 1 Votes (Like | Disagree)
orbital~debris Avatar
62 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Came to the comments to make a similar point (this seems obviously developed with intention to use in the Car; side benefit may be application on personal devices too) but you've beaten me to it - and with a more detailed post :)
Score: 1 Votes (Like | Disagree)
anthonylambert Avatar
62 months ago
While this might be useful in a Car windshield.... It may prove a little dangerous if not limited to minimal relevant use!

Microsoft have gone on record saying that they will be launching Hololens for Windows 10 next year and it will be integrated into the Windows Desktop or some new 3d AR version of it.

It makes sense for Apple to be developing some Operating System level AR functionality where you can run multiple AR applications at the same time. So you can be running a virtual TV on your wall while you play a virtual game on the table in front of you and look out the window and see the names of stars and planets in the sky provided by your astronomy app for example.

For this is be viable Apple need to update all the Mac range with GPUs that are powerful enough to support AR.

If Apple aren't working on this they will miss the next big thing. This will hit before any Apple Car. This could be the next Mac, iPod, iPhone, iPad type product. iSee?
Score: 1 Votes (Like | Disagree)

Top Stories

2021 mbp hdmi slot 3d

2021 MacBook Pro Leaks Confirm Returning MagSafe and Ports

Friday May 14, 2021 3:06 am PDT by
Apple's upcoming MacBook Pro models are expected to feature a number of major changes such as larger display options and powerful new Apple silicon chips. Among the more surprising updates to this year's MacBook Pro models is the return of three ports that have been missing from the machines for over five years. Expected to come in 14- and 16-inch sizes, the 2021 MacBook Pro models are...
General Music and AirPod 3 Feature

Rumor: Apple to Announce Third-Generation AirPods and HiFi Apple Music Tier on May 18

Thursday May 13, 2021 10:32 pm PDT by
A new rumor suggests that Apple will announce the third-generation AirPods and the recently rumored HiFi, or high-fidelity Apple Music tier, on Tuesday, May 18, via a press release on its website. The new rumor comes from Apple YouTuber Luke Miani who shared the alleged exclusive news with the AppleTrack website. According to the YouTuber, Apple plans to release the next-generation AirPods...
m1 ipad pro early customer

M1 iPad Pro Arrives Early for Lucky Customer

Saturday May 15, 2021 11:57 pm PDT by
Days ahead of their expected launch and seemingly before official review embargoes lift, one lucky customer has already gotten their hands on the brand new 12.9-inch M1 iPad Pro. Reddit User PeterDragon50 Posted on Reddit, u/PeterDragon50 has already received their 12.9-inch iPad Pro through retailer Nebraska Furniture Mart. The Reddit user says they placed their order when pre-orders...
Twitter Feature

Twitter's 'Blue' Subscription Service May Cost $2.99, Will Offer Undo Tweet Option

Saturday May 15, 2021 11:08 am PDT by
Twitter has been working on some kind of subscription service since last summer, and Jane Manchun Wong, who often digs into new features coming in apps, has shared details on just what Twitter is exploring. Twitter's subscription service could be called Twitter Blue, and at the current time, it's priced at $2.99 per month. There will be a "Collections" section that allows users to save and...
magic mouse space gray discontinued

Apple Discontinuing Space Gray Mac Accessories Now That iMac Pro is Dead

Friday May 14, 2021 11:52 am PDT by
Following the discontinuation of the iMac Pro, Apple also appears to be discontinuing Space Gray "Magic" accessories that it sold separately alongside the iMac Pro. The iMac Pro was the only Space Gray Mac, and Apple designed special matching accessories for it. The Space Gray Magic Mouse 2, Magic Keyboard, and Magic Trackpad all now say "While supplies last" in small wording at the bottom...
tile amazon sidewalk integration

Apple Says Tile Trackers Sold Poorly in Apple Stores

Friday May 14, 2021 4:53 am PDT by
Earlier last month, Spotify, Tile, and Match (owner of Tinder), testified at an app store antitrust hearing spearheaded by the U.S. Senate. During the hearing, Spotify called Apple's App Store "an abusive power grab," while Tile said Apple uses its platform to "unfairly limit competition for its products." Now, in response to their testimonies, Apple's vice president and chief compliance...
apple music change forever

Apple Music Teaser: 'Get Ready – Music is About to Change Forever'

Sunday May 16, 2021 2:39 pm PDT by
The Browse tab in the Music app across Apple's platforms has started displaying a prominent teaser hinting at an upcoming major announcement for Apple Music. Under the heading "Coming soon," the headline says "Get ready – music is about to change forever." An accompanying "Tune-In Video" simply shows an animated Apple Music logo. Rumors have indicated that Apple is preparing to launch a...
Top Stories 60 Feature

Top Stories: M1 iMac Benchmarks, iPhone Battery Life Tips, Colorful MacBook Air?

Saturday May 15, 2021 6:00 am PDT by
Anticipation is building for the new iMac and iPad Pro models, which have started shipping out ahead of a launch around May 21. Benchmarks for the new machines are starting to come in, and the M1 chips inside of them are registering at around the same figures seen with other M1-based Macs, making for significant upgrades over previous-generation models. This week also saw some rumors about...
syng cell alpha

Longtime Apple Designer Christopher Stringer's Latest Project Is a High-Fidelity Speaker With AirPlay 2

Friday May 14, 2021 7:30 am PDT by
Christopher Stringer, a key member of Jony Ive's design team who spent 21 years at Apple before departing in 2017, is resurfacing today with his new venture Syng, which seeks to make an impact in the high-end audio market. Stringer, who contributed to many of the most iconic product designs in Apple's history, announced his plans roughly a year ago, and Syng is today introducing its flagship ...
google photos

PSA: Google Photos Unlimited Storage Ends Next Month, Here's How to Export Your Pictures to iCloud

Thursday May 13, 2021 5:26 am PDT by
For as long as it's existed, Google Photos has offered free unlimited storage for uploading images at a reduced yet good enough quality for most users. From June 1, 2021, however, all photos and videos uploaded to Google accounts will count against users' cloud storage. If you've been relying on Google to back up your media library, it may be time to move that content elsewhere. This article...