Apple has been granted an augmented reality navigation patent stemming from its acquisition of AR startup Flyby Media earlier this year (via AppleInsider).

The patent was published today by the U.S. Patent and Trademark Office under the title "Visual-based inertial navigation", and describes a system that allows a consumer device to position itself in three-dimensional space using data from cameras and sensors.

AR patent flyby
The system combines images from an onboard camera with measurements gleaned from a gyroscope and accelerometers as well as other sensors, to build a picture of the device's real-time position in physical space.

The patent notes that visual-based inertial navigation systems can achieve positional awareness down to the centimeter scale without the need for GPS or cellular network signals. However, the technology is unsuitable for implementation in typical mobile devices because of the processing demands involved in variable real-time location tracking.

To overcome the limitation, Apple's invention uses something called a sliding window inverse filter (SWF) that minimizes computational load by using predictive coding to map the orientation of objects relative to the device.

AR patent flyby 2
The system could be used in a navigational AR device that overlays an output image with location-based information. One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles. Another describes the use of depth sensors to generate a 3D map of a given environment.

Whether or not Apple uses the patent in an upcoming product is obviously unknown at this time, but the company has been relatively open about its interest in innovating in the virtual reality and AR space. Apple is said to have a large team experimenting with headsets and other technologies and is believed to have been working in the area since at least early 2015.

The patent was filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Michigan professor Stergios Roumeliotis, as its inventors.

Tag: Patent

Top Rated Comments

Thunderstrike76 Avatar
84 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Score: 3 Votes (Like | Disagree)
iamnotme Avatar
84 months ago
It would be nice if this could be used to help guide blind people too....
Score: 2 Votes (Like | Disagree)
SandboxGeneral Avatar
84 months ago
One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles.
This is something that I would find quite useful. I'm the type of shopper that wants to get in the store and get out as quickly as I can. As long as I already know where everything I need/want is in the store, like my weekly grocery shopping task, I'm good. I don't particularly like having to hunt for something in a store that I don't normally frequent. If I had the ability to map out a specific items location in an unfamiliar store and then walk right to it when I get that, that would be quite useful to me.
Score: 1 Votes (Like | Disagree)
orbital~debris Avatar
84 months ago
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Came to the comments to make a similar point (this seems obviously developed with intention to use in the Car; side benefit may be application on personal devices too) but you've beaten me to it - and with a more detailed post :)
Score: 1 Votes (Like | Disagree)
anthonylambert Avatar
84 months ago
While this might be useful in a Car windshield.... It may prove a little dangerous if not limited to minimal relevant use!

Microsoft have gone on record saying that they will be launching Hololens for Windows 10 next year and it will be integrated into the Windows Desktop or some new 3d AR version of it.

It makes sense for Apple to be developing some Operating System level AR functionality where you can run multiple AR applications at the same time. So you can be running a virtual TV on your wall while you play a virtual game on the table in front of you and look out the window and see the names of stars and planets in the sky provided by your astronomy app for example.

For this is be viable Apple need to update all the Mac range with GPUs that are powerful enough to support AR.

If Apple aren't working on this they will miss the next big thing. This will hit before any Apple Car. This could be the next Mac, iPod, iPhone, iPad type product. iSee?
Score: 1 Votes (Like | Disagree)

Popular Stories

Multi Display CarPlay 1

Apple Launching All-New CarPlay Experience Later This Year With These 5 Features

Sunday January 29, 2023 10:15 am PST by
In June 2022, Apple previewed the next generation of CarPlay, promising deeper integration with vehicle functions like A/C and FM radio, support for multiple displays across the dashboard, personalization options, and more. Apple says the first vehicles with support for the next-generation CarPlay experience will be announced in late 2023, with committed automakers including Acura, Audi,...
dewey airtag

Report Highlights Danger of Using AirTags for Tracking Dogs

Monday January 30, 2023 1:45 pm PST by
AirTags may be a convenient way for tracking dogs that might get off leash or otherwise lost, but there are dangers associated with the practice, as outlined by a report from The Wall Street Journal. At 1.26 inches in diameter, AirTags are able to fit easily on a dog's collar, but that size also makes the tracking devices small enough to swallow, at least for a medium to large-sized dog, and ...
maxresdefault

Kuo: Apple to Release Foldable iPad With Carbon Fiber Kickstand in 2024

Monday January 30, 2023 12:55 am PST by
Apple will launch a foldable iPad with a carbon fiber kickstand sometime next year, according to analyst Ming-Chi Kuo. Subscribe to the MacRumors YouTube channel for more videos. In a series of tweets, Kuo said he expects an "all-new design foldable iPad" to be the next big product launch in the iPad lineup, with no other major iPad releases in the next nine to 12 months. The analyst said he...
tim cook data privacy day

Apple Violated U.S. Labor Laws With Anti-Leak Email

Monday January 30, 2023 3:43 pm PST by
Apple violated United States labor laws when it sent out an email warning employees about leaking confidential information about the company, the National Labor Relations Board (NLRB) said today in a ruling shared by Bloomberg. Rules that Apple has established around leaks "tend to interfere with, restrain or coerce employees" from the exercise of their rights under the National Labor...
top stories 28jan2023

Top Stories: iOS 16.3 Released, iPhone 15 Pro Rumors, macOS Tips and Tricks, and More

Saturday January 28, 2023 6:00 am PST by
Following last week's hardware announcements, this week saw the actual release of several of the new products as well as operating system updates bringing new features and bug fixes across Apple's platforms. This week also saw some fresh rumors about the iPhone 15 lineup and Apple's upcoming AR/VR headset, while we shared some tips to help you get the most of your macOS experience, so read...
iphone 15 pro wifi 6e

Internal Apple Document From Leaker 'Unknownz21' Confirms Wi-Fi 6E Will Be Limited to iPhone 15 Pro Models

Friday January 27, 2023 10:01 am PST by
Multiple rumors have suggested that the next-generation iPhone 15 models will adopt the Wi-Fi 6E standard that Apple has already introduced in the iPad Pro and MacBook Pro, and now a leaked document appears to confirm Apple's plans. Sourced from researcher and Apple leaker Unknownz21 (@URedditor), the document features diagrams of the iPhone 15's antenna architecture. D8x refers to the...
iPhone 14 Pro Purple Side Perspective Feature Purple

iPhone 15 Pro Rumored to Have These 8 Features

Friday January 27, 2023 2:11 pm PST by
Apple's next-generation iPhone 15 Pro and iPhone 15 Pro Max are expected to be announced in September as usual. Already, rumors suggest the devices will have at least eight exclusive features not available on the standard iPhone 15 and iPhone 15 Plus. An overview of the eight features rumored to be exclusive to iPhone 15 Pro models:A17 chip: iPhone 15 Pro models will be equipped with an A17...