Apple Patents Detail AR/VR System Suitable for Smartphones, Explore Object Recognition Challenges

Tuesday January 31, 2017 3:00 AM PST by Tim Hardwick

Apple was granted a pair of patents on Tuesday that depict a mobile augmented reality system that can detect objects in the surrounding environment and overlay them with virtual information.

Picked up by AppleInsider, the first patent published by the U.S. Patent and Trademark Office is titled "Wearable information system having at least one camera" and describes an AR device with intelligent object recognition capabilities.

AR patent
Combining cameras, a screen, and a user interface, the device is described as being ideal for a head-mounted display, but the patent also suggests a use for the system in future smartphones, owing to the power efficient way in which it monitors the environment.

The device is described as having a default low-power scanning mode for normal operation, with high-power modes activated for short periods – when downloading and displaying AR content, for example. The rest of the patent gets quite technical as it details methods of optical tracking and determining camera orientation, two areas of AR that will prove particularly difficult to perfect.

Detecting, matching, and describing environmental features quickly and accurately are two challenges that Apple suggests could be overcome using a combination of dedicated hardware and pre-learned data. One aspect of such a system would use depth detection, achieved via dual-lens cameras similar to those found in the iPhone 7 Plus, which also utilizes depth mapping algorithms to produce Portrait Mode photos.

Apple AR patent
The second patent, titled "Method for representing virtual information in a real environment", details a way of labeling points in a topological real-world space while taking in environmental features such as object occlusion as the user moves around.

In one example, a user views an AR city map on a transparent display – or on a smartphone screen feeding a live image from a rear-facing camera – and pulls up information relating to nearby buildings or landmarks.

The patent goes on to suggest using geometry models, depth-sensing, positioning data and other advanced technologies to compensate for shortcomings in current visual modeling systems. The second patent also relies on a dual-camera system for creating a depth map of the immediate environment onto which the virtual data is superimposed.

The described technologies may or may not appear in future Apple products, but it's worth noting that both patents were originally filed for by German AR specialist Metaio just before Apple acquired the firm in 2015. Existing staff would have likely joined Apple's team dedicated to AR/VR research, to help explore how the technologies can be incorporated into future products.

Several times over the last year, Tim Cook has spoken of Apple's deep interest in augmented reality. "AR can be really great. We have been and continue to invest a lot in this. We're high on AR in the long run," he said in July.

Related Roundup: Apple Glasses
Tag: patent

Top Rated Comments

(View all)
Avatar
42 months ago

That's right, one more device to charge every night. :p

Well, my 6-port USB hub still has 2 ports available...
Score: 2 Votes (Like | Disagree)
Avatar
42 months ago
Why is the dude in the first graphic touching himself.
Score: 1 Votes (Like | Disagree)
Avatar
42 months ago
I'm excited for the future.
Score: 1 Votes (Like | Disagree)

Top Stories

New Low-Cost 'iPhone SE' Could Launch as Soon as Tomorrow

Thursday April 2, 2020 4:06 pm PDT by Juli Clover
Apple's new low-cost iPhone is set to launch as early as Friday, April 3, according to a new report from 9to5Mac that cites a tip from a "highly trusted reader." The site says that while it can't be certain about the launch date, "Apple could reveal and begin taking orders for the new iPhone as soon as tomorrow." The iPhone 8 Apple is said to be planning to call the new iPhone, which is...

Apple's 2020 MacBook Air vs. 2020 iPad Pro

Wednesday April 1, 2020 2:45 pm PDT by Juli Clover
Apple in March updated both the MacBook Air and the iPad Pro, and with the iPad Pro increasingly positioned as a computer replacement, we thought we'd compare both new machines to see how they measure up and which one might be a better buy depending on user needs. Subscribe to the MacRumors YouTube channel for more videos. We're comparing the base model 12.9-inch iPad Pro and the base model...

AirTags Referenced in New Apple Support Video

Thursday April 2, 2020 12:12 pm PDT by Joe Rossignol
Apple has accidentally referenced its widely rumored AirTags item tracking tags in a video that it uploaded to its Apple Support channel on YouTube today. The video was first spotted by the blog Appleosophy and has quickly been removed. The video was titled "How to erase your iPhone." AirTags were mentioned in Settings > Apple ID > Find My > Find My iPhone under Enable Offline Finding, with...

Intel Unveils 10th-Gen Processors Suitable for Next 16-Inch MacBook Pro With Wi-Fi 6 and Turbo Boost Speeds Above 5GHz

Thursday April 2, 2020 7:53 am PDT by Joe Rossignol
Intel today announced the launch of its latest 10th-generation Core processors for high-end notebooks, potentially including the next 16-inch MacBook Pro. The batch of 45W chips, part of the Comet Lake family, are built on Intel's 14nm++ architecture. The new H-series chips have the same base clock speeds as the 9th-generation chips in the current 16-inch MacBook Pro, but Turbo Boost speeds...

Apple Paid Hacker $75,000 for Uncovering Zero-Day Camera Exploits in Safari

Friday April 3, 2020 3:58 am PDT by Tim Hardwick
Apple paid out $75,000 to a hacker for identifying multiple zero-day vulnerabilities in its software, some of which could be used to hijack the camera on a MacBook or an iPhone, according to Forbes. A zero-day vulnerability refers to a security hole in software that is unknown to the software developer and the public, although it may already be known by attackers who are quietly exploiting...

Apple Offers Employees Deep Discounts on HomePod, Beats Headphones and More

Friday April 3, 2020 11:14 am PDT by Juli Clover
Apple is offering employees significant discounts on products that include the HomePod and Beats headphones, according to details on the deals shared by MacGeneration and 9to5Mac. Apple is said to be clearing stock of Beats headphones and HomePods, providing them to employees through an internal deals program. The HomePod is available at a 50 percent discount, dropping the price to $149.50,...

iPhone 8 Screen Protector Updated With 'iPhone SE' Compatibility on Apple's Online Store

Thursday April 2, 2020 8:10 pm PDT by Joe Rossignol
Another clue has surfaced to suggest that Apple's rumored lower-cost iPhone SE successor will likely be released soon. Earlier today, a product listing for a Belkin screen protector on Apple's online store was updated to reflect compatibility with not only the iPhone 7 and iPhone 8, but also the iPhone SE. Given the original iPhone SE was a 4-inch device, and the iPhone 7 and iPhone 8 are...

Apple Adding Some 2013 and 2014 MacBook Air and MacBook Pro Models to Vintage Products List at End of April

Wednesday April 1, 2020 2:24 pm PDT by Joe Rossignol
In an internal memo obtained by MacRumors, Apple has indicated that the following 2013 and 2014 models of the MacBook Air and MacBook Pro will be added to its vintage and obsolete products list on April 30:MacBook Air (11-inch, Mid 2013) MacBook Air (13-inch, Mid 2013) MacBook Air (11-inch, Early 2014) MacBook Air (13-inch, Early 2014) MacBook Pro (13-inch, Mid 2014)Apple defines vintage...

Zoom Updates Mac App Installer to Remove Controversial 'Preflight' Installation Method

Thursday April 2, 2020 11:42 am PDT by Eric Slivka
The popular video conferencing company Zoom has been under scrutiny over the past few days for the method used to install the Mac version of its app, which essentially abused a "preflight" compatibility check to install the app without the user needing to explicitly grant installation permission, as highlighted by developer Felix Seele earlier this week. Screenshot of preflight compatibility ...

Top Stories: Apple Leaks iPhone SE and AirTags, Apple Buys Dark Sky, and More

Saturday April 4, 2020 6:00 am PDT by MacRumors Staff
With the calendar rolling over to April this week, we yet again saw several leaks and rumors, most notably including Apple itself leaking some references to a pair of long-rumored products: a new budget iPhone SE and AirTags item trackers. Subscribe to the MacRumors YouTube channel for more videos. Apple also acquired popular weather app Dark Sky, while Amazon's Prime Video app now allows...