Apple Patents Detail AR/VR System Suitable for Smartphones, Explore Object Recognition Challenges

by

Apple was granted a pair of patents on Tuesday that depict a mobile augmented reality system that can detect objects in the surrounding environment and overlay them with virtual information.

Picked up by AppleInsider, the first patent published by the U.S. Patent and Trademark Office is titled "Wearable information system having at least one camera" and describes an AR device with intelligent object recognition capabilities.

AR patent
Combining cameras, a screen, and a user interface, the device is described as being ideal for a head-mounted display, but the patent also suggests a use for the system in future smartphones, owing to the power efficient way in which it monitors the environment.

The device is described as having a default low-power scanning mode for normal operation, with high-power modes activated for short periods – when downloading and displaying AR content, for example. The rest of the patent gets quite technical as it details methods of optical tracking and determining camera orientation, two areas of AR that will prove particularly difficult to perfect.

Detecting, matching, and describing environmental features quickly and accurately are two challenges that Apple suggests could be overcome using a combination of dedicated hardware and pre-learned data. One aspect of such a system would use depth detection, achieved via dual-lens cameras similar to those found in the iPhone 7 Plus, which also utilizes depth mapping algorithms to produce Portrait Mode photos.

Apple AR patent
The second patent, titled "Method for representing virtual information in a real environment", details a way of labeling points in a topological real-world space while taking in environmental features such as object occlusion as the user moves around.

In one example, a user views an AR city map on a transparent display – or on a smartphone screen feeding a live image from a rear-facing camera – and pulls up information relating to nearby buildings or landmarks.

The patent goes on to suggest using geometry models, depth-sensing, positioning data and other advanced technologies to compensate for shortcomings in current visual modeling systems. The second patent also relies on a dual-camera system for creating a depth map of the immediate environment onto which the virtual data is superimposed.

The described technologies may or may not appear in future Apple products, but it's worth noting that both patents were originally filed for by German AR specialist Metaio just before Apple acquired the firm in 2015. Existing staff would have likely joined Apple's team dedicated to AR/VR research, to help explore how the technologies can be incorporated into future products.

Several times over the last year, Tim Cook has spoken of Apple's deep interest in augmented reality. "AR can be really great. We have been and continue to invest a lot in this. We're high on AR in the long run," he said in July.

Related Roundup: Apple Glasses
Tag: patent

Top Rated Comments

Abazigal Avatar
50 months ago

That's right, one more device to charge every night. :p

Well, my 6-port USB hub still has 2 ports available...
Score: 2 Votes (Like | Disagree)
gtg465x Avatar
50 months ago
Why is the dude in the first graphic touching himself.
Score: 1 Votes (Like | Disagree)
macTW Avatar
50 months ago
I'm excited for the future.
Score: 1 Votes (Like | Disagree)

Top Stories

maxresdefault

Italy Fines Apple $12 Million for Misleading iPhone Water Resistance Claims

Monday November 30, 2020 3:10 am PST by
Apple has been slapped with a 10 million euro ($12 million) fine by Italy's antitrust watchdog for unfair commercial practices related to its iPhone marketing in the country. One of the Apple ads cited in the Italian watchdog's proceedings (credit: setteBIT) Specifically, Apple is being charged for misleading claims in promotional messages about how deep and how long iPhones can be submerged...
m1 chip macbook air pro

Developer Delves Into Reasons Why Apple's M1 Chip is So Fast

Monday November 30, 2020 1:57 pm PST by
Apple's M1 chip is the fastest chip that Apple has ever released in a Mac based on single-core CPU benchmark scores, and it beats out many high-end Intel Macs when it comes to multi-core performance. Developer Erik Engheim recently shared a deep dive into the M1 chip, exploring the reasons why Apple's new processor is so much faster than the Intel chips that it replaces. First and foremost,...
General cyber monday 20 sale feature

Apple Cyber Monday 2020: Discounts on iPads, Macs, AirPods, and More [Updated]

Monday November 30, 2020 6:25 am PST by
Today is Cyber Monday, a shopping event that sees many of the same deals from Black Friday bleed over into a new week, along with a few brand new offers on everything from Apple products to related accessories. In this post we'll highlight the best online discounts that you can find on Apple devices today. Note: MacRumors is an affiliate partner with some of these vendors. When you click a...
16 inch MBP Mini Led

Mini-LED M1 MacBook Pro and Mini-LED iPad Pro Models Coming First Half of 2021

Monday November 30, 2020 2:24 am PST by
Apple is widely reported to be embracing mini-LED display backlighting technology for some products next year, and a new report today by DigiTimes has named several of Apple's partners in the supply chain that are expected to benefit from the switch. According to the report, Apple is set to launch its first mini-LED iPad Pro in the first quarter of 2021 and mass produce mini-LED MacBook Pro...
Mac Mini 2018

Apple Developers Now Able to Natively Run macOS Within AWS With Amazon EC2 Mac Instances

Monday November 30, 2020 9:01 pm PST by
As AWS re:Invent kicks off, Amazon Web Services today announced new Mac instances for Amazon Elastic Compute Cloud, allowing AWS customers to run on-demand macOS workloads in the AWS cloud for the first time. Amazon says that the new feature extends the flexibility, scalability, and cost benefits of AWS to all Apple developers as those creating apps for iPhone, iPad, Mac, Apple Watch, Apple...
periscope iPhone feature2

Apple Moving Forward With Plans for 'Folded' Periscope Cameras to Significantly Increase iPhone's Optical Zoom

Monday November 30, 2020 8:01 am PST by
Apple is advancing its plans to significantly enhance the iPhone's optical zoom range using periscope lens technology, according to ETNews. Citing industry sources familiar with the development of iPhone cameras, ETNews report says Apple is currently seeking an appropriate "folded" periscope camera solution for the iPhone and has already initiated discussions with suppliers. Apple has...
apple fitness plus instructor kim ngo

Apple Fitness+ Promotion Ramps Up as Launch Approaches

Sunday November 29, 2020 11:57 am PST by
Promotion for Apple Fitness+ is ramping up, with advertising initiatives and teases from instructors being spotted on social media, indicating that the new subscription service may be launching imminently. MacRumors reader Chris Weaver spotted that Best Buy has started advertising six-month free trials of Apple Fitness+ with Apple Watch purchases. The advert suggests that retailers have...
iphone 12 colors

iPhone 12 Colors: Deciding on The Right Color

Thursday November 5, 2020 8:35 am PST by
The iPhone 12 and iPhone 12 Pro arrived last month in a range of color options, with entirely new hues available on both devices, as well as some popular classics. The 12 and 12 Pro have different color choices, so if you have your heart set on a particular shade, you might not be able to get your preferred model in that color. iPhone 12 mini and iPhone 12 The iPhone 12 mini and iPhone 12 are ...
mac mini macbook pro macbook air m1

Notable Developer Starts Patreon to Fund Apple Silicon Linux Port

Monday November 30, 2020 4:23 pm PST by
Developer Hector Martin, who describes himself as someone who "likes putting Linux on things," has launched a plan create a Linux port for Apple Silicon Macs. Martin, also known as "Marcan," has created a Patreon with the goal of earning enough funding to be able to take on the complexities of bringing Linux to Macs equipped with an M1 chip. Creating a Linux port is no easy task, and Linux...
leather sleeve iphone 12 box

Hands-On With the Leather Sleeve for iPhone 12 Models

Monday November 30, 2020 2:13 pm PST by
Apple this year came out with a handful of accessories for the new iPhone 12 models, and in addition to the standard Silicone and Leather cases, there's also a new Leather Sleeve that recently went on sale. Subscribe to the MacRumors YouTube channel for more videos. We picked up one of the Leather Sleeves to check it out to see if it's worth the $129 that Apple is charging. The Leather Sleeve ...