Apple Patents Detail AR/VR System Suitable for Smartphones, Explore Object Recognition Challenges

by

Apple was granted a pair of patents on Tuesday that depict a mobile augmented reality system that can detect objects in the surrounding environment and overlay them with virtual information.

Picked up by AppleInsider, the first patent published by the U.S. Patent and Trademark Office is titled "Wearable information system having at least one camera" and describes an AR device with intelligent object recognition capabilities.

AR patent
Combining cameras, a screen, and a user interface, the device is described as being ideal for a head-mounted display, but the patent also suggests a use for the system in future smartphones, owing to the power efficient way in which it monitors the environment.

The device is described as having a default low-power scanning mode for normal operation, with high-power modes activated for short periods – when downloading and displaying AR content, for example. The rest of the patent gets quite technical as it details methods of optical tracking and determining camera orientation, two areas of AR that will prove particularly difficult to perfect.

Detecting, matching, and describing environmental features quickly and accurately are two challenges that Apple suggests could be overcome using a combination of dedicated hardware and pre-learned data. One aspect of such a system would use depth detection, achieved via dual-lens cameras similar to those found in the iPhone 7 Plus, which also utilizes depth mapping algorithms to produce Portrait Mode photos.

Apple AR patent
The second patent, titled "Method for representing virtual information in a real environment", details a way of labeling points in a topological real-world space while taking in environmental features such as object occlusion as the user moves around.

In one example, a user views an AR city map on a transparent display – or on a smartphone screen feeding a live image from a rear-facing camera – and pulls up information relating to nearby buildings or landmarks.

The patent goes on to suggest using geometry models, depth-sensing, positioning data and other advanced technologies to compensate for shortcomings in current visual modeling systems. The second patent also relies on a dual-camera system for creating a depth map of the immediate environment onto which the virtual data is superimposed.

The described technologies may or may not appear in future Apple products, but it's worth noting that both patents were originally filed for by German AR specialist Metaio just before Apple acquired the firm in 2015. Existing staff would have likely joined Apple's team dedicated to AR/VR research, to help explore how the technologies can be incorporated into future products.

Several times over the last year, Tim Cook has spoken of Apple's deep interest in augmented reality. "AR can be really great. We have been and continue to invest a lot in this. We're high on AR in the long run," he said in July.

Related Roundup: Apple Glasses
Tag: patent

Top Rated Comments

(View all)
Avatar
45 months ago

That's right, one more device to charge every night. :p

Well, my 6-port USB hub still has 2 ports available...
Score: 2 Votes (Like | Disagree)
Avatar
45 months ago
Why is the dude in the first graphic touching himself.
Score: 1 Votes (Like | Disagree)
Avatar
45 months ago
I'm excited for the future.
Score: 1 Votes (Like | Disagree)

Top Stories

Apple Warns Against Closing MacBooks With a Cover Over the Camera

Friday July 10, 2020 11:12 am PDT by
Apple this month published a support document that warns customers against closing their Mac notebooks with a cover over the camera as it can lead to display damage. Image via Reddit Apple says that the clearance between the display and the keyboard is designed to very tight tolerances, which can be problematic. Covering the camera can also cause issues with automatic brightness and True Tone....

iPhone Users Who Experienced 'Batterygate' Can Now File to Receive Around $25 Settlement From Apple

Monday July 13, 2020 6:50 am PDT by
Earlier this year, Apple agreed to settle a U.S. class action lawsuit that accused the company of "secretly throttling" older iPhone models. Now, eligible iPhone owners are beginning to be notified about their legal rights and options. Under the proposed settlement, Apple will provide a cash payment of approximately $25 to each eligible iPhone owner who submits a claim, with its total payout ...

Possible 'iPhone 12' Battery Certifications Suggest Lower Capacities Than iPhone 11 Series

Monday July 13, 2020 4:22 am PDT by
MySmartPrice has spotted certifications for three new Apple batteries that it believes could be for the upcoming iPhone 12 lineup, despite them being less capacitive than the batteries in the current iPhone 11 series. The batteries are identified with the model numbers A2471, A2431, and A2466, and appear on Safety Korea, China's 3C, and the Danish agency UL Demko. Apple is expected to...

Google to 'Dramatically' Improve Chrome Impact on Mac Battery Life

Sunday July 12, 2020 1:56 pm PDT by
Google will address long-standing battery life issues, particularly on Mac devices, reports The Wall Street Journal. Chrome will improve "tab throttling" by better prioritizing active tabs and limiting resource drain from tabs open in the background. This is said to have a "dramatic impact on battery and performance." Google has reportedly been performing early tests on Mac laptops in...

Arm-Intel-PowerPC Universal Binaries Are Possible

Saturday July 11, 2020 1:42 pm PDT by
Casual MacRumors visitors may not realize that we have a very active PowerPC forum where users discuss issues related to PowerPC Macs that have not been produced since 2006. Threads range from hardware upgrades and software options to nostalgia: Photo by AphoticD Apple's recently announced transition to Apple Silicon (Arm) based Macs raised some interesting questions about future support...

Apple Pays Samsung an Estimated $950 Million for Missing OLED Panel Purchase Targets

Monday July 13, 2020 10:03 am PDT by
Apple in the second quarter of 2020 paid Samsung approximately $950 million for not meeting OLED panel purchase goals established in agreements between the two companies, according to display analysts at Display Supply Chain Consultants. Samsung last week shared guidance on revenue and operating profit for the second quarter of 2020, which included a one-time gain related to its display...

Apple Shares Humorous 'Working-From-Home Thing' Video

Monday July 13, 2020 9:31 am PDT by
Apple today shared a funny video focused on the problems that people working from home have to deal with, including noisy children, chaotic schedules, communication issues, and more. The video focuses on showing off Apple products and their capabilities that can be useful when working from home, such as the ability to scan a document with an iPhone, mark up a PDF, Siri Reminders, and more.The...

SoftBank Considering Possible Sale of Arm Holdings as Apple Gears Up for Arm-Based Macs

Monday July 13, 2020 2:00 pm PDT by
SoftBank, the company that owns chip designer Arm Holdings, is exploring options that include a full or partial sale or a public offering, reports The Wall Street Journal. SoftBank is working with Goldman Sachs Group as an advisor, and the explorations are at an early stage. The Wall Street Journal says that it's unknown how much interest there would be in Arm from financial or industry...

Rumor Suggests New Apple App for Windows Could Be Coming Soon

Tuesday July 14, 2020 1:54 am PDT by
Apple could be working towards the release of a new app for Windows 10, according to a report this week from an Italian website. The blog Aggiornamenti Lumia suggests that an app from Apple is "coming soon" to the Microsoft Store, but stops short of providing additional details. Apple still maintains a Windows version of the iTunes app, which has been discontinued on Mac and replaced by...

Deals: Amazon Taking $400 Off 16-Inch MacBook Pro, Starting at New Low Price of $1,999.99 for 512GB

Monday July 13, 2020 5:34 am PDT by
Amazon has introduced a new low price on Apple's 16-inch MacBook Pro today, including both 512GB and 1TB models. Starting with the 512GB model (2.6GHz 6-Core, 16GB RAM), you can get this notebook for $1,999.99, down from $2,399.00. Note: MacRumors is an affiliate partner with Amazon. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running....