Apple's Work on Mixed Reality Headsets Revealed in New Patent Filings

A number of Apple patent applications, published earlier today, appear to be directly related to its long-rumored mixed reality headset, covering a range of aspects including design elements, lens adjustment, eye-tracking technology, and even software.

headset patent document software

The patents, filed with the U.S. Patent and Trademark Office by Apple, were made public earlier today and seemingly relate to specific features of its mixed-reality headset product.

Firstly, Apple applied for a patent related to several of the design elements of a head-mounted display unit in a filing titled "Head-Mounted Display Unit With Adaptable Facial Interface." This filing attempts to explain how a number of individual design elements can prevent a headset from being moved around by facial movements, increase the ability of a user to move their face when wearing a headset, and improve general comfort.

headset patent design

In supporting a headset on both the upper and lower facial regions separately, Apple seeks to reduce tension and facial compression for a more comfortable fit. This is achieved with "a light seal that conforms to the face of the user," which also blocks out environmental light, various facial supports of different stiffnesses, and even a "sprung" lower section. Among the recent deluge of reports surrounding Apple's mixed reality headset was The Information's remark that Apple is using a supportive "mesh material" around the headset for comfort.

In another application published today titled "Electronic Device With A Tunable Lens," Apple describes a lens-adjustment system for a head-mounted display unit. To present content optimally to a specific wearer, the optical lenses inside a VR/AR headset usually have to be adjusted.

headset patent lens adjustment

Apple's system for adjusting lenses involves using a first and second lens element "separated by a liquid-filled gap with an adjustable thickness." In modulating how much fluid is allowed in this gap, the headset is able to move the lenses closer together or further apart to be suitable for a specific user. Unlike many other VR headsets, which require users to manually move lenses, Apple's system is entirely electronic and controlled by actuators. Lens elements may also be "semi-rigid" to be able to adjust their curvature as needed.

Earlier this month, The Information claimed that Apple's headset would feature "advanced technology for eye tracking." Now, a new patent application from Apple simply titled "Eye Tracking System," outlines a process to detect the position and movement of a user's eyes in a head-mounted display unit.

The eye tracking system includes at least one eye tracking camera, an illumination source that emits infrared light towards the user's eyes, and diffraction gratings located at the eyepieces. The diffraction gratings redirect or reflect at least a portion of infrared light reflected off the user's eyes, while allowing visible light to pass. The cameras capture images of the user's eyes from the infrared light that is redirected or reflected by the diffraction gratings.

Infrared cameras and a light source can be placed within the headset behind the lenses to detect infrared light bounced off of a user's eyes. The "diffraction grafting," placed between the camera and a user's eye, may take the form of a thin holographic film laminated to the lens, and serves to direct the light from a user's eye directly to the camera, while allowing visible light from the headset's display to pass through as normal. The IR camera would need to be placed at the edges of the display panels, near to a user's cheekbones.

headset patent eye tracking

The filing goes on to describe in detail how this system allows the headset to precisely locate and track the movement of a user's "point of gaze." The system is so accurate that it is even able to detect pupil dilation. In terms of software applications for eye tracking, Apple remains vague, but it does suggest that the technology could be used for "gaze-based interactions" such as creating eye "animations used in avatars in a VR/AR environment."

The Information said that Apple's mixed reality headset will use "cameras on the device, the headset will also be able to respond to the eye movements and hand gestures of the wearer." It is also of note that in 2017, Apple acquired SensoMotoric Instruments, a German firm that made eye-tracking technology for VR headsets. One purpose suggested by The Information was as follows:

Apple has for years worked on technology that uses eye tracking to fully render only parts of the display where the user is looking. That would let the headset show lower-quality graphics in the user's peripheral vision and reduce the device's computing needs, according to people with knowledge of the efforts.

Finally, Apple has revealed a user interface concept for "a virtual reality and/or augmented reality device," in a filing called "3D Document Editing System." The patent application sets out how documents can be edited in a virtual 3D space. The system involves a keyboard being paired with the headset via Bluetooth or a wired connection to edit text.

The VR device may be configured to display a 3D text generation and editing GUI in a virtual space that includes a virtual screen for entering or editing text in documents via a keypad of the input device. Unlike conventional 2D graphical user interfaces, using embodiments of the 3D document editing system, a text area or text field of a document can be placed at or moved to various Z-depths in the 3D virtual space.

The filing adds that the headset may be able to detect a user's gestures to move selected content in a document such as highlighted text, shapes, or text boxes, and explains that users could move document elements in three dimensions on a Z-axis. It goes on to list a number of specific finger gestures to perform specific actions in a 3D text editor application.

In some embodiments, a document generated using the 3D text editing system may be displayed to content consumers in a 3D virtual space via VR devices, with portions of the document (e.g., paragraphs, text boxes, URLs, sentences, words, sections, columns, etc.) shifted backward or forward on the Z axis relative to the rest of the content in the document to highlight or differentiate those parts of the document. For example, active text fields or hot links such as URLs may be moved forward relative to other content in the document so that they are more visible and easier to access by the consumer in the 3D virtual space using a device such as a controller or hand gestures.

headset patent document software 2

The patent application also proposes that the headset could pass through a view of the user's real-world environment in the document editing software.

In some embodiments, the VR device may also provide augmented reality (AR) or mixed reality (MR) by combining computer generated information with a view of the user's environment to augment, or add content to, a user's view of the world. In these embodiments, the 3D text generation and editing GUI may be displayed in an AR or MR view of the user's environment.

This is similar to The Information's claim that the "cameras on the device will be able to pass video of the real world through the visor and display it on screens to the person wearing the headset, creating a mixed-reality effect," and the general parity of these patent applications to recent rumors about Apple's mixed reality headset is striking. While patent applications cannot be taken as evidence of the exact technologies Apple is planning to bring to consumer products, it is difficult to look past the way in which these patents fit into the bigger picture surrounding Apple's AR/VR project.

Analyst Ming-Chi Kuo has said that Apple will reveal an augmented reality device this year, and according to JP Morgan, the device will launch in the first quarter of 2022. The headset is expected to be priced around $3,000, competing with the likes of Microsoft's HoloLens 2, which costs $3,500. Yesterday, designer Antonio De Rosa shared photorealistic renders of what Apple's mixed reality headset is believed to look like based on recent rumors.

Related Roundup: Apple Glasses
Tag: Patent

Top Rated Comments

Mr. Dee Avatar
20 months ago
I saw a guy at the beach last summer using a oculus rift. It was the most ridiculous thing I ever saw. It was like watching the Macarena and a seizure at the same time.
Score: 4 Votes (Like | Disagree)
TheYayAreaLiving ? Avatar
20 months ago
It's happening!!!
Score: 3 Votes (Like | Disagree)
Duane Martin Avatar
20 months ago
For approximately the first 10 years after motion pictures (film) was invented almost everything that was shot were Actualities ('https://www.loc.gov/collections/edison-company-motion-pictures-and-sound-recordings/articles-and-essays/history-of-edison-motion-pictures/overview-of-the-edison-motion-pictures-by-genre/'); essentially a single set up watching an event take place. There is even an apocryphal story of Edison hiring and training 200 cinematographers on a 2 year contract because after 2 years they would have shot everything there was to shoot. People had no idea what could be done with the new medium.

That is where I feel we are with AR/VR. We simply have not imagined the use cases yet and we will not be able to until powerful and useable tools get into more hands. That happens to be something Apple is good at. Whether they will succeed here we cannot yet know but I, for one, will not be betting against them.
Score: 2 Votes (Like | Disagree)
AngerDanger Avatar
20 months ago

In terms of software applications for eye tracking, Apple remains vague, but it does suggest that the technology could be used for "gaze-based interactions" such as creating eye "animations used in avatars in a VR/AR environment."
It'd be cool if Apple used eye-tracking to dynamically render objects the user is looking towards at a high resolution and the objects in their peripheral at a lower, blurrier res.

That way, the effective resolution of the displays could be much higher while the computing power needed to drive them could be lower.
Score: 1 Votes (Like | Disagree)
iapplelove Avatar
20 months ago
I’m failing to get excited over this. Maybe I need a pill.
Score: 1 Votes (Like | Disagree)
717 Avatar
20 months ago
I think this is really exciting, even sober
Score: 1 Votes (Like | Disagree)

Related Stories

apple ar headset concept 2

Apple's 'realityOS' for Rumored AR/VR Headset Referenced in Source Code

Wednesday February 9, 2022 2:12 am PST by
New references to "realityOS," the AR/VR operating system that will run Apple's rumored mixed reality headset, have been found in App Store upload logs and Apple open source code this morning. Concept render based on purported leaked information by Ian Zelbo Apple is working on at least two AR projects that include an augmented reality headset set to be released in late 2022 or 2023, followed ...
apple view concept right corner

First Apple Mixed Reality Headset Rumored to Focus on Gaming, Media, and Communication

Sunday December 5, 2021 9:34 am PST by
Apple's rumored augmented reality (AR) and virtual reality (VR) headset will focus on gaming, media consumption, and communication, according to Bloomberg's Mark Gurman. Concept render of Apple's rumored AR headset by Antonio De Rosa Gurman, who often reveals accurate insights into Apple's plans, outlined the broad use-cases that users can expect from the company's mixed reality headset in...
apple mixed reality headset mockup feature orange

Apple's Upcoming AR/VR Headset to Require Connection to iPhone

Thursday September 2, 2021 10:08 am PDT by
The first AR/VR headset that Apple has in development will need to be wirelessly tethered to an iPhone or another Apple device to unlock full functionality, reports The Information. It will be similar to the WiFi-only version of the Apple Watch, which requires an iPhone connection to work. The headset is meant to wirelessly communicate with another Apple device, which will handle most of the ...
apple ar headset concept 2

Apple's Rumored $3,000+ AR/VR Headset Said to Enter Mass Production in August-September

Tuesday February 22, 2022 12:49 am PST by
Apple's first AR/VR headset will enter mass production this fall with a launch taking place before the end of the year, according to industry sources cited today in a report from DigiTimes. Concept render based on purported leaked information by Ian Zelbo Yesterday, DigiTimes previewed a report stating that Apple's headset has completed production tests, and now, in the full report published...
apple mixed reality headset mockup feature yellow

Apple AR Headset 'Approaching Liftoff' as Development Mirrors Period Before Apple Watch Launch

Tuesday November 16, 2021 8:20 am PST by
Apple's development of its rumored augmented reality (AR) headset is beginning to mirror the period before the launch of the Apple Watch, according to Morgan Stanley analysts. In a note to investors, seen by Investor's Business Daily, Morgan Stanley analysts explained that after years of building expertise and a number of setbacks, Apple's long-awaited AR headset is "approaching liftoff."...
apple mixed reality headset mockup feature orange

Apple's Upcoming AR/VR Headset to Feature Three Displays

Tuesday January 4, 2022 10:29 am PST by
Apple's mixed reality headset that's set to launch in 2022 will be equipped with three displays, according to a research report shared today by display analyst Ross Young. The display configuration will include two micro OLED displays along with one AMOLED panel, with Sony set to supply the micro OLED displays that Apple will use. The micro OLED displays will be the main displays for the...
apple ar headset concept 1

Artistic Renders Offer Best Look Yet at Apple's Upcoming AR/VR Headset

Tuesday December 21, 2021 7:05 am PST by
As 2022 is shaping up to finally be the year Apple announces its long-awaited and rumored AR/VR headset, a new set of artistic renders have offered us our best look yet at what Apple's first headset will look like, based on credible reports citing sources from within the company. The renders, created by concept maker Ian Zelbo, are based on details provided in a report from The Information...
apple view concept right corner

Apple Planning to Replace the iPhone With AR Headset in 10 Years

Wednesday December 1, 2021 2:29 am PST by
Apple is planning to replace the iPhone with an augmented reality (AR) headset in 10 years, a process that is apparently due to start as soon as next year with the launch of a head-mounted device, according to a recent report. Concept render of Apple's rumored AR headset by Antonio De Rosa In a note to investors seen by MacRumors, eminent analyst Ming-Chi Kuo explained that "Apple's goal is...

Popular Stories

Apple Watch Series 7 Starlight Midnight

Standard Apple Watch Series 8 Rumored to Feature Same Design as Series 7

Friday August 5, 2022 7:46 am PDT by
The standard 41mm and 45mm models of the Apple Watch Series 8 will feature the same design as the Apple Watch Series 7, according to Twitter user @ShrimpApplePro, who was first to reveal that iPhone 14 Pro models would feature a new pill-and-hole display. Titanium will not be an option for the standard Apple Watch Series 8 models either, according to @ShrimpApplePro, but Bloomberg's Mark...
cook sept 2020 event

Gurman: Apple Preparing Pre-Recorded iPhone 14 and Apple Watch Series 8 Event

Sunday August 7, 2022 6:13 am PDT by
Apple has "started to record" its virtual September event, where it's expected to announce the upcoming iPhone 14 lineup, the Apple Watch Series 8, and a new "rugged" Apple Watch model, according to Bloomberg's Mark Gurman. Writing in his latest Power On newsletter, Gurman says the event, which is expected to take place in the early part of September, is already under production, implying...
iPhone 14 Pro Purple Front and Back MacRumors Exclusive

Five iPhone 14 Rumors You May Have Missed

Thursday August 4, 2022 6:05 am PDT by
With August upon us, the countdown is officially on. We're just weeks away from when we're expecting Apple to announce the iPhone 14 lineup. Rumors of the next iPhone start early in the year, and as a result, some details about the upcoming device sometimes get lost in the crowd. Exclusive MacRumors iPhone 14 Pro renders by graphic designer Ian Zelbo To help MacRumors readers, we've created a ...
banish safari app pop ups

New iOS App Blocks Those Annoying 'Open in App' Pop-Ups in Safari

Friday August 5, 2022 2:47 am PDT by
You've probably experienced visiting a website like Reddit or LinkedIn on your iPhone only to be greeted with an annoying, almost full-screen pop-up urging you to view the content in their app instead of on the website. It's a common practice for websites that have accompanying iOS apps to push users to open (if they already have the app installed) or download their app from the App Store to ...
top stories 7aug22

Top Stories: iPadOS 16 Delayed, iPhone 14 Pro Rumors, Studio Display Speaker Issues

Saturday August 6, 2022 6:00 am PDT by
The big Apple news this week was word that the upcoming iPadOS 16 update apparently won't be arriving alongside its counterpart update for the iPhone in September, largely due to a need to continue refining the new Stage Manager multitasking feature. Other popular stories this week included more hints about the iPhone 14 Pro's rumored always-on display, potential design leaks for the...