Apple AR/VR Projects
Apple is rumored have a secret team of hundreds of employees working on virtual and augmented reality projects.
At a Glance
- Apple is investigating multiple ways virtual and augmented reality could be implemented into future iOS devices or new hardware products. It isn't yet known when a VR or AR product will launch, but Apple's focus on the technology has ramped up over the past couple of years.
- Ultra-powerful AR/VR headset
- Augmented reality smart glasses
Apple's Secret Augmented and Virtual Reality Project
Apple has been exploring virtual reality and augmented reality technologies for more than 10 years based on patent filings, but with virtual and augmented reality exploding in popularity with the launch of ARKit, Apple's dabbling may be growing more serious and could lead to an actual dedicated AR/VR product in the not-too-distant future.
Apple is rumored to have a secret research unit comprising hundreds of employees working on AR and VR, exploring ways the emerging technologies could be used in future Apple products. VR/AR hiring has ramped up and Apple has acquired multiple AR/VR companies, suggesting something is afoot in Cupertino.
There are dozens of possibilities for VR/AR technology in Apple products, and in 2017, Apple is betting big on both AR and VR. VR support is included in Metal 2 in macOS High Sierra, and in iOS 11, Apple has developed an ARKit API that lets developers create impressive AR-based apps and games with little effort.
Along with software support for AR/VR, Apple is said to be working on multiple virtual and augmented reality headset prototypes as engineers search for the "most compelling application" for such a device.
A Powerful AR/VR Headset
According to the latest rumors, Apple is working on an incredibly powerful AR/VR headset that's not quite like anything else on the market. It is said to feature an 8K display for each eye that would be untethered from either a computer or a smartphone, and it would work with both virtual and augmented reality applications.
Rather than relying on a connection to a smartphone or a computer, the headset would connect to a "dedicated box" using a high-speed short-range wireless technology called 60GHz WiGig. The box would be powered by a custom 5-nanometer Apple processor that's "more powerful than anything currently available." At the current time, the box apparently resembles a PC tower, but it "won't be an actual Mac computer."
To use the headset, users will not need to install special cameras in a room to detect their location as with some available VR headsets. All of the technology will be built into the headset and the box.
Apple is said to be aiming to launch this AR/VR headset in 2020, but the company's plans could be delayed or scrapped.
AR Smart Glasses?
Along with rumors of a powerful AR/VR headset, there are rumors pointing towards work on augmented reality smart glasses. It's not yet entirely clear if the augmented reality smart glasses and the AR/VR headset are one and the same project or two different projects.
Rumors have described Apple's augmented reality project as "smart glasses" or an "augmented reality headset" which would be quite a bit different from the full AR/VR headset described above, but it's possible Apple's AR project evolved or that elements were left out of the original smart glasses rumors. Both projects use the code name T288, but that seems to be a blanket code name for several AR/VR prototypes.
The AR headset rumors suggest Apple is developing an AR-focused product with a dedicated display, a built-in processor, and a new "rOS" or reality operating system. rOS is said to be based on iOS, the operating system that runs on the iPhone. For the AR headset, Apple is developing a "system-on-a-package" chip similar to what's in the Apple Watch.
As for input methods, Apple is considering touch panels, voice activation, and head gestures, and a range of applications from mapping to texting are being prototyped. Virtual meeting rooms and 360-degree video playback are also concepts that are being explored.
Apple is said to be aiming to finish work on the augmented reality headset by 2019, and a finished product could be ready to ship as soon as 2020. Apple's timeline is said to be "very aggressive," though, and could change, but the hardware still has a few years of development to go.
Again, all of these rumors could potentially apply to a single AR/VR headset product, or they could be referring to two separate projects. We'll need to wait for more information for Apple's specific plans to become clearer.
In November of 2017, Apple purchased VRvana, a company that developed a mixed reality headset called Totem. VRvana's technology could potentially be used in a future Apple headset.
Rumors have also suggested Apple could incorporate its augmented reality research into its ongoing car project as part of an in-car software system that could include a heads-up display or other features.
Apple CEO Tim Cook's Comments on AR/VR
Recently, Apple CEO Tim Cook said that at the current time, the technology does not exist to create a pair of augmented reality smart glasses "in a quality way."
"But today I can tell you the technology itself doesn't exist to do that in a quality way. The display technology required, as well as putting enough stuff around your face -- there's huge challenges with that, he said. "The field of view, the quality of the display itself, it's not there yet." Cook did, however, add that most technology challenges can be solved, but it takes time.
Apple executives have shared little information on the topic of virtual reality, but Apple CEO Tim Cook extolled the virtues and benefits of AR many times in the months leading up to the launch of ARKit in September 2017.
"AR can be really great," he said in 2016. "We have been and continue to invest a lot in this. We're high on AR in the long run." Cook said he believes AR content will make the iPhone even more essential than it already is. "I think AR is big and profound. This is one of those huge things that we'll look back at and marvel at the start of it," he said.
According to Cook, augmented reality should encourage, not replace, human contact, and it should not be a "barrier." "There's no substitute for human contact," said Cook. "And so you want the technology to encourage that."
When it comes to AR vs. VR, Cook believes augmented reality is the "larger of the two" because it allows people to "be very present" while using the technology. He thinks less people would be interested in virtual reality, but admits it has interesting use cases for education and gaming.
Cook says AR is a "big idea like the smartphone," equating it to a product that's for everyone. "I think AR is that big, it's huge. I get excited because of the things that could be done that could improve a lot of lives," he said in early 2017.
Augmented Reality vs. Virtual Reality
Augmented reality (AR) and virtual reality (VR) are similar technologies, but there's a major difference between them and their potential applications vary widely. Virtual reality refers to a full immersive experience in a virtual world, while augmented reality refers to a modified view of the real world.
The difference can perhaps be best summed up by a comparison between two products, one AR and one VR. Google Glass, Google's now-defunct set of smart glasses, is an example of augmented reality. The eye-worn Google Glass let users view the world as it is, but it offered a heads-up display that overlaid relevant computer-provided information over that real world view, such as local weather, maps, and notifications.
This is similar to what Apple is said to be working on with its "smart glasses" or augmented reality headset.
In comparison, Facebook's Oculus Rift is a virtual reality headset offering an immersive visual experience that doesn't augment the real world with additional sensory information -- it fully replaces the real world with a simulated world.
Augmented reality provides computer-generated context and information about the world around us while allowing us to interact with our surroundings, while virtual reality is designed to isolate us from our surroundings so we can interact with fictitious worlds.
Potential applications for the two are vastly different. Virtual reality is singularly focused on immersive content consumption because it makes the wearer feel as if they're actually experiencing what's going on in the simulated world through visual, tactile, and audio feedback. Virtual reality is largely linked to gaming right now, but it also has the potential to recreate real world experiences for educational or training purposes.
Augmented reality doesn't hinge on immersive content and while less exciting because it's augmenting reality instead of replacing it, it has a wider range of potential applications. In fact, augmented reality apps and games are readily available in iOS 11 thanks to ARKit.
With ARKit, an iOS device is able to identify a surface like a table, and then virtual objects can be added to it. Because of the computing power of the iPhone and the iPad, ARKit's augmented reality capabilities are impressive. ARKit is already being used to create a huge range of apps and games, blending digital objects with the real world.
Apple may aim for a product that works with both augmented and virtual reality, as suggested by recent rumors indicating work on an AR/VR headset that would connect via Wi-Fi to a PC-like box that features a powerful processor. Such a product could be used for serious virtual reality applications as well as augmented reality experiences.
In macOS High Sierra, Apple introduced for VR in Metal 2 and partnering with Valve, Unity, and Unreal to bring VR creation tools to the Mac, which could hints at some kind of future hardware. At the very least, Apple's latest Macs and upcoming iMac Pro will support existing virtual reality hardware and VR content creation.
KGI Securities analyst Ming-Chi Kuo believes Apple's AirPods and its rumored high-end over-ear headphones are the future of Apple's AI and augmented reality ambitions. He believes the two audio devices will complement Apple's rumored augmented reality glasses.
Apple's VR/AR Team
Apple's work on virtual and augmented reality dates back multiple years, but rumors picked up starting in March of 2015 when news hit that Apple had a small team of people working on augmented reality. In 2015 and into early 2016, Apple's team grew as the company hired employees with expertise in AR/VR technology and made multiple related acquisitions.
Apple's AR/VR team includes several hundred engineers from across Apple, all of whom have expertise in virtual and augmented reality. The team works across office parks in both Cupertino and Sunnyvale, and Apple is exploring several hardware and software projects under the code name "T288."
Apple's augmented reality team combines "the strengths of its hardware and software veterans," and is led by Mike Rockwell, who came from Dolby. Former employees of companies like Oculus, HoloLens, Amazon (from the VR team), 3D animation company Weta Digital, and Lucasfilm are working on AR at Apple.
Apple developers have also joined the WebVR community group dedicated to make VR content viewable on any device and on the web. One of Apple's most prominent AR/VR hires was computer science professor Doug Bowman, who previously led the Virginia Tech's Center for Human-Computer Interaction. He specializes in three-dimensional user interface design and has written a book on the subject covering 3D interfaces and the benefits of immersive virtual environments. He has expertise with both virtual and augmented reality. Apple has also hired employees that have worked on virtual or augmented reality products at Microsoft and Lytro. Some recent hires are said to be from Microsoft's HoloLens team, while others worked at Lytro, a company working on a camera able to blend live action and computer graphics for a live action VR experience. Employees coming from the HoloLens team would have experience creating an advanced augmented reality headset. Zeyu Li, who served as a principal computer vision engineer at Magic Leap (a startup developing a head-mounted AR/VR display), is now working at Apple as a "Senior Computer Vision Algorithm Engineer." Yury Petrov, a former research scientist at Facebook-owned Oculus, is now serving as a "research scientist" at Apple. According to his LinkedIn profile, Petrov studied virtual reality experiences, prototyped optics, and developed computer simulation software. Augmented reality expert Jeff Norris joined Apple in April as a senior manager working on the company's augmented reality team. Norris founded the Mission Operations Innovation Office and JPL Ops Lab at NASA. He led multiple projects focused on human-system interaction with an emphasis on virtual and augmented reality. With Apple's team encompassing hundreds of employees, there are many other virtual reality expert hires that have gone under the radar. On LinkedIn, there are multiple software engineers with virtual reality experience that are employed by Apple, but it is unclear if they work on the secret AR/VR team. Many members of Apple's AR/VR team may have joined the company though acquisitions. Since 2015, Apple has purchased several companies that created AR/VR-related products, and some of its AR/VR acquisitions even date back several years. In November of 2017, Apple purchased VRvana, a company that developed a mixed reality headset called Totem. Totem, which was never released to the public, was designed to combine both augmented and virtual reality technologies in a single headset, merging full VR capabilities with pass-through cameras to enable screen-based augmented reality features. Totem essentially used a set of cameras to project real world images into its built-in 1440p OLED display, a somewhat unique approach that set it apart from competing products like Microsoft's HoloLens, which uses a transparent display to combine virtual and augmented reality. Apple could be planning to use some of Totem's technology in a future product. Apple purchased Israeli-based 3D body sensing firm PrimeSense in 2013, sparking speculation that motion-based capabilities would be implemented into the Apple TV. PrimeSense's 3D depth technology and motion sensing capabilities were used in Microsoft's initial Kinect platform.
Apple developers have also joined the WebVR community group dedicated to make VR content viewable on any device and on the web.
One of Apple's most prominent AR/VR hires was computer science professor Doug Bowman, who previously led the Virginia Tech's Center for Human-Computer Interaction. He specializes in three-dimensional user interface design and has written a book on the subject covering 3D interfaces and the benefits of immersive virtual environments. He has expertise with both virtual and augmented reality.
Apple has also hired employees that have worked on virtual or augmented reality products at Microsoft and Lytro. Some recent hires are said to be from Microsoft's HoloLens team, while others worked at Lytro, a company working on a camera able to blend live action and computer graphics for a live action VR experience. Employees coming from the HoloLens team would have experience creating an advanced augmented reality headset.
Zeyu Li, who served as a principal computer vision engineer at Magic Leap (a startup developing a head-mounted AR/VR display), is now working at Apple as a "Senior Computer Vision Algorithm Engineer."
Yury Petrov, a former research scientist at Facebook-owned Oculus, is now serving as a "research scientist" at Apple. According to his LinkedIn profile, Petrov studied virtual reality experiences, prototyped optics, and developed computer simulation software.
Augmented reality expert Jeff Norris joined Apple in April as a senior manager working on the company's augmented reality team. Norris founded the Mission Operations Innovation Office and JPL Ops Lab at NASA. He led multiple projects focused on human-system interaction with an emphasis on virtual and augmented reality.
With Apple's team encompassing hundreds of employees, there are many other virtual reality expert hires that have gone under the radar. On LinkedIn, there are multiple software engineers with virtual reality experience that are employed by Apple, but it is unclear if they work on the secret AR/VR team.
Many members of Apple's AR/VR team may have joined the company though acquisitions. Since 2015, Apple has purchased several companies that created AR/VR-related products, and some of its AR/VR acquisitions even date back several years.
In November of 2017, Apple purchased VRvana, a company that developed a mixed reality headset called Totem. Totem, which was never released to the public, was designed to combine both augmented and virtual reality technologies in a single headset, merging full VR capabilities with pass-through cameras to enable screen-based augmented reality features.
Totem essentially used a set of cameras to project real world images into its built-in 1440p OLED display, a somewhat unique approach that set it apart from competing products like Microsoft's HoloLens, which uses a transparent display to combine virtual and augmented reality. Apple could be planning to use some of Totem's technology in a future product.PrimeSense
Apple purchased Israeli-based 3D body sensing firm PrimeSense in 2013, sparking speculation that motion-based capabilities would be implemented into the Apple TV. PrimeSense's 3D depth technology and motion sensing capabilities were used in Microsoft's initial Kinect platform.
PrimeSense uses near-IR light to project an invisible light into a room or a scene, which is then read by a CMOS image sensor to create a virtual image of an object or person. This enables motion-based controls for software interfaces, but it's also able to do things like measure virtual objects and provide relative distances or sizes, useful for augmented reality applications like interactive gaming, indoor mapping, and more. PrimeSense technology can also create highly accurate 360 degree scans of people and objects, potentially useful for virtual reality applications.Metaio
Apple acquired augmented reality startup Metaio in May of 2015. Metaio built a product called the Metaio Creator, which could be used to create augmented reality scenarios in just a few minutes. Prior to being purchased by Apple, Metaio's software was used by companies like Ferrari, who created an augmented reality showroom.
Metaio technology was also used in Berlin to allow people visiting the site of the Berlin Wall to use a smartphone or tablet to see what the area looked like when the Berlin Wall was still standing. Metaio's technology is one that could potentially be used to implement augmented reality capabilities into Apple apps like Maps.Faceshift
Apple acquired Faceshift in August of 2015, marking its second augmented reality purchase in 2015. Before being acquired by Apple, Faceshift worked with game and animation studios on technology designed to quickly and accurately capture facial expressions using 3D sensors, transforming them into animated faces in real time. Faceshift was also working on a consumer-oriented product that would allow people to morph their faces into cartoon or monster faces in real time in Skype.
Faceshift's technology has a wide range of possible use cases, and Apple appears to be using the feature to power Animoji in the iPhone X.Emotient
Emotient, a company that built tools for facial expression analysis, was acquired by Apple in January of 2016. Emotient's technology uses artificial intelligence and machine learning to read human emotion, features that have been used in the real world by advertisers to determine emotional reactions to advertisements.
There are dozens of things Apple could do with Emotient, ranging from better facial detection in the Photos app to analyzing customer feelings in Apple retail stores to unlocking iOS devices, but it also has potential AR/VR uses. Like Faceshift, Emotient's technology could be used to analyze and transform facial expressions for the creation of virtual avatars, useful for social media purposes and games. Emotient technology was likely used for Animoji.Flyby Media
Purchased in early 2016, Flyby Media is another company that worked on augmented reality. Flyby created an app that worked with Google's 3D sensor-equipped "Project Tango" smartphone, allowing messages to be attached to real world objects and viewed by others with one of Google's devices.
For example, a person could "scan" a landmark like San Francisco's Golden Gate Bridge and write a message attached to it. A person visiting the bridge later would then be able to scan the bridge with the Flyby app to see the message. The Flyby app likely drew the attention of Apple because it was able to recognize and understand different objects that were scanned, technology that could be used by Apple in a number of ways in apps like Photos and Maps.RealFace
In February of 2017, Apple purchased RealFace, a cybersecurity and machine learning company that specializes in facial recognition technology, which could potentially be used for future augmented reality features.
RealFace developed facial recognition technology integrating artificial intelligence for frictionless face recognition. Realface technology was likely employed in the iPhone X, Apple's first smartphone with facial recognition capabilities in the form of Face ID.
Several of Apple's competitors are working on virtual or augmented reality projects, which is another factor that's likely renewed Apple's interest in the matter. Many competing technology companies have already launched VR/AR products or are planning to launch in the near future putting development ahead of whatever Apple may be working on.
Facebook - Oculus
Oculus Rift is one of the first virtual reality headsets available for consumers. It was released at the end of March 2016, and is priced competitively at $599. The Oculus Rift debuted on Kickstarter in 2013 and has been under development since then, attracting a lot of attention and bringing virtual reality into the spotlight.
Worn over the head, the Oculus Rift has two 1080 x 1200 resolution OLED panels for each eye, both with a 90Hz refresh rate for an immersive visual experience that isn't subjected to motion blurring or choppiness. It includes a gamepad, external infrared tracking sensors to track where a user's head is positioned, and optional Oculus Touch motion controllers that can be tracked in 3D space with the sensors.
The Oculus Rift is subject to some technical limitations. It is not a standalone device, requiring a connection to a Windows PC equipped with a powerful GPU, and it is targeted at the gaming market. Software must be specially created for the Oculus Rift using the Oculus SDK. Oculus has said that it cannot bring the Rift to Macs because Apple does not prioritize high-end GPUs. Current Macs cannot support the Oculus Rift.
Oculus Rift users are also be able to watch both conventional movies and 360 degree videos, and there are social-oriented apps in development as Oculus is now owned by Facebook. Virtual worlds where people can interact with one another socially are one of the areas where there's significant general consumer potential for devices like the Oculus Rift.
Facebook in May of 2018 launched the Oculus Go, a $199 virtual reality headset that operates on a standalone basis without the need for a connection to a smartphone or a computer.
Design wise, the Oculus Go looks similar to the Oculus Rift. It's a VR headset that fits over the eyes and attaches to the head with adjustable straps and breathable fabrics. It is a standalone device, which means it does not require a connection to a computer or a gaming system to operate.
Oculus Go features a 5.5-inch display with a 2560 x 1440 resolution (1280 x 1140 per eye), a Qualcomm, Snapdragon 821 processor, and built-in spatial audio drivers. It shipped with more than 1,000 available apps.
Microsoft - HoloLens
Microsoft's HoloLens is similar to Google Glass, with a head-worn pair of glasses that resemble safety goggles. The HoloLens incorporates a stereoscopic 3D display, spatial sound, a microphone array, a video camera, and sensors like an accelerometer, gyroscope, and magnetometer. It runs the Windows Holographic operating system, which allows people to interact with the HoloLens through gestures, eye movement, and voice.
The HoloLens combines both virtual and augmented reality, creating virtual objects Microsoft calls "holograms" that can be viewed and manipulated alongside real-world objects. Microsoft has showcased multiple use cases for the HoloLens, including an interactive version of Skype, an augmented reality first person shooter, an augmented reality version of Minecraft, a 3D modeling application for creating objects to be 3D printed, and human anatomy lesson.
Microsoft's HoloLens has been attracting a lot of attention because of the way that it mixes virtual reality and augmented reality, going beyond typical augmented reality applications but without losing the benefit of real-world interaction.
Microsoft has also developed the Windows Mixed Reality platform, which is essentially a virtual reality platform for developers and hardware makers. Microsoft has partnered with Acer, Dell, HP, Lenovo, and Samsung to create headsets for the device, and there are a range of them now available at prices starting at $329.
Samsung has developed the $499 HMD Odyssey headset for Windows Mixed Reality, offering high-resolution dual 3.5-inch AMOLED displays with a 110-degree field of view and built-in AKG headphones for 360 degree spatial sound to create an immersive experience.
Google - Cardboard
Google Cardboard is an ultra simple virtual reality implementation, building a cardboard viewer around an existing smartphone. Given its simplicity, the Google Cardboard isn't likely a product Apple will emulate, but it does raise the possibility of an accessory that could be used alongside the iPhone to enable virtual reality experiences.
Google Cardboard headsets are constructed from low-cost components and require a compatible app to split an iPhone's display into two to create a stereoscopic 3D image. As an open platform, Google Cardboard has been used by multiple companies to build apps with 3D experiences and games. Other features include a photography app for taking 360 degree photos and an educational component for sending students on virtual field trips.
Google is said to be working on a second-generation Cardboard headset, along with a full standalone virtual reality headset that would be similar in design to the Oculus Rift but without the need for a connection to a computer.
PlayStation VR - PlayStation VR is similar to the Oculus Rift with a full head-mounted display that works with the PlayStation 4 console. The PS VR includes a 1920 x 1080 120fps OLED display with a wide field of view, stereoscopic 3D, and head-tracking capabilities. It was released in the fall of 2016.
Samsung Gear VR - Developed in partnership with Oculus, the Gear VR is similar to the Google Cardboard, incorporating a Samsung Galaxy Note 5, Galaxy S6, Galaxy S6 Edge to serve as the display and processor. It includes custom sensors and high field of view lenses that make it more accurate than the simpler Google Cardboard.
HTC Vive - The HTC Vive is a head-mounted display HTC has developed in partnership with Valve. Like the Oculus Rift and the PlayStation VR, it's primarily aimed at gamers and works with the Steam platform. It includes two 1080 x 1200 90Hz displays in each eyepiece and it incorporates more than 70 different sensors. With an included base station, it can detect a person's movements to avoid obstacle collisions.
Apple has filed at multiple patents that relate directly to a virtual reality headset, all dating back several years. While technology has likely advanced somewhat beyond these, they provide an interesting look at the ideas Apple has explored in the past.
A 2008 patent application covered a fairly basic "personal display system" designed to mimic the experience of being in a movie theater when watching video.
A second patent described a "Head Mounted Display System" with a "laser engine" that projected images onto a clear glass display worn over the eyes, similar to glasses. In this configuration, the headset connected to a handheld video player such as an iPod to provide processing power.
A third patent originally filed for in 2008 was similar in design, covering a goggle-like video headset designed to let users watch movies and other content. It outlined two adjustable optical modules lined up with the user's eye, which could provide vision correction and allow for the viewing of 3D content. Apple described this as offering a personal media viewing experience.
A fourth patent from 2008 covered a video headset frame similar to the Google Glass, which would allow a user to slide their iPhone or iPod into the headset to provide video. The headset was described as an augmented reality product that would let users do things like watch a video or check email while keeping an eye on their surroundings.
Beyond headset-related patents, Apple has also filed for patents describing other ways virtual and augmented reality features could be implemented into its devices. A 2009 patent application, for example, covered camera-equipped 3D displays that would shift in perspective based on a user's relative position.
Such a display would detect head movement, allowing a user to move their head around to look at a 3D image from different angles while also incorporating elements of a user's environment.
2010 and 2012 patents described the use of motion sensors to create a 3D interface for iOS devices using augmented reality techniques. Apple described the interface as a "virtual room" navigated by manipulating the orientation of the device through built-in sensors or through gestures.
In 2011, Apple filed a patent for an augmented reality feature in the Maps app related to mapping the distance to notable landmarks. With the camera, a user could look at the area around them and get real-time estimations of the distance between two points along with overlays of relevant information.
A patent filed in 2014 and granted in 2017 covers a mobile augmented reality system able to detect objects in the environment and overlay them with virtual information through the use of cameras, a screen, and a user interface. Apple describes the system as ideal for a head-mounted display, but it also shows it being used in smartphones.
Apple's work on both "smart glasses" and a virtual reality headset is still in the prototype stage, so it could be years before a product launches. Apple often creates prototypes that never go on to become actual products, so there's a possibility that its virtual and augmented reality work could be shelved if the company doesn't feel that it's a worthwhile market to enter, or it could be implemented in a simpler way.
Apple is, however, currently aiming to finish work on an augmented or virtual reality product by 2019, and a finished product could be ready to ship as soon as 2020. Apple's timeline is said to be "very aggressive," though, and could change.