visionOS

visionOS is a new operating system developed for the Apple Vision Pro augmented and virtual reality headset or "spatial computing device" as Apple calls it. We know a limited amount about visionOS at the current time because it is in beta testing and is not widely available, but this roundup aggregates what we've learned from those who have access and what Apple has shared.

With visionOS, Apple provides a full three-dimensional interface that starts out with a Home View that is similar to the iPhone's Home Screen. It displays all of the apps that are installed on the Apple Vision Pro, with the visionOS interface able to respond dynamically to natural light and shadows to help users interpret distance and scale.

visionOS is navigated using eye movements, hand gestures, and voice control because it cannot be directly interacted with. Users can, for example, browse through apps by looking at them, and then select one with a tap of the fingers. This navigation scheme has been described as natural and intuitive by those who have been able to test Apple Vision Pro.

visionOS apps can be placed anywhere in virtual space visible to a user, and they can be resized to be ultra large or small. App windows can be stacked on one another, arranged around a room, and manipulated at will. Multiple apps can run at the same time, and it is unclear if there is an upper limit beyond what fits in the visual space. Apple describes visionOS as offering an infinite canvas, but users can also opt to focus on a single app that expands fully into the space.

The level of immersion in visionOS is adjustable. Users can either see their content in their actual surroundings through the Apple Vision Pro's cameras, or a virtual "Environment" can make the real world fade away. Apple designed several Environments like Mount Hood that can be activated as a backdrop in visionOS.

Apple adapted existing apps like FaceTime, Photos, Messages, Safari, and more to work in a 3D environment. In Photos, for example, images can be viewed larger than life, and panoramas will wrap around the user. In FaceTime, life-size tiles for each person can be positioned around the room. For the headset wearer, visionOS creates a Persona, or a digital representation that reflects face and hand movements in real time. In Messages, Safari, and other apps, users will be able to pull out 3D USDZ files to look at and manipulate virtual objects.

With content apps like Apple TV+, visionOS can display TV shows and movies on a personal movie screen that feels 100 feet wide, plus visionOS offers Apple Immersive Videos that are 180-degree high-resolution recordings. visionOS supports Spatial Audio through the Apple Vision Pro's speakers.

There is a dedicated App Store for visionOS, which will house apps and content from developers. It will also provide thousands of iPhone and iPad apps that will automatically run on visionOS and that will be adapted to the eye and hand controls. With iCloud, content is synced with iPhone, iPad, and Mac.

Bluetooth accessories like keyboards, mice, trackpads, and game controllers work with visionOS, and it also integrates with the Mac to allow the Apple Vision Pro to serve as a display for a Mac. The Mac's screen shows up in the virtual space provided by visionOS for a private, portable 4K display.

For security and authentication, visionOS supports Optic ID, an iris scanning function that works like Touch ID or Face ID. Optic ID can be used for logging into visionOS, making purchases, and logging into websites.

visionOS came out on Friday, February 2, the official launch date of the Vision Pro headset.

The current version of visionOS is visionOS 1.1 released in March. visionOS 1.1 adds Apple Device management to the Vision Pro, allowing IT departments at companies and educational institutions to manage Vision Pro headsets. It also includes support for iMessage Contact Key verification and it improves Personas in several ways to make them look more natural.

The visionOS 1.1 update adds support for resetting the Vision Pro on device should the user lose their passcode, and EyeSight looks better than before. There are several fixes for the virtual keyboard and Mac virtual display features.

Apple is testing visionOS 1.2, a second update to the visionOS operating system. Two betas have been released so far.

visionOS is an operating system built from the ground up for the Apple Vision Pro, and it is Apple's first spatial computing OS. It is unlike anything we've seen before because it is designed to be used in a virtual space, but at the same time, it is reminiscent of iOS. It's basically bringing digital content into your physical space.

There is an iOS-like Home View that serves as the main hub, and it is akin to the Home Screen. It's where you go to access all of the content that is available on the Vision Pro, including apps that you have installed.

Software runs through apps, as on macOS and iOS. So if you want to watch a movie, play a game, or get work done, you need to open the appropriate app. It's an all-new experience, but it will be familiar to iPhone and iPad users.

visionOS features a full three-dimensional interface and all of the app icons, windows, and other objects respond dynamically to natural light and shadows to help you figure out the distance and scale of the virtual elements in the room.

App Windows

Apps open up like they do on the Mac, and you have an adjustable app window that you can position anywhere in the space around you. You can make windows big or small, and have as many windows open as you want around you.

You basically have an "infinite canvas" to work with, as Apple puts it, because all of the space around you is at your disposal. Apps can be right in front of your face, pushed further back in the room, tiled on top of each other, or spread around in the room.

A single app can fill up the entire area in front of you if you want to focus on one activity like a meditation.

Environments

Apple's cameras are designed to show you your surroundings so you can place apps and other digital objects in the room around you. With the Digital Crown, though, you can be transported into an entirely virtual space that blocks out the world.

Turning the Digital Crown adjusts your level of immersion, activating what Apple calls "Environments." Environments are virtual backdrops that take you to another place, such as Mt. Hood. Apple has described these as full, 3D like places that make the real world fade away.

Available Environments include Haleakalā, Yosemite, Sky, Joshua Tree, Lake Vrangla, Mount Hood, The Moon, Beach, White Sands, Winter Light, Fall Light, Summer Light, and Spring Light.

Screen Sharing via AirPlay

Content that is displayed on the Apple Vision Pro can be mirrored to an iPhone, iPad, Mac, Apple TV, or AirPlay-compatible smart TV using AirPlay. Content can be mirrored to any AirPlay-enabled device at up to 720p, which allows the wearer to show people what's on the headset.

Personas

Because the Vision Pro is worn on the user's head and is not able to capture the user through an outward-facing camera for video calls and other purposes, Apple designed Personas.

Personas are a digital representation of the Vision Pro wearer, with a 3D scan captured using the Vision Pro headset while it is not worn. Personas reflect facial expressions and hand movements in real time, though some users have found them to be a bit creepy due to the uncanny valley effect. Personas are primarily used in FaceTime and other video calls.

Apple in April 2024 released Spatial Personas in a beta capacity. Spatial Personas are a version of the Personas feature that are not confined to a tile.

With Spatial Personas, there is a translucent background and a visual of the hands, allowing users to feel as if they are located in the same physical space. Using Spatial Personas requires visionOS 1.1 or later.

Controlling visionOS is done through eye movements, hand gestures, and voice commands. You basically look at something to highlight it, and then use a gesture to interact with it. The available gesture options are listed below.

• Tap - Tapping the thumb and the index finger together signals to the headset that you want to tap on a virtual element on the display that you're looking at. Users have also described this as a pinch, and it is the equivalent to tapping on the screen of an iPhone.
• Double Tap - Tapping twice initiates a double tap gesture.
• Pinch and Hold - A pinch and a hold is similar to a tap and hold gesture, and it does things like highlighting text.
• Pinch and Drag - Pinching and dragging can be used to scroll and to move windows around. You can scroll horizontally or vertically, and if you move your hand faster, you'll be able to scroll faster.
• Zoom - Zoom is one of two main two-handed gestures. You can pinch your fingers together and pull your hands apart to zoom in, and presumably zooming out will have a pushing sort of motion. Window sizes can also be adjusted by dragging at the corners.
• Rotate - Rotate is the other two-handed gesture and based on Apple's chart, it will involve pinching the fingers together and rotating the hands to manipulate virtual objects.

Gestures will work in tandem with eye movements, and the many cameras in the Vision Pro will track where you are looking with great accuracy. Eye position will be a key factor in targeting what you want to interact with using hand gestures. As an example, looking at an app icon or on-screen element targets it and highlights it, and then you can follow up with a gesture.

Hand gestures do not need to be grand, and you can keep your hands in your lap. Apple is encouraging that, in fact, because it will keep your hands and arms from getting tired from being held in the air. You only need a tiny pinch gesture for the equivalent of a tap, because the cameras can track precise movements.

What you're looking at will let you select and manipulate objects that are both close to you and far from you, and Apple does anticipate scenarios where you might want to use larger gestures to control objects that are right in front of you. You can reach out and use your fingertips to interact with an object. For example, if you have a Safari window right in front of you, you can reach your hand out and scroll from there rather than using your fingers in your lap.

Eye and gesture-based navigation is demonstrated in onboarding videos that were pulled from the visionOS software. Apple walks users through looking at a system element and selecting it with a tap of the fingers. The headset is able to detect hand movements even when the hands are resting in a comfortable position.

Developers can create custom gestures to be used with visionOS in their apps, and there are also accessibility accommodations.

There is also an air typing feature that is designed to let you type on a full virtual keyboard. There is no tactile feedback, with Apple instead focusing on visual feedback when a key is "pressed." The virtual keyboard experience has been described as a "complete write-off" that does not work well. Some reviewers have said that it's fine, while others have expressed frustration with how difficult it is to type quickly.

Input Devices

Though visionOS is meant to be controlled with hand gestures and eye movements, it does support Bluetooth mice, keyboards, and trackpads, along with game controllers for playing games.

Mice, keyboards, and trackpads can be used alongside a Mac.

Apple reimagined many of its familiar apps for a 3D interface, including FaceTime, Photos, Messages, and Safari.

FaceTime in particular is of interest because it will show the people you're conversing with in large tiles around the room, which you can adjust. People you're chatting with will see your "Persona," a scan of your real face that reflects your facial expressions and hand movements in real time so you look like yourself rather than a person wearing a headset. Personas are also able to be used in other video apps.

The Photos app shows you images in larger than life scale, with panoramas wrapping around you, for example, plus you can view 3D photos and videos. There's a built-in camera on the headset so you can capture video that you can watch back in full 3D.

With Messages, Safari, and other apps, users can pull out 3D USDZ files to look at and manipulate virtual objects. Built-in apps on the Vision Pro are listed below.

Pre-installed apps optimized for Vision Pro:

• App Store
• Capture
• Encounter Dinosaurs
• FaceTime
• Files
• Freeform
• Keynote
• Mail
• Messages
• Mindfulness
• Music
• Notes
• Photos
• Safari
• Settings
• Tips
• TV

Pre-installed apps not optimized for Vision Pro:

• Books
• Calendar
• Home
• Maps
• News
• Podcasts
• Reminders
• Shortcuts
• Stocks

Third-Party Apps

The visionOS App Store launched with 600 third-party Vision Pro-optimized apps available.

• Vision Pro Apps Worth Checking Out

iPhone and iPad Apps

There is a dedicated visionOS App Store where you can find apps created specifically for the device, but it will also allow you to download existing iPhone and iPad apps. iPhone and iPad apps will run on visionOS in a 2D environment, so you can use your favorite apps even if they're not updated for a 3D interface.

Entertainment

Apple TV+ and other streaming apps will provide a variety of TV and movies that can be watched through the headset. Apple says that the effect is like a personal movie screen that feels 100 feet wide.



Streaming services and sports apps that will be on the Vision Pro at launch include Disney+, ESPN, MLB, PGA Tour, Max, Discovery+, Amazon Prime Video, Paramount+, Peacock, Pluto TV, Tubi, Fubo, Crunchyroll, Red Bull TV, IMAX, TikTok, and MUBI, along with Apple TV+.

Other streaming services such as Netflix can be accessed via Safari or, in some cases, through iPad apps that are able to run on the Vision Pro.YouTube, Spotify, and Netflix will be limited to the browser, as developers are able to opt out of having their iPad apps accessible through the Vision Pro App Store.

Some apps that are customized for Vision Pro have unique features. Disney+, for example, allows users to select Disney-themed backgrounds for watching content. Users can choose backgrounds such as the Avengers Tower or Tatooine as their setting, with content played on a screen over the background.

Vision Pro supports both 2D and 3D movies. Disney+ and other apps will offer select movies in 3D, and will bring 3D versions of new movies alongside or shortly after 2D releases. More than 150 3D movies will be available to purchase through the Apple TV app on Vision Pro, including Avatar: The Way of Water, Dune, Spider-Man: Into the Spider-Verse, and The Super Mario Bros. Movie.

Immersive Experiences

The Apple TV app will include a selection of content recorded with the Apple Immersive Video format, which offers 180-degree 3D 8K recordings captured with spatial audio.

• Alicia Keys: Rehearsal Room - Provides a glimpse into Alicia Keys creative process through an intimate rehearsal session.
• Adventure - Follows pioneering athletes as they take on awe-inspiring challenges in some of the world's most sensational locations.
• Wild Life - Allows viewers to get up close and personal with some of the most charismatic creatures on the planet.
• Prehistoric Planet Immersive - Transports users into the daily lives of dinosaurs.

Gaming

visionOS includes support for Apple Arcade games that can function with a connected Bluetooth gaming controller, plus Apple is working with gaming partners to create new gaming experiences for the device. Select "spatial games" have been created for the Vision Pro and will be available when the device launches, including Game Room, What the Golf?, and Super Fruit Ninja.

App Store

Like Apple's other operating system updates, visionOS has its own dedicated App Store where users can find third-party apps and games designed for the device.

Possible Apple Pencil Integration

A future version of the Apple Pencil could work with the Vision Pro, as Apple has tested a version of the device that has visionOS integration. The Apple Pencil would be able to be used with Vision Pro drawing apps like Freeform and Pixelmator.

How the Apple Pencil would work with the Vision Pro is unknown, but it is possible that users would be able to draw with the Apple Pencil on a flat surface, with the drawing showing up in the visionOS app.

Visual Search

visionOS includes a Visual Search feature that can be used to get information about an item, detect and interact with text in the world around them, copy and paste printed text from the real world into apps, translate text between 17 different languages, and more.

Real world text that includes contact information, webpages, and unit conversions and similar information can be acted upon in visionOS. So, for example, if a printed handout has a website link in it, you can scan the link with the Vision Pro, opening up a Safari window to view the website. Or, if a recipe calls for grams and you need ounces, you can convert using the headset.

Real-time text translation will also be useful for traveling and other instances when you might want to quickly translate what you're seeing in the real world. The Apple Vision Pro headset will be able to automatically detect text and documents, similar to how the iPhone can detect text in photos and allow it to be interacted with.

Travel Mode

Travel Mode can be turned on through the Control Center, and it is designed to stabilize the visuals on the Vision Pro for users that are wearing the headset on an airplane.

Guest Mode

Guest Mode allows a user's family and friends to use the Vision Pro with limited access to certain apps, such as Photos and Safari. The primary user's Apple ID account, settings, Optic ID, passcode, passwords, Apple Pay, Persona, and health data will be restricted while the Guest User mode is active.

The Vision Pro integrates with the Mac, serving as a display for either a notebook or desktop machine through a feature called Mac Virtual Display. You can see all of your Mac apps in a virtual space over AirPlay 2, using a keyboard and mouse as usual to get work done.

Note that you can use the display with a Mac mini or a Mac Studio that does not have its own display. A secondary display is not required.

With Mac Virtual Display turned on, visionOS apps can be used alongside the virtual Mac screen. The Vision Pro itself can be mirrored to a Mac over AirPlay or FaceTime.

There does not appear to be a feature for using Mac apps individually in a mixed reality workspace, with separate windows for each. The Vision Pro simply serves as a display for a standard Mac desktop, so you can only operate apps within the single virtual desktop.

• Your Mac and Apple Vision Pro: What You Can Do (and What You Can't)

Rather than Face ID or Touch ID, visionOS uses Optic ID, an all new iris scanning security feature. Apple uses infrared lights and cameras inside the headset to look at the unique pattern of your iris for authentication purposes.

Optic ID can be used for logging into visionOS, making purchases, and in lieu of passwords in apps and websites.