Apple Vision Pro: First year

UPDATE: Metallica Immersive concert (recorded with 14 Apple Immersive 8K 3D cameras), Vision3D for making your room a canvas, Formula 1 real-time map (and other sports), theatrical 3D premieres in Holographic 3D, medical 3D scans, 16K Immersive Videos, online multiplayer tabletop games, world recognition for visually impaired people, 2d to 3D video player, are some of the new possibilities available for Vision Pro.

Apple Vision Pro saw major advancements from both Apple and the developer community. Added to new features, and new content, developer community found new uses and use cases, pushing the limits of the technology.

We’ll start with the community advancements that achieved to push the limits of the official capabilities.

Community breakthroughs

The Apple Vision Pro wasn’t initially designed with traditional VR gaming in mind, particularly due to its lack of dedicated motion controllers. Instead, it relies on advanced hand and eye tracking for interaction. However, the enthusiast and developer community has made significant strides in adapting it for PC-based VR gaming, including controller-free experiences, by leveraging open-source tools and innovative workarounds.

1. Streaming PC VR Content to Vision Pro

One of the biggest breakthroughs came from adapting existing VR streaming technologies to work with the Vision Pro. The open-source project ALVR (Air Light VR) has been pivotal. Originally developed for streaming SteamVR content to standalone headsets like the Meta Quest, community developers ported ALVR to the Vision Pro shortly after its release. By early 2024, ALVR was demonstrated running SteamVR on the Vision Pro, allowing users to tap into a vast library of PC VR games. This eliminated the need for native Vision Pro VR titles and opened the door to PC-powered experiences, which are typically more graphically demanding than standalone headset capabilities.

• How It Works Without Controllers: ALVR forwards the Vision Pro’s built-in hand tracking data to SteamVR. SteamVR, the dominant PC VR platform, traditionally expects input from tracked controllers, but the Vision Pro’s high-precision hand tracking—enabled by its array of cameras and sensors—can be mapped to emulate controller inputs in some cases. For games that don’t strictly require button presses or joystick movement, this allows a controller-free experience. Community tweaks to ALVR have refined this process, making hand gestures like pinches or swipes usable as basic inputs.

2. Hand Tracking Integration

The Vision Pro’s hand tracking is among the most advanced in consumer hardware, using a combination of high-resolution cameras and machine learning to detect intricate hand movements without additional hardware. Community developers have capitalized on this by integrating it directly into PC VR ecosystems:

• SteamVR Compatibility: By mid-2024, ALVR updates allowed Vision Pro hand tracking to be recognized by SteamVR as a valid input method. This meant that games with native hand-tracking support (e.g., titles like The Lab or Job Simulator, which were adapted for controller-free play on other platforms) could be played on the Vision Pro without controllers. For other games, enthusiasts have created custom input mappings, translating hand gestures into actions like grabbing or pointing, effectively bypassing the need for physical controllers in less complex scenarios.
• Custom Gestures: Some developers have gone further, creating visionOS apps or ALVR mods that define custom spatial gestures (e.g., a wave to open a menu or a fist to “shoot”). These are then sent to the PC, broadening the range of playable games without requiring traditional VR controllers.

3. Leveraging VisionOS Updates

Apple’s visionOS 2, released in 2024, introduced support for room-scale VR, a feature that tracks a user’s movement within a defined space. The community quickly integrated this into ALVR, enhancing the Vision Pro’s ability to handle full VR experiences streamed from a PC. This update improved positional tracking accuracy, making it feasible to play games that rely on body movement rather than controller input—think walking simulators or exploration games like Myst VR. By combining this with hand tracking, users can navigate virtual worlds and interact with objects using natural movements, reducing reliance on controllers.

4. Optimizing Latency and Performance

Streaming VR content from a PC to a headset introduces latency, which can disrupt immersion. The community addressed this by optimizing ALVR for the Vision Pro’s hardware, thanks to Wi-Fi 6 and high-level compression.

5. Game Compatibility and Workarounds

Not all VR games are inherently playable without controllers, as many rely on buttons, triggers, or analog sticks. This situation may be addressed in the future thanks to Apple and PlayStation collaboration. But for the moment the community has tackled this in creative ways:

• Modding and Remapping: For games requiring minimal controller input, enthusiasts have modded SteamVR bindings to reinterpret Vision Pro hand gestures as button presses. For instance, a pinch might simulate a trigger pull, or a pointed finger could mimic a joystick click. This isn’t universal but works for slower-paced or experimental titles.
• External Input Devices: While the focus here is controller-free, some users pair Bluetooth gamepads (like Xbox or PlayStation controllers, officially supported by Vision Pro) for hybrid setups, though this sidesteps the no-controller goal, the Wii and Switch controllers are good candidates for VR controllers too.

Additionally, there are hackathons entirely dedicated to further push the limits of Vision Pro, like the one is going to take in March 16 in Japan.

VisionOS Updates

Let’s start with the official new features made by Apple. It introduced features like 2d-to-Spatial, SharePlay for shared memories, Spatial Personas, and new navigation gestures for easier use. Also they improved the Mac Virtual Display with wide and ultra-wide formats, boosting productivity.

• visionOS 1.1 (Released April 12024):
  Apple introduced its new Spatial Personas feature shortly after launch, on April 2, 2024. This feature enhances the way users interact in mixed reality environments. These experiences simulate being physically present with others in a shared space.
  This represents a significant advancement over the previous Persona feature by allowing users to engage in a more lifelike manner. Here are some key improvements:
  • 3D Avatars: Spatial Personas are 3D avatars that show users’ eye movements, facial expressions, and hand gestures in real-time. This creates a more realistic interaction where avatars can appear to float in the user’s physical space. Is like having a floating Hologram in front of you!
  • Enhanced Interaction: Users can interact with each other as if they were in the same room. For instance, during FaceTime calls, participants can share applications, draw on virtual whiteboards, or watch movies together in a shared virtual environment. The avatars maintain spatial context, meaning they can be positioned relative to one another just like in real life.
  • Multiple Interaction Modes: Developers can utilize three positioning templates for Spatial Personas. These are Side-By-Side, Surround, and Conversational. This allows for various interaction styles depending on the context of the meeting or activity.
  • Immersive Audio: Spatial audio technology enhances the sense of proximity and directionality of voices, making conversations feel more natural.
  • Third-Party Support: Spatial Personas support up to five participants simultaneously. Third-party Apps, including gaming and collaborative tools -like Freeform and SharePlay-enabled apps- are able to use Spatial Personas.
• visionOS 2 (Released September 12024):
  • Spatial Photos: Users can view 2d photos as Spatial Photos with a single touch, adding depth and dimension. Users judged that the conversion is excellent.
  • SharePlay: Enables sharing and interacting with spatial photos and videos during FaceTime calls, fostering social experiences.
  • New Navigation Gestures: Introduced intuitive hand gestures for quicker navigation, improving user interaction.
  • Web Videos in Environments: Allows watching web videos in immersive settings, like the white sand beaches of Bora Bora.
  • Enhanced Guest User: Simplified sharing the device with others, enhancing accessibility.
  • New APIs and Frameworks: New tools to create compelling spatial experiences. One example is surfaces recognition, so Apps can use tables or walls as a canvas. This feature is used by many Apps today.
• visionOS 2.2 (Released December 12024):
  • Improved Mac Virtual Display: Added wide and ultra-wide formats, equivalent to dual 5K monitors, significantly enhancing productivity for Mac users.

New Apps and Contents

The Vision Pro ecosystem grew rapidly, with over 600 apps at launch in February 2024, reaching over 1,000 two weeks later. Now the number exceeds 2000. Notable apps include JigSpace for 3D and AR models, the NBA app for immersive sports viewing with Multiview, and What If…? An Immersive Story for a Marvel experience. Games like Loóna and THRASHER leveraged spatial computing, while health apps like Visage Ease VP transformed healthcare. The 2024 App Store Awards highlighted these innovations, with finalists pushing spatial computing boundaries. We showed a lot of that early Apps on our article 1 month after the release of Vision Pro. On our Telegram chat we shared some of the new Immersive 8K 3D videos released exclusively for Vision Pro, like this one:

Several apps stood out, particularly those recognized in the 2024 App Store Awards, announced on November 25, 2024 (App Store Awards 2024):

• JigSpace: Empowers users to view 3D and augmented reality models, ideal for education and presentations.
• NBA App: Multiview allows up to five live broadcasts and real-time stats, enhancing fan engagement. The impressive new Tabletop mode recreates the entire basketball court and its players in 3D in front of you. Much better than watching the 2d streaming.
• What If…? An Immersive Story: Offers a Marvel experience, leveraging spatial computing for immersive storytelling.
• Loóna: A relaxing 3D puzzle game, providing a serene gaming experience.
• THRASHER: Features smooth controls and a mesmerizing alien world, showcasing gaming innovation.
• Health Apps (e.g., Visage Ease VP): Transforms healthcare with spatial experiences in clinical education, surgical planning, and medical imaging, as noted in Apple’s newsroom (Health App Opportunities).
• Vision3D: allows to design 3D models directly with your fingers.

These apps, detailed in various tech reviews (Best Vision Pro Apps), illustrate the diverse applications of Vision Pro, from entertainment to professional use.

What’s next

Now, Apple have introduced several new features for the Apple Vision Pro as part of the visionOS 2.4 beta. These features include:

• Apple Intelligence: This suite of AI tools brings features like Writing Tools, which help with text summarization, rewriting, and proofreading; Image Playground, allowing users to create playful images; and Genmoji, for crafting custom emojis. These tools aim to enhance productivity and creativity on the Vision Pro.
• Guest User Feature: Enhancements to Guest mode make it easier to share the Vision Pro with others. Users can control which apps guests can access and use View Mirroring via AirPlay to guide them through the experience.
• Apple Vision Pro App for iPhone: This app allows users to manage their Vision Pro from their iPhone, including downloading apps, exploring new content, and accessing device information.
• Spatial Gallery: A new app featuring curated collections of spatial photos, videos, and panoramas to showcase the immersive capabilities of the Vision Pro. Here are all the details:

The Spatial Gallery app features a curated collection of spatial photos, spatial videos, and panoramas. Here are its main features:
• Curated Content: The app offers a selection of content curated by Apple, featuring works from artists, filmmakers, photographers, and iconic brands like Cirque du Soleil, Red Bull, and Porsche.
• Diverse Categories: Content spans various categories such as art, culture, entertainment, lifestyle, nature, sports, and travel.
• Regular Updates: New content will be released regularly, ensuring users have access to fresh experiences.
• Exclusive Experiences: Includes behind-the-scenes moments from Apple Originals and special moments from top artists.
• Availability: Spatial Gallery will be available in all Apple Vision Pro countries except mainland China.

This app is designed to showcase the immersive capabilities of the Apple Vision Pro, providing users with breathtaking and intimate experiences through spatial media.

These features are set to be released to the public in April 2025.

Conclusion

The past year has been transformative for Apple Vision Pro, with significant software enhancements, a booming app ecosystem, and community advancements are still emerging, the device’s trajectory suggests continued innovation, particularly in spatial computing applications.