visionOS

How can independent developers gain access to the front-facing cameras for app development?

• Access to stereoscopic camera pairs is available through standard Apple Developer Program accounts, typically for incorporated businesses.
• Developers with a clear use case and need for camera access can apply for it.
• When developers encounter issues accessing cameras and have a valid use case, post on the developer forums.
• Share the use cases for front-facing camera access on Feedback Assistant to help Apple understand developer needs.

How can developers debug applications in immersive spaces without losing context or relying solely on pass-through?

• Enable the "Mac virtual display" developer setting within an immersive environment.
• This allows developers to run and debug the app in Xcode while remaining within the immersive space, providing a safer and more efficient workflow.
• The Mac Virtual Display offers a large, high-quality display for debugging and triaging issues.

Are there any agent skills from WWDC that are particularly relevant for visionOS development?

• The most relevant agent skill helps resize existing iOS apps for visionOS.
• This skill assists in translating iOS UI elements to the visionOS form factor, maintaining best practices for the platform.

What is the recommended workflow for exporting 3D models to USD, especially when dealing with FBX or GLB files?

• Start with USD export directly from the 3D content creation tool (3DCC) to avoid format conversion issues.
• Be aware that different 3DCCs may have variations in how they build files, which can sometimes complicate USD export.
• For iterative workflows, consider using the USDA (ASCII) format over USDC (binary) to modify individual files without repackaging the entire scene.
• Modern LLMs can potentially assist with USD file understanding and manipulation, leveraging the core USD specification.

Can spatial accessories be tracked when they are outside of the user's direct line of sight?

• visionOS 27 has expanded support for spatial accessories, opening the spec to new accessory types.
• Accessories use a combination of Bluetooth, onboard IMUs for high-frequency tracking, and infrared emitters visible to Vision Pro's cameras.
• Tracking fidelity for occluded or distant accessories depends on the accessory's size, distance, and the Vision Pro's ability to see the infrared emitters.
• A new debugging mode allows developers to see the infrared feed from Vision Pro to assess accessory visibility.
• Developers are encouraged to share use cases, such as foot trackers, via Feedback Assistant.

When using Gaussian splatting, why are splats culled when the user's face moves close to or inside them, and can this be disabled?

• There is currently no known way to disable the vignetting or culling effect for Gaussian splats when they get too close to the user's face.
• This effect is also present for other 3D models to ensure an unobstructed user field of view.
• Developers are encouraged to file feedback with detailed use cases, especially for telepresence applications, to inform future development.
• Providing sample projects, videos, or screenshots with specific distances can help engineers reproduce and address issues.

Is Gaussian splat rendering supported on platforms other than visionOS, such as iOS?

• Gaussian splatting should be supported across both iOS and visionOS.
• When developers encounter issues or documentation mismatches in Xcode, file feedback.
• Beta versions of features may not always work as expected, so reporting issues is important.

Why aren't there user profiles on Vision Pro, similar to macOS, to allow for easy switching between different users (e.g., owner, child, guest)?

• Vision Pro is designed as a personal device, prioritizing the best experience for the primary user.
• To facilitate sharing, enrollment data can be saved to an iOS 26+ device, allowing quick switching via an App Clip code when in guest user mode.
• Guest user mode with nearby device pairing allows others to access the Vision Pro without needing the owner's passcode, with screen mirroring via AirPlay for assistance.

Is there an API to detect the owner's lips and face to mimic them on 3D models like Persona, but with different effects?

• Face tracking provided by ARKit is an iOS-only feature and is not available on visionOS.
• While raw face data is not exposed, third-party apps can access a simulated video feed from the front-facing camera equivalent.
• This simulated feed provides a matted version of the persona (persona in front of a background), suitable for video conferencing but not for applying lens-style effects to the face.
• Developers are encouraged to file feedback for use cases involving custom face manipulation.

Does Visual Intelligence require third-party apps to access the main camera?

• Visual Intelligence is a system-level capability and does not require third-party apps to access the main camera.
• Information is not shared with third parties, ensuring a privacy-preserving experience.
• The system can provide immediate answers about viewed content, including real-world objects.

How do game controllers fit into the visionOS interaction model in version 27?

• Apple initially supported PlayStation VR2 Sense controllers and the Logitech Muse stylus via the Game Controller framework and ARKit.
• These inputs are deeply integrated at the system level, usable for launching apps and controlling window behavior.
• visionOS 27 introduced expanded spatial accessory capabilities, allowing developers to build custom accessories with high frame rate and accurate pose tracking.
• The primary interaction model remains gaze and pinch, with controllers and spatial accessories offering extended capabilities for specific experiences.
• The type of input should be thoughtfully chosen based on the intended user experience, whether for refined input, gaming, productivity, or natural interaction.

Are there any updates to WebXR in Safari with visionOS 27, particularly for immersive AR mode?

• Safari on visionOS supports WebXR, but currently only in an immersive-only rendering mode (VR).
• Immersive AR mode for WebXR is not yet supported.
• Developers are encouraged to investigate visionOS's platform APIs for augmented reality experiences, which may offer capabilities beyond current WebXR limitations.
• New features like the <model> tag allow for positioning USD assets within a webpage and then pulling them out into the user's environment.

For Visual Intelligence on visionOS, is it purely camera-based, or does it leverage depth sensors?

• The system uses a blend of all available sensors on the device to provide the best possible and most accurate results.
• This approach is not unique to Visual Intelligence and applies to other system-level functionalities like SLAM and hand tracking.
• The specific sensor implementation varies by device to optimize for its capabilities.

How can Vision Pro stay awake on a desk during development?

• There is no sanctioned way to keep the device awake on a desk during development due to security reasons.
• Vision Pro lacks a lock button, and the device automatically locks when removed from the head to ensure authentication (Optic ID or passcode) upon re-wearing.
• Mac Virtual Display (MVD) is the recommended workflow for development, allowing developers to keep the device active and connected while debugging.
• Setting up Optic ID can significantly speed up re-authentication when removing and re-wearing the headset.

What is the state of automated testing for visionOS, ARKit, and RealityKit?

• For RealityKit, use the RealityRenderer class for driving rendering through automated tests.
• Explore the "Migrate to Swift testing" and "Get the Most Out of Device Hub" sessions for automated testing strategies.
• Device Hub offers significant capabilities for automated testing.
• RealityKit is written in Swift, so existing Swift automated testing infrastructure and Xcode tooling should work.

What are the biggest mistakes new visionOS developers make?

• Information Density: Avoid overwhelming users with too much text or information in large windows, as it differs from 2D platforms.
• Tap Target Size: Adjust the size of tap targets and UI elements for eye and hand interaction, which is different from touch-based interfaces.
• Leveraging visionOS Capabilities: Don't just port 2D apps; create new features and experiences that are uniquely possible in visionOS, such as 3D product models that can be placed in the user's environment.
• Content Quality: Pay close attention to encoding settings and visual quality for compressed content (video, images) as Vision Pro's high-fidelity display can reveal artifacts.
• Thinking Beyond Existing Paradigms: Embrace spatial computing's unique capabilities rather than solely relying on past technological approaches.
• Review the Build Next Generation Experiences for visionOS 27 session for Unity, Unreal, and streaming pathways.

Regarding agentic coding in visionOS, if an LLM generates visionOS project code with incorrect syntax, is there a way to fix this or teach the AI about visionOS code?

• LLM-generated code may contain errors; always double-check the output ("trust but verify").
• Use the robust library of visionOS sample projects as training data for local LLMs.
• Newer LLM models may not be trained on the latest API SDK capabilities, leading to potential discrepancies.
• Utilize specific visionOS sample projects like "Hello World," "Petite Asteroids," "Canyon Crosser," and the "Model Manipulator" sample for targeted learning.

For object tracking in visionOS, how fast can it get, and what are the possible frame rates and latency?

• visionOS 2.0 primarily focused on tracking stationary objects at approximately 5 frames per second (Hz).
• visionOS 27 introduces a high frame rate mode, supporting up to 30 frames per second with significantly lower latency.
• Engineers have implemented smoothing algorithms that make tracked objects feel more attached and responsive.
• The same trained object tracking models can now work on iOS devices, with an extended training mode in CreateML for higher accuracy and lower latency.
• Object tracking updates are now frame display time-aligned, eliminating guesswork about frame timing.

Will Apple expose more underlying sensors and frameworks for accessibility telepresence?

• Apple does not comment on future plans.
• Developers are encouraged to file feedback with specific use cases, especially for accessibility.
• Accessibility is a core principle, and Vision Pro has significant potential for enabling accessibility features through developer or internal initiatives.
• Features like the magnifying glass and wheelchair control powered by Vision Pro highlight the platform's accessibility potential.

How can Mac Virtual Display work on shared or enterprise Macs without a shared iCloud account?

• Mac Virtual Display currently requires both the Mac and Vision Pro to be on the same iCloud account.
• Alternative methods like reverse AirPlay or screen mirroring from Mac to Vision Pro exist but do not offer the same experience as Mac Virtual Display.
• Spatial Preview functionality is not iCloud dependent and can be used to share USDZ compliant content to other devices.
• Remote Immersive Space allows rendering on a Mac and streaming stereoscopically to Vision Pro without requiring the same iCloud account.
• Foveated streaming also enables bringing PC-rendered content to visionOS, with the visionOS app retaining native capabilities like SharePlay.

Can Gaussian splats be part of a USDZ file, perhaps mixed with other meshes, and used in a website environment?

• OpenUSD supports "particle fields," a generic form of Gaussian splats, but this feature is not yet directly integrated with Reality Kit's splat capabilities.
• Apple is working on closing this gap in future releases.
• USD can express multiple elements in a scene, allowing for the containment of different types of data, including splats and meshes, within the same scene.
• The development process involves aligning specifications, runtime capabilities, and authoring tools for a complete Gaussian splat ecosystem.