Hello, everyone. Welcome to the Vision OS Group Lab. My name is Adarsh Bhavani. I'm a technology evangelist for Apple Vision Pro. I'm a part of the worldwide developer relations team over here at Apple. I'll be your host for today, and I'm joined by an expert panel on Vision OS. These are folks from the Vision Products Group, which is the engineering team that was responsible for building Apple Vision Pro and all the subsequent Vision OS versions. Let's go through a quick round of introductions. You want to start off, Katie? Yeah, fantastic. Hi, Katie. I help developers build apps for Apple Vision Pro and Vision OS. Hi, my name is Norman. I work on all the technologies related to rendering, simulation, and gaming on Vision OS. This includes RealityKit, RealityComposer Pro, USDKit, and the latest Voviet streaming. Hi, I'm John. I'm an engineer on the Realticut team, and this year I also had the privilege of working on the new USD Kit framework. Hi, I am Matt. I sit within the Vision Products Group, focused on everything related to professional and enterprise applications, as well as gaming and interactive with third parties. I'm Travis. I work in Vision Products Group, and I focus on software standardization strategy, and I work quite closely on technologies like USD. us have a background in media and streaming. Thank you, everyone. In addition to the ones that you see on screen, we have a team behind the scenes that is going to help us answer your questions. I already see some questions rolling in, so thank you for submitting those. If you have code-specific questions, please do take those to developer forums. That's developer.apple.com slash forums. If you have bug reports or feedback, that's very welcome. Please do share that with us on feedbackassistant.apple.com. It's the feedback that you've shared over the years, ever since the launch of Vision Pro, that has made it into an amazing platform. And we have some exciting new features that we have announced with Vision OS 27. That's the latest Vision OS version. We have a completely redesigned Reality Composer Pro. We have many new features in RealityKit, brand new frameworks like Spatial Preview Framework or USDKit. We also have a Foveated Streaming Framework alongside support for enhanced object tracking and accessory tracking. So there's a lot. I probably won't be able to read out all of the new things that we have introduced this year, but it has become possible only because of the feedback that you've submitted to us. So please keep that going. So with that, let's kick it off. I'll start with what were some of the features that were your favorites. Katie, do you want to kick it off and share what was your favorite feature in VisionOS 27? Yeah, okay. I've thought about this a lot since yesterday. I have two. I'm going to go with one, spatial panos on VisionOS 27. Unbelievable to view your panos that you took on iPhone in a whole brand new way on VisionOS 27. Fantastic. And then separately, just like a general Apple thing, Xcode is amazing this year. Like fully embraced agentic development. It's unbelievable. I'm very happy to get away from the command line and get fully back into Xcode. Fantastic updates there. I don't have to cheat like Katie having two features. Yeah, you just picked one. One of my favorites from VisionOS 27 is physical space lighting. We came a long way in terms of being able to understand the environment you're in. So when ARKit was first released, we have the ability to understand the room you're in, right? The physical light direction, color temperature, light positioning, and being able to generate an environment map. But this is the first year we actually are able to affect the lighting you see here in your physical space by introducing the new physical space lighting API. So you'll be able to create that immersion that to the customers you wouldn't able to tell what's real versus what's fake. When I first tried out this feature, some of the demo I've seen with a flashlight coming out of the portal, shining on my walls, shining around me, I almost had a sense of I can feel the warmth of these lights on my face. So that was really, really delightful for me to see how the developers like you all are able to take it to the next level. That's a great one. John? I'm really excited about USDKit this year. USD is such a fantastically expressive 3D model format. And I'm really excited about it being much more easy to use, easy to pull into an app that works with 3D models. A lot easier to make your app export an expressive USD scene on all kinds of Apple platforms. So I'm really excited to see what people do with that. Katie did too, so I'm doing too. That's fine. That's cool. I set the precedent. Sometimes I think that the most magical features are just these little ones, the quality of life. So I'll start there. And I think that, for me, that's the ability to do high-quality recording on Vision OS directly in device for the first time. So for those less familiar, today when you do a screen recording on Apple Vision Pro, historically it has been foveated, meaning in the same way that when we're showing content to the user, we optimize the content in the region where the user is looking. Now, when you are taking a screen recording, that screen recording won't be embedded with that foveation, meaning that if you want to do a high-quality capture, whether it's for marketing materials or it's to put on your App Store page, you actually have the ability to do that directly on Vision Pro. Historically, the way that you would do that is you plug in via the developer strap and you use the developer capture feature, but a lot of people don't have developer straps, and it's an additional piece of equipment you have to bring with you. So now within settings, you can actually toggle that on so you can do up to a three-minute fully unfoveated capture that makes just capturing the beauty of your Vision OS experience for traditional platforms that much easier and that much cleaner. So that's one. I think the other one, I'm sure we'll be talking more about it, we launched two new features this year really focused around how you can expand applications that exist on the computer to Vision Pro. So one of those two personal favorite of mine is the foveated streaming framework. Back actually in 26.4 a few months ago, we announced foveated streaming with NVIDIA's CloudXR. And we're expanding upon that this year with VisionOS 27. And really what that's all about is in the professional sector specifically or really high-end gaming workflows, people have always agreed that Vision Pro is this incredible standalone device. But for people that had existing workflows or existing workstation set up to play high-end simulations and games, they didn't have the ability to use that with their Vision Pro. They had two different devices. So now thanks to foveated streaming, we can actually stream wirelessly from content that's being rendered offline to Apple Vision Pro. And thanks to this foveation, we're still able to maintain the privacy of the user. The app is still not aware of where the user is looking, but we can benefit and actually optimize the streaming based on the approximate region where the user is looking. So this opens up a tremendous amount of possibilities for bringing additional content to Vision OS. And it's also not either or. It's not like if you're going the foveated streaming route, you can't continue to use RealityKit content. You have the ability to blend the best of both worlds, and it's a truly magical experience. Awesome. Yeah, for me, it's spatial preview. It's a really exciting opportunity to see the Mac and Vision Pro kind of working together to kind of step function what you can do in terms of visualization and experience. And plus, I have a soft spot in my heart because a preview at one time at Apple was my product. And then the other thing I think is really interesting is, you know, over 10 years ago, we made an investment in USD as a technology. And it's excellent for expressing scenes with complexity and advanced materials. But there's an aspect of USD that's often forgot that's fundamental to technology, which is fostering collaboration. And you're seeing that in the ability for people to manipulate objects in Vision Pro and have that seamlessly sync back to apps like Preview or other developer opportunities there. So it's going to be really fantastic to see that as a first step along the way of not only introducing new collaborative paradigms, but then additionally exploring the full capabilities of a technology like USD. Excellent. Those were a lot more than what I started out with. So thank you for sharing all of your favorite features. Let's turn to some of the questions that we are receiving from the community here. I have a highly upvoted question here from Wes Matlock. It says, as an independent developer, what is the best way to gain access to the front-facing cameras for learning and trying something for an app? I understand that you need to be a part of an enterprise development group. Am I missing something, or is it just impossible for an individual developer to gain access? Matt, do you want to take that one? Yeah, absolutely. So when we launched Vision Pro, we knew that there would be desire from certain developers to be able to leverage the cameras. And we first announced the ability to access the main camera. We actually initially just announced, or initially just launched access to the left main camera. And then last year with Vision OS 26, we launched access to the stereoscopic pair. So you have the ability to do it. The primary way in which we've provided the ability for developers to approach this is really through demonstrating what their application is going to be doing, primarily with a professional and enterprise-oriented target. So it's a little bit nuanced. I do want to suggest that we have a few different types of developer programs. This is not limited to folks in the Apple Developer Enterprise Program. This is available to developers who want to apply, who have standard Apple Developer Program accounts. Usually that's for people that have incorporated a business or have their developer account attached to a business. But it doesn't mean that you yourself have to be within the enterprise program themselves. But if you are yourself having issues and you have a desire and you have a real use case on why you want to use this and you can't access it today, please just post in the forum. We're always looking for new ways to provide access to these features for developers. For the folks who have tuned in, if you have use cases that you think will benefit from having access to the front-facing cameras, please do share that with us on Feedback Assistant. That will be of good learning for us. So appreciate that. All right, moving on to the next question here. It's from Thomas Bastable. It's also a highly upvoted question. Hitting a breakpoint in an immersive space means standing inside a frozen universe of your own making, squinting at Xcode through pass-through. I would love to know how you guys debug on device, as I feel like I'm missing a trick here. Wants to take that one. It's a really good question. Katie, why don't you take it? You've really liked Xcode. Go for it. Tell me about it. I'm getting made fun of nonstop by my team, I imagine. So there is a developer setting as of, I think, last year that allows you to use Mac virtual display within an immersive environment. So, for example, if this developer is like building an app where they're surrounded by, I don't know, some sort of virtual world, like it sounds like they are, they could be building and running to Xcode and have Mac virtual display up as well. So when they hit that breakpoint, they can very easily stay where they are and be safe because it is a safety issue, of course, at the end of the day. And then also triage and debug. It's very helpful. I mean, internally, generally, like I think a lot of us, like when we are working in Xcode or we're exploring other applications, the workflow is primarily in Vision Pro with Mac virtual display and the immersive content. Yeah, that has been one of the common use cases. It's a huge way to do that. I think it's so efficient as well, right? Because otherwise you're donning and doffing quite frequently and Mac virtual display is like the best display that you can have. So why not do it all in one place? Plus a huge display. Yeah, it's incredible. Ultra wide, right? All right. Moving on to the next question here. It's someone with a username Jodotri. The question is, are any of the agent skills Apple's providing this week especially relevant for vision OS development? Kira, do you want to take that one? Yeah, for sure. I mean, there are a couple of discrepancies between an iOS app and a vision OS app, right? Just by the nature of the different platforms. And so I think of the skills that have shipped thus far, the one that's most directly relevant is to help resize your existing iOS app to VisionOS, right? Like, if you're running an app on a 2D iOS screen, it's naturally going to be expecting a certain form factor, a certain size, maybe a certain amount of information density, and I believe that we shift a skill that will enable pretty seamless translation of that UI for iOS to VisionOS in a way that, you know, maintains some of the best practices and principles for VisionOS. Great. Excellent. Alright, next question from Sysmedia. I'm sorry if I'm mispronouncing something here. But the question is, what is your usual workflow for exporting 3D models into USD? Some tools apply different transforms or remove certain materials on complex models. The usual models being FBX or GLB files. Travis, you want to take that one? Yeah, I mean, the best option is to start back in your 3DCC you're actually creating the applications. And rather than transiting through another file format, which may put its particular format fingerprints on the file, is to see if you can start as far back as there and get an export out of that. And that may help. But there's going to be obviously variants in what 3DCCs do, how they build any of these files, not just USD, that sometimes can be a little bit complicated. And part of that is represented where the state of the industry is, as it increasingly uptakes USD as a format that's used for scene description and interchange. There is some variability there, unfortunately, and it can be challenged. But just looking at the question, I think trying to start with USD as early in the process or straight out of the 3DCC is what we recommend. One thing I will add is depending on your workflow, If you are trying to iterate your 3D models, I think looking at the loose file format of USDA versus USDC, those are really good to give you the ability to modify individual files without having to repackage up the entire scene. So that truly will accelerate some of the iteration processes you're trying to do. Yeah, and one thing that's actually kind of interesting and has a little bit of a standard segue is that one of the important things that's happening in the standards space in general is that a standard is a human readable representation that normally a human reads and then translates for the computer. And now with modern LLMs, AI, you know, they're able to do that in some ways for themselves. And so one of the important things that's recently happened in the USD community is the core specification for USD came out, which details in depth about the essential parts of USD. And now that's a way for LLMs to kind of understand that and then potentially apply that to your model. And so you might be able to ask your favorite LLM to help out there. That's right. Great. Moving on to another upvoted question here. It's with someone with the name Eric in the username. Can spatial accessories be tracked outside of the range of direct view? I'm imagining designing foot trackers, but obviously the user will not always be looking down at their feet. I'll take that one if I can add it as my third favorite feature. I'm going to have to have a third now. We're going to heart all the features in the release. We're super, super excited by all of the updates that we've made to both object tracking and accessory tracking with VisionOS 27. So specifically, I think specifically this question asked about spatial accessories. And one of the big things that we did with VisionOS 27 is we opened the spec up. Last year when we announced spatial accessories, there were two accessories that we supported. We supported the PlayStation VR 2 Sense controllers and we supported the Logitech Muse, the stylus device. We've gone well beyond that now because we know that there are all these use cases, whether they're more consumer-facing use cases being used for games or interactive experiences or on the enterprise side. There's a tremendous amount of use cases, whether they have to do with the alignment of a physical scene or physical prop to a digital environment or something that you need to hold in your head and have high precision, even maybe if you're in a low-light environment. So one of the big things we announced yesterday is we've actually opened up the spec for new accessories to come to life. So there's a few companies, DF Robot, MicroE, that are actually selling now accessories that allow you to actually append an accessory onto an existing object, which makes it super easy. You can take a physical prop and attach, for example, the cap accessory to the rear of it, and now you have the ability to track a relative point relative to the physical device. But then we're also – these companies are also providing the boards themselves that have the infrared trackers on them as well as all the other components that would be required to be able to build your own accessory. If you are in the business of building hardware, you're 3D printing a prop, it makes it super easy to build your own accessory. So there's a great session on that. I highly recommend you take a look at the what's new in object tracking and spatial accessory session. But this question looks like it talks specifically around tracking objects that might be outside of direct views, for example, foot trackers. So I think the first thing I would say is that these accessories use a combination of two things primarily for tracking. There's obviously a Bluetooth connection between Vision Pro and the accessories. But then there is an IMU on board, an initial measurement unit that is doing a very high frequency tracking. And there's also this constellation of infrared emitters that the Vision Pro, the infrared cameras on Vision Pro are able to see, which is amazing because the user can never see those. And also in a low light environment, the Vision Pro can still see the IR light. So I'd say that like in a position where maybe they are occluded by either your body or they're further down, it really depends on what fidelity of tracking is required. I would say that, like, same thing goes if you're using a PlayStation VR 2 control and you put it behind your hand for a second, you're not going to fully lose tracking because the IMU takes over. But obviously you want, ideally based on the size of an accessory and the distance to that accessory as well as the Vision Pro's infrared cameras being able to see it, you want to optimize those things to be able to track to the highest fidelity. and also new with VisionOS 27 to make this easier is that we actually have provided a debugging mode where you can see the actual infrared feed that Vision Pro can see so that you can actually see whether or not the Vision Pro is capable of seeing the accessories at the specified position that you're looking for. So that I'd say is a long way. We'd love feedback on this. We've heard from other folks' desire in being able to track feet. I'm really interested in understanding why that is. Is that really to do more like breakthrough, which we obviously provide for your upper body today, but we don't provide today for your lower body. So we'd love to also understand this use case a bit if you can file a feedback item for us. Yeah, and like Matt pointed out, there's a dedicated session that we have released with WWDC set of sessions yesterday. It's, I think, something that says explore enhanced object tracking. It doesn't call out accessibility tracking in the title, but it does cover accessibility tracking in the actual session. So do check it out. All right, moving on to the next one here. Again from Eric, when using Gaussian splatting, we noticed that the splats are culled if you move your face close or inside of the splat. You're trying to use GS or Gaussian splatting to reconstruct an immersive scene through sensor data for telepresence. So the culling is not ideal. Is there a way to disable this? John, do you want to take this? Yeah, that's a great question. I'm not aware of any way to disable that vignette effect that you're talking about right now. I would definitely recommend filing feedback. That sounds like a really interesting use case, and we'd love to work with you to try to find a way to make that work. Yeah, and I think that's an important aspect. You've heard give us feedback. A lot of these are new features, so they're the first version of the feature. It's beta 1. Yeah, and some of the stuff is in beta, which gives us an opportunity to potentially course correct. But I think new features, particularly things like Gaussian splats, which are very recent areas of visualization, although technically the technique is actually older than people think, it's really interesting to find out how people are going to use it. And on the basis of how they want to use it and deploy it in their application, the type of things they want to create are going to inform the ultimate feature set over time. And so, yeah, definitely get us a feedback request on that. We want to hear your voice on particular areas, new areas. I will also just add, this vignetting effect is not just what we're limiting to Gaussian splats only. For any of the 3D models, we have the same level of treatment. And one of the motivations is when the object is getting a little bit too close to the user's face, we want to have the ability to kind of have an unobstructed user field of view. But obviously, if you have a different creative intent, please do let us know like what uh travis has mentioned we would love to figure out how to incorporate this feedback for future releases yeah and i think just to reiterate one thing that that you all both said when you're filing feedback like the the more context we get in terms of the types of use case you're trying to solve the better right like i want to use gaussian splatting reality kit is one thing but like those can take so many different forms right people want to use gaussian splats to reconstruct this cup which is very different than people want to use gaussian splats to recreate an entire uh environment the size of an airplane hangar right so the more context we can get on what you're exactly trying to achieve, the more we can actually help use that feedback to motivate our engineers and priorities in terms of what we're building. Yeah, and if you run into a bug, a really simple sample project also goes a very long way because that helps us reproduce issues and then ultimately triage them and fix them. In addition to the sample project, like a video or a screenshot of what you're seeing, oftentimes helps. Even if it's a feature request, like this is a perfect example where you're seeing some culling, it might be good to see at what distance were you with your object that you were using Gaussian splatting with for us to better address that feedback. So please do attach any screen recordings or screenshots alongside a sample project if you can. And of course, as much detail as you can give in the feedback ticket. All right, moving on to the next question. It's a highly We have a question again from Ethan. Is Gaussian splat rendering supported on platforms other than Vision OS? The documentation shows support for iOS, but Xcode does not show the symbols available. John, do you have an idea? Yeah, that's a great question. Definitely recommend filing a feedback if you have any issues or if the documentation doesn't appear to match what's-- or you can't get it working in Xcode, this is the perfect time for us to help you out. Right. Things are still in beta, so if something's not working as expected, do file for feedback. In general, Gaussian splatting should be supported across both iOS and VisionOS. But if you find things otherwise, do let us know. All right, next question here by Oliver Colon. Why aren't there user profiles like on Mac I want to put on the headset and choose who I am? Owner, child one, child two, grandma? Matt, do you have? Yeah, I mean, I'll take that one. I think that there are many scenarios where people are always interested in sharing devices. And there's definitely the family example, which I think was called out here. And there's definitely, we hear similar requests within the professional workplace as well. I think generally, like, Vision Pro is an incredibly personal device. And in order to create the quality of experience that has been paramount to this overall product, it's really about how do we ensure that we're providing the best experience for the primary user. That's where we started at least. But the more feedback that we've gotten from folks being interested in sharing, we've started to make some progress towards that goal. The flip side of a product being very personal is that it's often not by design, always the easiest thing to share. So there's some features we've done here. I think we look at it maybe more akin to like an iPhone or an iPad than maybe like a Mac that has profiles. But one thing I know we always get questions about the enrollment time that it takes when you're going through system enrollment to calibrate your eyes and hands. Oh, my God, this takes some time. I want to quickly share between a friend and I. We've actually done huge progress, made huge progress towards that. So what I do when I'm sharing my device with my family or my colleagues is last year we announced the ability to actually save your enrollment to your iOS device. So if you have an iOS device running iOS 26 or later, you can actually, once you go through the enrollment process, you can transfer that enrollment to your iOS device. And now every time I put on somebody else's device and it's in guest user mode, it's as easy as putting on the device. My phone automatically pops up with the card with my app clip code on it. I look at it and I'm in their device immediately. so that's I'd say like the biggest lift one of the biggest friction points I think we've been able to alleviate quite a bit as well as other things like the guest user with nearby device where if you have an iOS device, an iPhone or an iPad paired with the same iCloud as a Vision Pro, you can actually let somebody into your device without fortunately without needing to tell them your passcode being able to screen mirror through AirPlay and see exactly what it is they're seeing so that's how we've solved some of these friction points that exist with trying to share devices with your family or colleagues Great. Thank you. All right. Next question. Is there any API to detect the lips and face of the owner and mimic that on 3D like Persona, but differently? Do we have something like face tracking that we have on iOS, John, with Vision OS? The face tracking provided by ARKit is an iOS-only feature. Okay. But this sounds like a really fun use case, and I definitely recommend filing a feedback about this because the more use cases we have like this, we're very interested in this kind of thing. I think the only thing you can get, we're not providing raw face data or anything, but as a third-party app, you can call on the feed from what would be the equivalent of a front-facing camera on an iPhone, and you can get a, it's not the full spatial persona, but you can get the, like a simulated video. It's a matted version of the persona, right? So it's a video feed with your persona in front of a background. So you could use that for like a video conferencing app or anything like that, but you can't, you know, do some sort of lens style effect on someone's face or anything like that. That's right. Cool. Another upvoted question here by someone with the username Dillium. The question is, for visual intelligence on Apple Vision Pro, when Siri AI answers questions about content the user is looking at, either app windows or real-world objects, does a third-party app need access to the main camera, or is visual intelligence system handled entirely at the system level? Katie? Yeah, sure. That's a system-level capability. So no information is shared with third parties. Yeah, totally done in a privacy-preserving way. It's actually pretty magical, too. You look at something and immediately get an answer. I was actually quite thrilled with, like, do these shoes fit my bag? Yeah, that was very cool. I've got to say, as a sneakerhead, that was a nice little thing to see personally. But also the new representation of Siri AI on Vision OS 27 is pretty fantastic. It's beautiful. Excellent. Another question here. For apps that support game controllers but aren't games, How does Apple envision controllers fitting into the long-term interaction model of VisionOS alongside eyes, hands, and voice? Are there new controller capabilities in VisionOS 27 that developers should be designing for today? Norman, you want to take that? Sure. I think for us is like what Matt said, right? Initially, we opened PSVR, the Sense controller, as well as the Logitech Muse. And it's not only we provide that as a standard sort of API by using both game controller framework and ARKit to get the orientation as well as button presses, but we also have the ability to fully integrate all these input devices fundamentally at the system level. So being able to use that to control a launcher app versus control how the Windows Chrome behaves, it is deeply integrated as part of the Vision OS. As what Matt said, we also this year opened up the spatial accessory as a new form of tracking capabilities, right? So being able to add all the simple ingredients and to be able to build your own accessory, get really high frame rate tracking, really accurate pose. That's one of the brand new capabilities that will continue down this path to support at a system and SDK level. But we also feel it's important still, you know, gaze and pinch, you know, the things you already have. That's right. To make sure that if you're designing a solution that might want an alternate control, that you're very thoughtful about it. And it's adding some extended capability to the user to use something that's not what they already have. That's what I was going to add, right? Like the type of input depends on the type of experience. I feel like we keep hearkening back to it depends on what you're building, right? Like we want to know what you're building to be able to answer this question more thoroughly. So please, if you have specific questions, post to the Apple Developer forums. But in general, it really depends. You could do sort of refined input with the Logitech Muse that Matt referenced. You could build a great game using the PlayStation VR controllers, or you could just use natural input to build a different type of productivity app. Or even like regular game controllers. Yeah, exactly. That's right. Yeah, amazing. And potentially more integrated into the system on Vision Pro than on almost any other device in terms of being able to even move Windows around to do things. I said this to a group yesterday, and there was a few ophthalmologists in the room, and they verified what I said, so I'm going to say it again. in the case it sounds smart. Actually, accurate. But when you think about how we set out to build input for Apple Vision Pro, right, when we think about spatial computing, we wanted to best mimic the interaction systems that we use for the real world, right? If I want to grab this cup, the first thing that indicated that I wanted to grab this cup, even before the neurons fired to move my muscles to grab the cup, was the fact that my eyes led me to the cup. I looked at it, which is exactly why focusing on using gaze and your eyes for initial input as essentially your cursor. And then your hands, same way you interact with physical objects, no need to put down a controller as an abstraction layer between you and your virtual goods if you want to be able to translate between moving with virtual and physical items, right? So that's why we started there. But then at the same – like the flip side of that token is in this world, like we use tools all the time, right? Like none of us are finger painting beyond a certain age. Well, speak for myself. So that was the whole idea behind spatial accessories. It's like, well, yes, hands and eyes are fantastic for most things. There are times in which you want a device for precision inputs. There's a time where you want a device for haptics. There's a time where you want a device that allows you to have more easily can pull up multiple different features or menus like you'd have on a device like Logitech Muse. So that was really the intro there. And the whole idea of opening up the spec with VisionOS 27 is really just to let developers and creators and tinkerers build everything that their brains can imagine. Awesome. You mentioned something about eyes and hands. One cool feature that I really love is in Safari, or actually in most other apps as well, there's look to scroll. Yes. If you look at the bottom of the page, it scrolls automatically. It feels so magically. Like somehow it knows that I'm going to read my mind. All right, moving on to the next question here, another upvoted question. Are there any updates to WebXR in Safari with VisionOS 27? In particular, immersive AR mode. We have just released a WebXR app, which we would love to run with immersive AR mode enabled, but we are currently limited to immersive VR only. The app for interest, and there's a link in here, but do we support this feature, WebXR in Safari? We do support WebXR in Safari, but it is an immersive-only rendering mode, right? So being able to follow the – it's part of the WebXR standardization. When we look at, you know, if there are specific AR-based use case, I think, Travis, this is something your domain to think about for the future. Yeah, and then also I think an important thing is, you know, obviously there's web API and web experience. Certainly appreciate that, and the developers may focus on it. Yeah. But we also have built a tremendously capable set of platform APIs that really utilize the full capabilities of an augmented reality style experience. And so while we fully understand where the question's coming from, it also may be worth just investigating the full capabilities of our platform API. Because there may be things you can achieve outside of the web domain that may enable you to take your idea even further than you may expect. Yeah. For example, this year we have released a model tag for all platforms. And also, in addition to the model capabilities within your web page, the web page can also serve as a personal environment, being able to render that without all touching WebEx. And you can not only have the model element, can not only position a USD asset in page, but you can pull it out and place it in your environment, and it looks like it's there. And so we're already thinking along those terms, even in the web domain, but in a different pathway than WebEx are. Yeah. So if you have some specific features that you need support for, I know we have said this enough number of times, do file the feedback. But more importantly, what we are really looking forward to here is learning about how we can better address that. And knowing your use case, seeing what you're trying to build will definitely help. So if you can share that link that was posted with the question alongside any other context you can add, that will go a long way. Thank you. All right, moving on to the next question. Again, a highly upvoted question here by Raymond Yeh. For visual intelligence on Vision OS, is it purely camera-based or does it leverage the depth sensors under the hood? They want to know the magic. They want to know what's behind the curtain. The secret sauce. Yeah, I don't know if we can get too into the weeds on that one. The magicians never reveal the secrets. Yeah, we can have a deal of the treats. But, you know, we got some very smart folks who have combined data from local sources. One thing I would add is we aim to provide the best possible, most accurate result on all platforms as we can. And, you know, we have different devices with different capabilities, different camera versus different sensors. on different devices as well. So there's not a single unique implementation, but we do look at a blend of all the sensors that's available on each unique device and provide that best result. And that's not unique to visual intelligence, right? That's right. All of the system-level functionality around simultaneous localization and mapping and hand tracking and everything, right? We're leveraging what sensor makes sense for the use case and taking that combined amalgamation of sensor data to provide the most magical experience to the user. 100%, just in the same way that third-party developers do that when they build their app on multiple platforms. You don't build the same iOS app for macOS for VisualOS. They're different based on the input methods, the type of platform. More importantly, the context of the query. So depending on whether it's a visual query, audio query, or something that involves hands, something that involves interaction. Moving on, Thomas Bastable has one more question here. The device sleeps the instant it leaves your head. Taking the debug session with it, what's the sanctioned way to keep it awake on a desk during development? Any tricks? We kind of covered this one earlier. Yeah, MVD. We were talking about what we do internally. The rationale for this isn't just like we're trying to make someone's life more difficult. There's a very, very strong rationale behind this, which is that there's no lock button on Vision OS, right? Or on Vision Pro, right? You take the device off, the expectation for anyone that's ever used a device is that that device is now locked, and the next person that puts it on is going to have to authenticate, whether it's through optic ID or it's through a passcode. So that's why this exists, and anything that would do something like a caffeination that you see on caffeinating a device like we have on other platforms puts that kind of security model at risk. So that's why we haven't done it. But what we all do, I mean, do you want to speak more to your work? Yeah, I mean, I feel like we answered part of this earlier, but whenever I'm developing, I use Mac Virtual Display extensively. And that makes my build and iterate times super fast. It depends on what type of app I'm building, whether I'm building a predominantly SwiftUI-based app, where I would use SwiftUI previews, and just be building an Xcode and then deploying the device, or if I'm using Reality Composer Pro 3, any of those tools. Using MVD with Vision Pro is fantastic. I will also add, for me personally, if I have to remove my headset, having Optic IDE set up, it makes a world difference coming back. So if you haven't, that's one of the tips and tricks we use in addition to MVD. Yeah, absolutely. Cool. All right, next question here. Can you give us an update on automated testing for Vision OS and ARKit and RealityKit? Good question. John, you want to give that a shot? Yeah. For RealityKit, I definitely recommend trying-- seeing if you can get some of your automated testing needs resolved using the RealityRenderer class, which is a pretty useful way of driving a lot of RealityKits rendering through an automated test or programmatically in general. There's a couple of new videos, new session videos this year that I would recommend checking out in general on automated testing this year. There is a Migrate to Swift testing session, which is really nice. And there's a Get the Most Out of Device Hub session as well. You can do a surprising amount of automated testing through Device Hub. That's definitely an angle I would recommend people investing and looking into. In addition to the new Device Hub, Reality Kit is also written in Swift. So any of your existing infrastructure with Swift automated testing, Xcode infrastructure there should all work out of the box. Excellent. All right. Next question. What are the biggest mistakes new Vision OS developers make? A quoted question here, Katie. Yeah, for sure. So if you guys recall, when we announced Vision Pro, there was a little bit of a lag time between the announcement and then the actual release of the product. And in that period, we had these workshops and labs where developers could come in and test their apps that they had been building for the simulator at that time on device. And I always think about that when something like this comes up, because there were so many folks who came into those labs and were just astonished at how different it was to have their app go from the simulator to on device. Things like information density, right? Like when you have this large window right in front of your face, you don't necessarily want the same amount of text or information density that you would have on other Apple platforms. So those sort of design considerations, I think, are huge. Also, the size of your tap targets or your buttons or whatever it might be, it's just different when you're engaging with the UI through your eyes and hands. It's completely different. We have a lot of really good design sessions over the last couple years from the design evangelism team, your folks. Sarah, I think, had one last year. Linda as well. Just really great, useful content. The HIG has also been updated. But it's quite different to have an app that runs on a 2D platform and bring it into 3D. I would also say, like, outside of just a direct, you know, check-the-box deployment target update, you can take your app that is working great on iOS and bring new features to VisionOS, right? So, like, that's sort of that next step, that level up of, like, okay, if I'm making, like, an e-commerce shopping app, I might have a product display page with the information about a particular product, and maybe on iOS I have an array of images that the user can kind of scroll through. But then on VisionOS, I have the ability to have a full 3D model that represents my product. Are you using your hand? Exactly. Yeah, and then like point it out of the window and putting it onto my desk or putting it onto my floor, depending on what it is. It's such a magical way to allow a user to experience your app and experience your product. It feels much more engaging when you kind of take it to that next level and you build something that is truly only possible in Vision OS. So I think the best Vision OS apps are ones that are only available, that have features that are only available on Vision OS, right? The things that are new and novel. I think for in addition to what you said on the vision OS is let's be creative, right? There are a lot of these experiences, I think, just barely scratching. We're in the phase of barely scratching the surface of spatial computing, being able to have the superpower, understanding the physical room you're in, changing the lighting of your room, right? Being able to give you customers, your players, that special capability is like, I'm here, and now I have superpowers. And being able to explore that rather than keep thinking about where we come from, technology, right? So we also seen this transition when we first roll out the iPhone, right? We introduced multi-touch. So rather than we also seen apps going through the transition of having these on-screen virtual controller gamepad to truly take advantage of the touch surface, like what we've seen with Angry Birds years ago, right? Being able to truly innovate on the space. So I would say this is just the journey has just started, right? We are barely scratching the surface. Lots of new unique capabilities. We don't have to get limited to where we all come from previously. Yeah. And one thing, there may be just content creators or media developers out there. And there's an interesting thing that I've seen, is that when a lot of the encoding settings and visual quality that we build towards on our other platforms or for other deliveries such as television, that display is at some distance from you. And so you will optimize for that viewing distance. But in Vision Pro, you have a flexible screen that you can make as big as you want, and you can bring it quite close. And one thing I've seen quite frequently is encoding settings that look awesome. Normally, when you deploy them on Vision Pro, because it has such a high-fidelity media reproduction, that you begin to notice things like that. And this is not just in things like video content or movies. It could also be things in, like, the bitmap images that you're using in your app. They may work great in other regimes. You bring them in, and you start seeing macro blocking and artifacts. So always look for that. Look critically at your, particularly your compressed content, and make sure it looks as awesome as it can on Vision Pro. I want to take a moment and point to Norman, your session from WBDC this year, Build Next Generation Experiences for VisionOS 27. That session covers different pathways that someone can take to bring existing experiences or maybe brand new experiences to VisionOS. It doesn't just cover all the native tools and frameworks like RealityCit, RealityComposer Pro. They are great, but you're not limited just by that. There is an option for you to use Unity or Unreal or Godot. Or maybe if you already have an app on Mac or PC, you can stream those or stream some of the content using spatial preview or foveated streaming. So all of this is covered in detail in your session. So do check out that session, build next generation experiences for Vision OS 27. It's a pretty good pathway discussion there. I have one more thing to add. Norman reminded me of something. I started developing apps March of 2008 when the iOS SDK first came out, right? I was so excited. And one of the things that was, like, so interesting to me was that there was no right way to do things at that point in time, right? Like, iOS had just come out. Apple had a perspective on what was correct for the platform. But then that perspective was affected by third-party developers who started building these amazing apps that I'm sure, you know, were not even considered by the folks at Apple at that time, right? Yeah, of course. A thousand percent. Yeah, absolutely. I think we can name a few in our heads if I fit that. But I feel like very much we're in that stage now with this platform, right? Like, what makes a great spatial app, we have one perspective, but I would leave it to the developer community as well to, like, develop that perspective. And when we have these events in person, we get to see the work that comes from the developer community. Like, for me, it's some of the best times at Apple. Yeah, because you get to see, like, oh, wow, we shipped this API, and we expected it to be used in this one particular way, and it is being used in a completely different way, And it's resulted in this amazing experience that we just hadn't thought about. So again, when we ask for use cases for feedbacks, it's because we can't come up with everything. That's a good one. Yeah, 100%. Another upvoted question here. Agentic coding in Vision OS, this is regarding agentic coding in Vision OS. I tried agentic coding in Xcode to generate a Vision OS project. Just hello world entity in a reality view following a local LLM, WWDC 26 session, but it generated the app with wrong syntax. Is there any way to fix these or teach the AI to know more about Vision OS code? Yeah, I feel like I have to take that one because I was standing next to it, really. So a couple of things. One, we all know code generated by an LLM might have errors. I would think it's pretty important to be deliberate about how we're prompting things. I myself have run into this. Always double-check the output. Always double check the output for sure. Trust but verify, I would say. And then secondarily, we have a really robust library of sample projects that could be potentially used to feed as almost training data to the local LLM that this developer is using. So those would be the two things I would recommend. I don't know. You guys have any additional thoughts? I think for me is some of the LLM project I tried, it was quite amazing with Vision OS, being able to instantly create a game like Tetris using RealityKit and all the APIs was all compiled out of the box. I was really surprised how far we come along in terms of these AI development, right? But I think what Katie said is, because everybody's evolving so fast, right? We have the API SDK new capabilities coming out. Some of the existing models might be trained on the previous years of APIs. So there's a little bit of overlapping period that we might have to transition together, right? - So your LLM may vary. - That's right. - Double check. I mean, to Katie's point, we have some amazing samples. We've got Hello World, Petite Asteroids, the Canyon Crosser sample. And new in VisionOS 27, there's a few more amazing developer samples that might be super relevant to what people are specifically building. I don't remember the name just offhand. Maybe, Adarsh, you can help me. But there was a great session that was done that really talks about complex 3D object manipulation. And it includes sample code and explains how you can take a... Oh, the model manipulator sample. Thank you, the model manipulator sample. And especially like on the professional enterprise side, we have so many companies really wanting to push the limits of what's possible with collaborative 3D design reviews. And we took some of our best practices in learning, the way that we use Vision Pro internally to literally design our products and our retail stores, et cetera. And we took those learnings and turned that into an internal sample that makes it so much easier for everybody to do super complex hierarchy manipulations, right? I mean, it's super cool to be able to go in and the example in the video in the session is an AirPods case. Super cool to go in and reach in and pull a piece, a component out. But when you've got hundreds of components in a hierarchy, it's important to be able to go down and tab down a hierarchy view or do an exploded view or do cross-sectioning so you can grab the right component. So, for example, in that use case for people doing complex 3D object manipulation, providing an LLM-specific sample there where we provide best practices on here's how to build an app in this space would be probably even further helpful. Yeah, and that sample in particular uses a lot of new APIs, including you mentioned cross-sectioning. That's a new reality kit thing that's coming out. Yeah, the session you were referring to is collaborate on structured 3D models in Vision OS. Wonderful. So, yeah, do check that one out. another question probably this one's for you Matt for object tracking how fast can it get now what will be a possible frame rate and latency maybe I'll start with I was looking at the demo when this was getting made for the session and the presenter literally like spins the globe and it retains the orientation of the globe as it completes the spin and it was just mind blowing to see how fast it does But, Matt, if you have any specifications that you can share with the community here, that'll be great. Yeah, I mean, I mentioned this earlier, but we're super, I think for the first time, for me at least, like object tracking and the success of tracking has gone into a place where it's crossed the threshold. It's just magic at this point. So I'm super excited to see how everybody uses this. In Vision OS 2.0, I believe it was that we were tracking, we were primarily focused on tracking stationary objects, mostly stationary objects, and therefore we were focused. The tracking was about like 5 hertz, 5 frames per second, if I recall correctly. But new in Vision OS 27, we have this high frame rate mode where you actually can go up to 30 frames per second at significantly lower latency. And I do have to also call out the engineers that worked on this and worked tirelessly for the past handful of months. There was also this amazing change just a few weeks ago where they were able to do some fantastic smoothing on the algorithm. And things truly feel attached in your hand. We knew we'd be able to achieve that when it comes to the spatial accessories where you have the onboard IMU and the constellation of IR, the infrared emitters. But to be able to just have a device that is fully disconnected with no electronics on board that you can go like this and the 3D asset is tracked directly to the physical model is pretty magical. So we're super excited. It's gotten to a really amazing place. And also for the first time in iOS 27, we're also expanding this capability. The exact same models that you train can now also work on our iOS devices. So we've created more places to experience this. We've created an extended training mode within CreateML that enables even higher accuracy and lower latency. So a lot of improvements on this front. I will also add to what Matt said is this year we also made changes so that the object tracking updates that you get is actually a frame time align timestamp so that you will not try to guess, oh, is this the previous one? Is this the next one? Sometimes when they are not perfectly aligned, you could have a plus one, minus one frame delay. So now we are always frame display time aligned. And if you want to dig into the details of what exactly that means for your specific use case, our developer documentation have a lot of details on that. Great. Great. Next question here. The Vision Pro has awesome sensors and impressive frameworks like Persona Kit and Core IK running under the hood. Will Apple expose more of these to developers, particularly for accessibility-focused telepresence apps for disabled users working in hybrid environments? I can take that one. Yeah, the standard Apple line. We don't speak to future plans, as you folks all know. But I would encourage you, again, to file a feedback. and we read these things, we take them seriously. Particularly for accessibility. Of course. As a company, accessibility is a very central kind of principle. And we think Vision Pro has a lot to say there, a lot of capabilities that could be brought into being by developers or ourselves. So we're also very interested in understanding what that looks like in order to enable either a developer or a feature. So yeah, definitely give that feedback. Yeah. Some of our accessibility even benefit a normal person like me. You know, with the magnifying glass feature, being able to give me the superpower to be able to read. I really appreciate our accessibility team. I have to call out favorite feature number four. Oh, no! Look at you. You're stacking them up. I'm allowed to have this one because it didn't come to mind because this feature was actually announced before yesterday. But I believe it's part of VisionOS 27. And there's some amazing videos and stories on this. but we announced just a few weeks ago as part of, I think it was the Global Accessibility Day, exactly, some amazing features for wheelchairs being powered by Apple Vision Pro. So if you haven't seen that video, it's incredibly heartwarming, and it's amazing to see what a technology like Vision Pro can do to truly restore mobility and make people's lives a little bit easier. I think I missed an A. It's Global Accessibility Awareness Day. Thank you. Got to give it its proper. Yeah, cool. Cool. Question from Christopher at work. Mac virtual display seems tied to the Vision Pro's iCloud account, blocking use on shared or enterprise Macs. We want Vision Pro's to drive multiple shared photogrammetry and Gaussian splatting Macs. Is there a supported way today? And is native shared device support planned? Thank you. Any takers on this one? I'll take the first part of the question because I understand that part the most. Maybe someone can take the photogrammetry and gradient splotting. But yes, today your ability to use Mac virtual display is tied to the iCloud, so both devices do have to be on the same iCloud. There are other ways, I believe, though I do need to confirm, where you can actually do some sort of reverse airplay and screen mirroring from Mac to Vision Pro, but it's not the same. It's not as good as the Mac virtual display experience. So today those two things are quite linked. We'd be really interested in feedback in terms of where there are use cases. I know you're saying here in more enterprise and shared Mac. So I'd love to understand that a little bit more as we look at expanding the abilities to use Apple-- to Mac virtual display. I would just add that this sounds like an amazing app, right? Like being able to share a Gaussian splat model across a bunch of different devices, maybe even using SharePlay, that sounds like a get on that. Yeah. Yeah. The spatial preview functionality that we announced is not iCloud dependent. You can spatial preview to other devices. So if what you are referring to here is being able to share something that falls within a USD compliant spec and is more of the spatial preview functionality, you can do that. We also announced something last year, which is the remote immersive space. So for users that want to be able to actually handle the rendering on the Mac and stream that over stereoscopically from the Mac to Vision Pro, you can host this remote immersive space on the Mac side. And that, too, is not specific to requiring the same iCloud on both devices. Yeah. Just also echoing what Adar said earlier, is just there are many ways nowadays to be able to bring your content to Apple Vision Pro. This includes 4V streaming as well. If you have Gaussian splat content with your creative pipeline that's running on the PC, you also have that ability in addition to the spatial preview framework to bring that in the Vision OS. And on the 4V streaming stuff, one thing I want to call out is that your Vision OS app can still host a bunch of capability that is native to Vision OS, like SharePlay or pairing up with Swift UI for import and so on. So, for weird streaming doesn't necessarily have-- it's not just one way communication between the PC instance and Vision Pro. You can actually send data back. Yeah. There was a developer I was talking to the other day, and they're building basically like a pilot simulation for an airplane pilot. And they're actually rendering the cockpit itself natively in Vision OS as a reality entity. And then everything out the window, they're streaming in from a cloud server. So it's this really awesome experience Because there's certain things in, like, pilot training that maybe only require you to, if everything out the window is dark, that's okay, right? Because you just want to do your instrument controls, et cetera. But then you actually want to take off. And that's when the streaming starts. Now you can actually take off. And this blending of the content, I mean, three pieces of content being blended simultaneously. The real world, virtual elements being rendered directly on Apple Vision Pro on device, and then rendered content being streamed in from other devices, which is really, like, all working harmoniously together. It's pretty magical. And it's all sorting properly within the 3D space. Exactly, yeah. It's not like one is always at a higher depth level than the others. They can all have a shared across the entire depth spectrum. Cool. Next question here. Can Gaussian splats be part of a USDZ file, perhaps with other meshes, and used in a website environment? Travis. It's a little complicated. All right, so as an OpenUSD feature, support for what they call particle fields, which is a generic kind of form of Gaussian splat, is actually available within OpenUSD. Now, OpenUSD is the standard and open source. And then separately, there's what we've announced in Reality Kit, which is its ability to have a splat inside Reality Kit. Those planes didn't land at the same time, so they're not currently hooked up together. But that is something that Norman and I were just talking about moments before we started this show. So obviously, that's something that in the future we're looking to close that gap. And so that's just kind of an artifact of standards development and process and then OS development process. Sometimes the trains, I'm using transportation analogies all the time. The trains don't run on the same time. I think this year we added the fundamental capability of understanding what a splat is within re-advocate technology glossary. But now I think the next step is when we look at the harmony of expanding the Gaussian splat ecosystem, we are going to think about the end-to-end story. That include what is the standardization, like what Travis said in AOSD, Apple is playing a really big part in driving that standard. We drove actually particle fields. That's right. It was Apple and a couple other companies, I believe Adobe and others, were pretty instrumental in driving that through. It's just standards process runs at one clock rate, and then Apple runs at another clock rate. Yeah, so we want to definitely look at the alignment between the specification to the runtime capability and to the authoring story as well, right? Because there has to be a way for you to capture the object you wanted to capture in splat format and be able to store it and be able to have the runtime API to display it. Yeah. Yeah. And USD is able to express multiple different things in a scene. And so their question they're asking, can I mix and match? I mean, yes, you can contain them in the same scene. That's right. Excellent. Talking about landing planes. From a period. Pretty much. We're out of time here. But I wanted to say that, like, on that particular question, it seems like a very interesting use case, and we would love to hear more. So please do share that feedback. But we are out of time right now to go into deeper discussion there. Thank you so much for all the questions that, amazing questions, I should say, that all of you brought to us today. I really appreciate the enthusiasm and the passion for the Vision OS development, and the community brings in so much excitement and is helping us make this platform better year after year. Thank you so much. If we couldn't get to your question, please don't be disheartened. We have an active team that's looking at forums this entire week, and it also keeps an eye out through the rest of the year as well. But please do post your questions on forums, and I'm sure you'll get answers to your questions. If you have feedback, I think that was something that was mentioned many times during our conversation. Please do share that with us as well. It's your feedback that makes this platform awesome. Thank you so much for tuning in and thanks to the panelists as well as the team that has been helping us answer your questions behind the scenes. We really enjoyed this session and we would look forward to what you build on Vision OS. Enjoy WBDC. Thank you. Thanks.