Good morning. Welcome to the Apple Intelligence Group Lab. We're so glad to have you here. My name is Ernie Svela. I'm an AI machine learning evangelist here at Apple. There have been some awesome announcements at WWDC for Apple Intelligence, Foundation Models Framework, private cloud compute, visual intelligence, and a whole lot more. And of course, App Intense and Siri. I am joined by an awesome team of engineers who are going to help you with those questions. And why don't we begin by some quick introductions? Yeah, I'll start. My name's Matt, and I work on the Foundation Models Framework. Hi, I'm James. I work on AppIntense. And I'm Rob. I work on the Evaluations Framework. I'm Louis. I also work on the Foundation Models Framework. I'm Dan. I work on Siri and AppIntense. Awesome. Thank you all. In addition to those you see here on stage, there's a whole slew of engineers behind the scenes working to answer your questions. Please submit those questions. If you have a code-related question or issue, definitely connect with us on the developer forums. That's a much better place for us to get into specific coding issues, but we really look forward to answering your questions today about all these exciting new updates. And if you have a bug or feature request, of course, go to feedbackassistant.apple.com, And, yeah, we'd love to get your ideas and feedback on any bugs. Although, I don't know, bugs in our software? I haven't seen any. Anyway, all right. We have no bugs. Not this year. On occasion, that might happen. Anyway, with that, we have a lot of questions. Let's get started. The first question we have today is from Mario Gravina. Thank you, Mario, for the question. The question is, what are the most useful Apple resources for learning and applying the new version 27 APIs and design changes in real apps, updated sample projects, migration guides, et cetera? That's a really good question. Thank you for that, Mario. Yeah, well, of course, there's WWDC videos, a lot of great documentation, a lot of sample apps. Any other suggestions, you guys? There's a dedicated sample section of the documentation page. Yeah, and there's definitely a lot of great technical articles as well. So a lot of great resources. Yeah. I think on App Intents this year, I think we had five sessions. Most worked on a few, like there was a few different sample apps that we put out. A lot of the sessions used them in the talk so you can kind of see how they were built, how they're used. We also did a big push on our documentation this year. I know for App Intents, I think for Foundation Models as well that really show, we built a lot this year. Yeah, definitely. And to kind of how to, what you want to understand it, the docs are not just, here is the API, it's how to use the API, how it benefits your users. Yeah, I think you made a good point. Like if it was in a session video, it will show up in that sample section. Yeah, no great points. And obviously WWDC from last year, previous years, lots of good material too, especially if you're starting out like say with App Intents. A lot of great foundational material as well. So definitely check that out. So awesome. Thank you, Mario, for that question. Next is from username LukeSafe. Hopefully I'm saying that right. Nice name, by the way. My app doesn't fit any of the predefined schema types. I think we've heard this one a few times. Is it better to try to use the closest related types, or should I stick with not supporting the schema for now? My app is for tracking time. There are types for timers and for calendar events, but they don't quite fit within my app offers. That's a very good question. I can take that. So new this year is the introduction of app schemas. These are schemas that are fine-tuned and designed by Apple to integrate with our great system experiences. The cool thing about them is that as a developer, you can simply pick and choose the schemas that best match your app's capabilities. So you mentioned, you know, timer functionality, calendar. We have schemas for those actions, and you can simply pick and choose the actions that match your app's capabilities. If your app has functionality that is outside of the schema, well, you can also combine it with custom app intents that you can write for those actions. And when you write an app intent, that action becomes available in many places across the system, the shortcuts app, Spotlight, and more. Now, it's important to know that for the new Siri AI integration, for integration with that, you do need to adopt one of our app schemas. Now, the other cool thing, too, is a lot of our schemas are sort of designed for certain specific types of actions. But we also have a domain called the system domain, which is more generic and broad. So if you have functionality in the search space, for instance, apps like e-commerce or maybe food ordering, well, you can integrate with system.search. And what that will do is it will allow your customers to use Siri to basically integrate directly into your app's search capabilities. So you can say something like, you know, order me a bicycle on my favorite marketplace app, and it will deep link to your app directly into your search capabilities so that you can continue your ordering flow through your app. We're really excited about this. There's also another scheme in this domain called system.open that allows you to open specific entities that can be donated to Spotlight, and Siri will be able to open those entities directly in your app. And if none of these schemas satisfy your needs, well, we also have App Shortcuts. James, do you want to talk a bit about App Shortcuts? Sure. So App Shortcuts is an API that we've had for a while. It allows you to add phrases for an app intent. And when you do that, the action becomes available in the system. So it's automatically available with Siri, Shortcuts, Spotlight, and a few other places. You can even use it on the iPad if you want to tap on an Apple Pencil. It all works with app shortcuts. And it's a great feature, as Dan mentioned, when you have a specific action in your app that you really want users to be able to do, and it has a memorable phrase. So app shortcuts does require a phrase, but for these very specific actions, these kind of hero use cases in your app, they are super impactful. And then, yeah, Dan, you mentioned a bunch of other things. We have a ton of APIs. We have a ton of different experiences. There is, it's kind of a choose your own adventure. Start with the schemas. If something makes sense, use it. And then there's a ton of other experiences as well. And the documentation this year is great for finding out what functionality is available through app schemas, also how app shortcuts work. Please check out the developer documentation to find out how you can integrate with Siri. That's awesome. Great. All right. Well, thank you. Great question. Moving on to our next one. This is from, gosh, let's see. Toic. Hopefully I'm saying that correctly. The message or the question, please, is my apps core entities and actions don't map to any predefined schema domain like the apps or like the apps of many others. What's the best way to bring them to do agentic Siri today? What should I do now to be ready for as the platform evolves? And how do you how should we think about their place in the agentic experience? Good question. I think we covered a little bit of that. Yeah, you know, I think we have these, take a look at the schemas, see if, as Dan mentioned, there are, you know, you can kind of pick and choose parts of these, you know, domains. If something makes sense, implement it. By conforming to an entity schema, I mean, kind of donating and interacting with spotlight, you do get a lot of other benefits. Even if maybe one of the intents doesn't make sense in your application, you still can use Siri AI to ask questions about the content of these or take action on it using any or any view annotation API. So there's a little bit there. Right. Yeah. No, that's great. Thank you, James, for that. All right, next question from Indigo J. Another nice username. App Intense exposes an app's actions to Siri Shortcuts, but there's no API for a third-party app to invoke another app's intents directly. Siri Shortcuts is the orchestrator. Can an app act at its own orchestrator and drive other apps, or does cross-app action still have to route through Siri Shortcuts? That's a very good question. Thank you. So new this year, we have a system orchestrator that can take action from App Intents from many different apps across the system. Now, one of the important things for us to consider with this is when it comes to privacy and safety. One of our key concepts of Apple Intelligence is that it is AI, personal, and private to you. When you integrate with App Intents, you're basically making actions from your app available to the system, and you're also modeling content in the form of app entities. Now, the cool thing about our semantic index and our spotlight integration is when you index content from your app, you're making it available to the system and to the system alone. So, you know, it's not like you're sharing data from your app that could potentially expose other apps from accessing this data. It's all sandboxed to you and the system and the system alone. Now, there is also a concept this year called transferable. Transferable is an API that you can adopt as a developer that allows you to control how you can share data from your app to other apps in the system. The API essentially has bidirectionality support, meaning you can choose which data you want to export out of your app and which data you want to import from other apps. So it's a really neat API that makes the boundaries clear between what is data that your app owns versus data that you can import from somewhere else. Excellent. Yeah. Great stuff. Yeah. Great year. A lot of great new stuff. Great question. All right. Next one. Tanner MDT. And the question is, with the new Siri in iOS 27, can my third party app take part in a multi-turn conversation? Another great question. Or am I still limited to the discrete actions and entities I expose through App Intense? I could take this one. The answer is yes. Please. This is what the schemas provide. You can take action from your app. You could ask questions about your entities. You can do follow-ups. It is, I've been working on it. I'm very happy to finally show it to people. I want people to take a look at the betas, try it in your app. it's a great way that we believe your users will be able to interact with your applications. You know, it's your app, it's your content, they have kind of a deep understanding of this, and to just be able to ask natural questions about it and do these follow-ups, it's just really awesome to see. Yeah, and I'll add on that, you know, at the heart of it, you have a large language model that is processing all of the input, so it gives users, you know, more natural capabilities for how to express and take actions with your app. Previously, the commands had to be very specific to form into a shape. Now, with app schemas, you have a lot more flexibility into how your users can phrase requests and have our model sort of parse that information and pick the right action for your users. Yeah, great stuff. Great videos, by the way, from this dub dub on these topics. So definitely encourage you, of course, to watch those as well. Yeah, a lot of great, great sample code to go along with that. All right, well, thank you for that question. Moving on, Zolfi Shah, I believe, is the way we say that username. Is it okay to mix and match non-schema app entities in tents with some that match different schemas from different domains all in the same app? I'm loving all these schema questions, by the way. The answer is absolutely yes. With schemas, our intention is that you pick and choose the schemas that work for your app, matching your app's capabilities. So if you have an app that can do calendar actions, but it can also do reminders actions, please pick schemas from any of these domains. The cool thing is that you can mix and match them, and they can work together to deliver full holistic experiences. Yeah, and it's also, this is not a binary choice where you have to use schemas or you have to use app shortcuts or search in app or something. They play nicely together. There are going to be some experiences. You're the developer of your app. You understand your users. And you should be, we expect people to have little bits of a schema, little bits of app shortcuts. It really builds this holistic experience for users to use your app. Yeah. Oh, that's great. All right. Moving on, next question from ABhi27. Not sure how to say that one. Is there a practical time work budget for foundation models during one background wake? If no fixed number, how should apps safely chunk work? Yeah, I can take that one for sure. So with the on-device system language model with foundation models, If you're running in the foreground, your app, there's never any rate limiting or anything like that, unlimited requests, right? If it's running in the background, we do apply some throttling if the system is under heavy load. And so what you can do with the API is catch the rate limited error. So if you're getting the rate limited error with the on-device model, that's the way that the system is telling you that your request, even though it ran on-device, that the system was under heavy load and you should try again later. Okay, that makes sense. You know, I have to say, I was so excited when I learned we're going to be making PCC available to developers. Oh, yeah. It's an awesome model. Yeah, anyway, that's really great. We'll talk more about that later, I bet. Awesome. All right, moving on. Also from ABHI27, the question is, what exact air types codes indicate foundation model throttling? Okay, I think you may have addressed that a little bit in the previous question. system busy, thermal pressure, quota exhaustion, which are retriable versus terminal. Is nuanced air reporting available? Yeah, I think, Matt, if you want to talk more about our handling in general. Yeah, for sure. So this year we came out with the language model protocol. And as part of that and kind of encompassing common errors that a lot of models will see, we have the language model error. And so there's a lot of those common use cases of rate limiting and refusals, those sort of things. But specific with the PCC model, like you guessed coming up, we also have quota usage. So you can get a signal through the property on that specific model saying, hey, are you getting close to that limit? Are you not? And we also have errors a part of that that will throw specifically for that model saying, hey, during this prompt, you have a quota exhaustion and you can handle it from there. Yeah, that's great. Yeah, so you can find this in the documentation as well? Documentation. Yeah, Levy has a great session. I'll plug that real quick. So, yeah, I definitely check those out. They're really great. Yeah, it's great to hear developers bringing those kinds of questions because, you know, it's the rainy day situations you've got to prepare for in your app too, right? Not just when everything goes well, right? So, yeah. Great. Thank you for that question. All right. Next one from Thub, T-H-U-B. Hopefully I got that right here. How to avoid language model session dot generation error dot guardrail violation. In my tests, I am getting quite often guardrail violation errors for prompts I cannot see any issues with. I'm thinking the evaluations framework. Yeah. Yeah. We ran into this a bunch when we were building samples for ours. There's a model judge evaluator in the evaluations type, and you can create safety rubrics inside of that where you just give it a scale, like let's say one to four. How safe is this? Does it have some safety implications? But if you actually ask the model, you know, or even ask, you know, a different model, let's say if you're, you know, you're getting this a lot in the on-device model, you could use the PCC model to kind of check that it being a larger model. It could kind of evaluate those and let you know if there are safety errors. But the guardrail violation is there to help you avoid unsafe responses in your app. No, great. Yeah, great advice. And I want to put a plug in for the evaluations framework. Oh, my gosh. It's one of the biggest questions when we would do workshops was about how do I test my intelligence-based features? And I have to say what we've done between the model judges, quantitative, qualitative metrics, it's really, really, really great stuff. I don't know of anybody else doing that. I'm a fan, but. Yeah. A little biased there. No, no, definitely check it out. And thank you for that question. Really, really appreciate that one. Okay, moving on. Tainard NDT, in iOS 27, can the Foundation Models Framework reason over video or work with a video model? Or is multimodal input limited to still images? That's a great math. That might be a good question for you. Yeah, I'll take that one. So back to that language model protocol. So as models and people come to the Foundation Models Framework, we have them have capabilities that they support. And one of these is vision. So if you add that as a part of the model, the framework can handle to say, yes, this is approved for this specific model. And if so, it can continue on and do that sort of thing. Specific to reasoning in Apple's models, the PCC model has that where you have the capability to specify light, moderate, and deep, and reason over those things. There's another great session talking all about that where we have a new part of the prompt where you can have an attachment and bring in those images that you want to look with. And we've shown that with my dog, Sully, which has been great to test with all of us and see that the model performs very well. Yeah. So yeah, those are the sort of things that it can do. That's sort of the key thing that we're making the framework, the foundation model framework, extensible this year, right? But also this year, we added support for image input, both with the updated on-device model we have, and then, of course, with the new server model running on PCC, right? But in terms of the extensibility, with the language model protocol, letting you really bring any model you want to the Foundation Models Framework, and the protocol itself is sort of open-ended, where if you have custom modalities, so for example, like video, that we have the custom segment API that you can really sort of build whatever things you want on top of it. - Yeah, the framework doesn't kind of hold you into, you have to be within these certain things. If your model's very specific, like it can smell something or crazy, You can use a custom segment to still reach the users. And with it being open source also announced, you have even more capabilities where you can run on the server with Swift and those sort of things. If I may. I mean, that was a lot of great information on foundation models. But again, if you're looking for a one-shot approach, I think everything they said kind of makes sense. But if you're willing to do processing ahead of time or if you want to build a tool, You could build a tool that uses Apple's other frameworks that could get a transcript of that video, maybe what it's hearing, audio describing it. There's a lot of good frameworks in our API. There's a lot of good APIs in our frameworks that allow you to extract those. And then you can just feed that text description. And at that point, I mean, if you're doing summarization or something like that, the on-device model is great for that. JOHN MCCUTCHAN: Yeah. There is a great what's new in image understanding with the vision framework. Oh, yeah. It's really awesome. It's a very good one to check out. Yeah, and it's like you build a tool, and it's like the tool does that processing, and it feeds it into the model. Yeah. No, that's really great stuff. I mean, the open source, oh, my gosh, that was such a huge announcement. I'm, like, so excited to hear that. Absolutely. For the developer community, right? I mean, yeah. Apple's behind that. I mean, that's huge. But the extensibility piece, yeah, that's really, really powerful as well. So, cool. And, hey, another quick plug for valuations framework. I know I saw a sales guy. You mentioned tools, right? But, like, you know, one of the problems I always had is, like, what tool got called when? Or, you know, now with evaluation, I'm able to see that. I mean, yeah. Oh, my gosh. Yeah, you feed your evaluation the transcript from the language model session. And then we have a tool call evaluator that will allow you to build assertions against that transcript and see when it's getting called, what values are being sent to it, et cetera. And if you're a visual, I like visuals. We have instruments now hooked up to also see as it goes. So you can really see everything that got called. It really helps. So you get kind of like batch evaluation, and then you can get kind of real time what's going on. Yeah, that's really cool. All right, well, moving on. Next question is from Kyle BSHR. Are app intent schemas something that give your intents an advantage when integrating with Siri, or can any intent entity be discovered by Siri? Siri AI the same way. Notably, there's no finance schemas. That's a very good question. We've heard this one. Yeah, I could take this and Dan, I'm sure you'll jump in here too. I think advantage is probably not the right way of thinking about this. You know, using schemas is the way that Siri AI can reason about the actions and entities in your application. It's how, and Vantage is really throwing me here. I think one similar question we get about this is how can you influence Siri? And for that, we have a ton of new APIs that can help you tell Siri about what actions are happening in your application. So we have new intent donation, like a new intent donation system that's used for when a user is using your app. You know, they're tapping around. If a user performs a similar action that has a corresponding app intent, you donate that intent. And that gets donated to a kind of temporary transcript that Siri can use to learn. So if you're kind of consistently messaging the same person and then you use Siri to like, hey, message Dan, it will know, Siri will learn that, oh, you might want to use message Dan in unicorn chat or, you know, or many samples, space themes sample apps this year. Yeah. Yeah, also add on that, you know, we have the App Shortcuts API that allows you, developer, to specify the phrases that can activate your app intents. But if you adopt one of our app schemas, the cool thing there is that Apple has done the heavy lifting of essentially providing those sample phrases, of training the model and making sure that we can deliver a consistent experience across apps when you integrate with a schema. So if you're messaging using the first-party messages app or you're messaging using many other third-party messages apps that are available out there that do conform to our schema, you get a consistent experience no matter which app you use. And that, I think, is really powerful about what we're building here is that we're building an ecosystem where users can use Siri AI to take actions across many apps in the system and tailor the experience to your users. You may message a coworker using a very specific messaging app. And so when you send a message to that coworker, the system will learn about these integrations, like James was saying. And so it won't even try to use iMessage to send that message. It'll use that app that you always use to communicate with your coworker. Just one more small plug. We know we have a lot of developers listening. And building app intents even takes a little bit of work. you have to define a lot of your properties. There's a lot of code that you have to write, which we all write code. By using app schemas, it actually lets you write a little bit less code for this. Because as Dan mentioned, Apple has done a lot of that heavy lifting for you. So some things that you have to provide in kind of custom intents, you no longer have to provide because we do it for you. So it's another developer benefit as well of hopefully all the user benefits. Totally. You know, I have to say, too, like going to App Intents workshops, working with developers who are new to App Intents, and when they get it, that aha moment and seeing how they're like, oh, wow. It's like, you know, you converted my device into my, you know, the app knows, the device knows so much more about my app now. And just seeing that light bulb turn on, yeah, it's really exciting. And then when you add all the new capabilities, it's pretty, it's fun. And the cool thing, too, to see is when they add the schema to their app intent, they get to delete a bunch of code because that's no longer needed. And developers love deleting code. I love deleting code. Especially when it's not mine. Awesome. All right. Well, thank you for that question. Next, moving on. Interferon. Oh, that's a cool one. Okay, cool. Username here. Can I use Image Playground to generate photorealistic images inside my app? I think I can take this one. The short answer is yes, you can. A lot of great new additions to the Image Playground API. There's a really great video this year, this dub dub. And basically what you do is you'd use the ImagePlaygroundStyle.all as a parameter there. So, yeah, please check that out. great video, and I'm sure there's some great documentation on that as well. Because that's running on private cloud compute now, right? That's right. Images, yeah. Yeah, you get the advantage of the more powerful model and the API itself. So great, great. Thanks for that question. All right, moving on here. Lewis 42. Hi, Lewis 42. Thank you for your question. Where's Lewis 41? Yeah, that's what happened. Yeah, where's Lewis 41? Lewis 41 in the house. Is he all right? I'm confused by the difference between indexed entity and things with defined entities. Example, app schema.books entity. Should we use index entities for everything that isn't a book, calendar event, etc.? Or are these serving totally different purposes? Great question. That is a great question. So the schema defines the shape of the content. It defines the properties that it can have. So if we're talking about a book, it can have the content of the book, the title of the book, the author, and so on. Indexed entity is a separate API that essentially allows you to index that content into the system's semantic index. So the content can essentially be retrieved by Siri AI when users are performing actions like, I want to read a book by a certain author. This is the API that allows the system to essentially look into that index and find the right entity that represents that specific content. So in a way, they actually complement one another. If you adopt an app schema and you also make it conform to the indexed entity protocol, and you then make that call at runtime to essentially index content from your app into the semantic index, you get the best Siri AI experience for your customers. Back to the building blocks, you can use index entity as well for your custom entities. And you can donate them to Spotlight. The advantage there is they show up in Spotlight. And you can have a system.open intent. So if you go from Spotlight and you tap a result that's in there, you can kind of deep link directly into your app. Another kind of advantage of the schemas that I mentioned before, the deleting code, is if you've worked with Spotlight before, you have these indexing keys. So the title of a book is the display name. For app schemas, we've done all that work for you. So you don't have to worry about it. You use one API to donate, and we handle everything for you in the back end. Yeah, that piece actually is really nice because when you're using the app schema, you don't have to worry about trying to figure out what's the right key value pair for Spotlight. Like Apple does the heavy lifting for you. As you, developer, you have to basically focus on the minimum amount of adoption needed, which is, you know, you define your app entity, it conforms to schema, and then you simply provide values for those properties, and we take care of the rest. Yeah. No, great points. And, you know, to plus one Dan's point about we've done the heavy lifting, I mean, oh, my gosh, the engineering that went into those domains, right, those schema, I mean, it's a lot. So you're really getting advantage of all that engineering effort from Apple, right? So, yeah, really, really great stuff. Well, cool. Well, thank you for that question. Moving on. Username, let's see, ezfranca2015. Thank you for the question. I love the evaluation framework session. Rob, plug for Rob here. Did you write this question, Ernie? Yeah. He paid me 20 bucks. Hey, you slip a few bells. It's both a professional developer and an academic researcher. Oh, well, thank you for being an academic researcher. I'm curious whether Apple envisions the framework being used for academic research, such as collecting metrics, generating evidence for peer-reviewed studies in addition to product evaluation. That's really interesting. I think there's a lot in what kind of academic research you could do. I think what it comes down to is what it is you're measuring. The evaluations framework was designed not just for evaluating language models, but really any stochastic system. So, you know, if you have a machine learning model, a more traditional model like linear regression or classifier, evaluations can absolutely be adopted for that. It's just a matter of implementing the framework. We split our metric collection into a couple of phases. So you iterate over a series of samples, and you feed that into kind of the function that is generating the model's response. And then you use a series of evaluators that will calculate those measurements. So if this is something that, you know, the metrics you want to collect is something that can be measured in those evaluators, and it can be scenario-based, so you're feeding it in a collection of input samples, then I would say yes. It doesn't matter whether you're doing product development or you're doing research. You could use this. But, you know, we modeled this over, you know, a lot of model providers have benchmarks that they submit. And they say, you know, this model performs this well on this benchmark. This model performs this well on this benchmark. We wanted you to be able to create your own app-specific or scenario-specific benchmark specific to what you're trying to do, whether that's research or product development. Yeah, no, I totally, and I have to say, oh, yeah, just to say really quick, the three sessions we have, right, you get the introduction, you get the more advanced features. I think the synthetic data capability we provide is freaking awesome. Yeah. And then the hill climbing, right, like how do you incrementally keep improving things? So it really is kind of a nice three session set. Yeah, I was going to say that Xcode integration, too, showing visuals also really helps maybe to provide that kind of academic side where you can kind of see graphically everything that's happening as well. That's actually great. And you can save those reports. You can export them. They're all JSON-based data structures. So if you want to feed it into another workflow. But, yeah, there's a lot of work going on in that academic research today specifically. So if it's non-deterministic output or responses to a system, then evaluations is a good fit. Yeah, that's great. Great question. Thank you for that. All right. Next question is from Lee Shuang Kwan. Hopefully I'm saying your username correctly here. This year, Siri can understand screen context. Can Siri also understand context on CarPlay screen when connected? Also, can Siri platform actions like play a song based on user voice input, like Siri play song on row two, column one? Does this also apply to CarPlay? Well, that's interesting. We haven't really talked a lot about on-screen context today, so I'm excited to talk about that. It's one of the three pillars of the new Siri AI is Siri can understand the content that's visible on your screen and take action based on that relevant context. It's really powerful. We have great APIs for that. We have integration with NSUserActivity. If you know of the old NSUserActivity API that's been around for a while, we integrate directly with that. But we also have new view annotations API so that you can annotate content that is on screen with your entity conforming schemas. I'm just really excited for people, for developers to see this work, for users to see this work. It's just cool. It's like you build a SwiftUI view, you have your entities, you attach them, you start speaking, and it does exactly what you want. It's just we have sessions. That's why they're in there. It's just really, really powerful. It's natural. It's kind of a natural way of interacting. Yeah, totally. And what about the car play part of that question? Yeah, I was. No. Sorry. If you have specific things you're looking for, we always take. You can file bug reports for your enhancement requests. Yes. But I will also say that on the car, you know, we want to make sure that you're focusing on the road, on the content on your screen. Yes. Totally. Yeah, no, that's a good point. But yeah, definitely, you know, for if you find a bug, OK, we have bugs. By all means, file a feedback request, but also enhancements. You know, we'd love to hear from you and we take that very seriously, your inputs from the developer community and how we prioritize new features and work. So, yeah, please take the time and file that about requesting that support for CarPlay with some safety features, of course, like Dan mentioned. Got to keep those hands on the wheel, you know? And don't follow the feedback while you're driving. Yeah, do that while you're driving, please. All right, next question from Chaz K. Are we going to be able to use App Intents with HomePod since currently there is not a beta for HomePod? The new Siri AI is available on iPhone, iPad, Mac, and Vision OS. It is not available on HomePod. Not on HomePod, okay. Yeah, but your existing... Your existing app shortcuts have worked on HomePod for a while, so we still have options for you to call into your app. Right, right. No, that's great to know. I love my HomePod. Okay, thank you for that question. Next is A. Harris-Crown. The message is, can we expect watchOS responses to App Intense to match up one-to-one with iOS? This is interesting. I think yes, the answer would be like, we want to make sure that you're kind of testing these different experiences. Dan, I don't know if you have anything else to... Yeah, I'll say, you know, the cool thing about the new Siri AI is that it's available on many different platforms. And we want to encourage developers that when you're adopting our new or old API, so you're making sure that you're testing your experiences across all the different devices that we support, including AirPods. You know, a lot of people forget about AirPods. And, you know, when I'm on the run and I have my AirPods on, Siri is extremely helpful to perform quick actions. And so we want to make sure that developers are leveraging the full capabilities of the Apple ecosystem. Yeah, I feel like it's a somewhat less used API that we provide in AppIntent. So when you perform in AppIntent, you can optionally provide some dialogue. But that dialogue can be changed depending on what we call it, full and supporting modes. So, you know, you might be in a – when you're using your AirPods, you don't have a screen in front of you. You might want to be a bit more verbose where you're on – you know, you're using your iPad and we have a nice snippet and you might want a shorter version of the text. So those APIs are all there. And it makes sense. You know, you might need a little extra content. Yeah, and, you know, App Intense testing framework this year is out. So for everything we've been discussing about App Intense – I was hoping for some questions about App Intense testing. Yeah. Yeah. Great. I also compliment that we have a great session this year that allows you to focus on the customizing the Siri experience, like intent dialogue, like James mentioned. Highly recommend to check out the sessions by our coworker Antonio. It's a great session. Yeah. Yeah. A lot of really great, actionable, good toolbox kinds of tips and tricks in that session. So really good stuff. Okay. Moving on. Username. Epplegax. That's interesting. All right, Apple Gex. I want to say Apple Geek. Anyway, all right. Does Image Playground now require an internet connection for PCC, or does it fall back to the previous model gracefully? Well, I can take this one, actually. Image Playground always requires an internet connection at this point to generate images because, as we mentioned, it uses PCC, which is our server-based model. yeah, it will not fall back to an on-device use of the model. So thank you for that question. All right, next one here is from MCRich23. What are each of you most excited about regarding the updates to Apple Intelligence? Wow, well, thank you. We can spend an hour on that one. Although a lot of us, you know, having worked on delivering those features, we're probably like, oh, my gosh, yeah, a lot of hard work here. We're going to go down party lines. Yeah, exactly. Right. But yeah, any, yeah, do you want to want to share what you're most excited about with Apple intelligence? The one that I was really excited about was the, like, intelligence features inside the phone, like when you're on a phone call. Like, I think the example they showed was the flight. Oh, right, right, right. I'm always struggling, like, what's your confirmation number? And you're going through the mail, and you're like, oh, I can't, you know. And having that right there, I was like, that's just natural, easy way to have that. I like those little features where it's so powerful, but a small little thing. Yeah. For me, I mean, two answers, so don't yell at me. I've always been doing a lot of App Intents and stuff, but it's been so great to see kind of some of our partner teams, like all the foundation models work, the evaluation framework. It's also cool, and I was a little less connected to that, so I have a long flight home, and I'm going to be watching a lot of dub dub videos. And you can find your number, right? I can, I can. But the one thing is App Intents testing. We haven't talked about it today. App Intents testing is a new way to kind of unit test your App Intents. So traditionally, you might build App Intents and we would say, you know, you want to test it with Siri, you want to test it with shortcuts. But what was missing is a way to ensure that this functionality was working in kind of your CI pipelines in any type of automated way. So Venkatesh has a great talk this year on it. But for testing your App Intents, testing your entities, the queries, integrating with Spotlight. There's a ton of features there. We're really excited for people to dig into that. And it's not just unit tests, right? You can test UI integrations. You can test Spotlight integrations. It's a really valuable framework. I love Invent Katesh's session, how he's got the spacesuit. You've got to watch the video. Yes. Really cool spacesuit. Spoilers, too. You've got to watch the whole thing. Don't just go to the end. Yeah, just see the whole thing. A lot of good stuff there. So, cool. Robin, what would you say? I'm going to borrow from James here. It was really interesting to see what a lot of teams did. Foundation models with kind of bringing in different language providers. But then also to see that integration go out to, say, core AI. And now I can just bring in any models from the web. But it's like I could still use foundation models. It's like this central API that's really neat. And then, yeah, I'll plug evaluations just a little bit. And if you're using it with foundation models, it's really easy to bring in evaluations because, like James was saying, testing for these things is really a nascent thing. A lot of people talk about it. A lot of people don't know how to do it. And I think we've taken a lot of steps this year to kind of say, all right, how can you think about this feature instead of just assuming the model is going to work? Lay it magic. That's great. Louie, what would you say? Let me guess. Foundation models. You've mentioned some of the other things already. So yeah. And we talked about image input support for the on-device model. But the big one, I think, is private cloud compute, right? That developers now get access to. Because we know people have been asking for it. You've been asking for it since last year. And so yeah, this year, new server model running on private cloud compute. You can easily access it through our API. And honestly, this is the perfect time to talk about my new book, actually. Oh! If you don't mind. - Yeah, I did it all. - Don't worry, I'll explain, I'll explain. It's not a real book, but just to talk about the benefits here. With private cloud compute, the real thing is that you don't have to worry about API keys or anything else, or authentication, things like that, like with typical server models, right? - Right. - And so, this is a joke, but it's real. - Yeah. - Where it's just one line of code to connect to a server model now, right? And so, by the way, in the video that we have, I'll do the rule. - Yeah, you can throw it here, man. Hoorah! That's the drive to a point home that, like, you know, it's easy. You don't have to worry about API keys. Right. So I think everybody back at the house just freaked out. They were like, oh, God. No books were harmed in this. No, okay. But, yeah, so there you go. Yeah, that's great. Dan, let me guess. Siri? I was going to say that's a tough act to follow. Did you also bring a book? Yeah, I did not bring a book. I missed a memo, apparently. But yeah, no, I am really excited about the new Siri AI. I mean, we have done a lot of great advancements to Siri, and we have a lot of great APIs this year for developers to adopt and bring Siri to their app. I mean, I think it's just really exciting, and I can't wait to see how we'll eventually get to this ecosystem where you have all of these apps that are deeply integrated with Siri and Apple intelligence, providing content to the semantic index so that people can ask natural questions about content from all these different kinds of apps and get answers instantly. And then when you integrate with Intent Schemas, Siri taking actions on a variety of apps across many different domains. I'm just really excited to see sort of like the bicycle for the mind. You get this ecosystem that is deeply integrated and Siri taking action across many different apps. Yeah, I want to change my answer to that, actually. it's cool to see also like i'm a little far removed but how much work you guys have put into that and yeah but how natural it becomes when you're the end user it's just like amazing to me i think that is so powerful large the illusion of simplicity from a lot of folks yeah definitely but i can't wait for all my favorite apps to adopt my apis and then just be able to have siri you know perform all these actions absolutely very excited incredible and i'd say for me all of it is really exciting you know one of the great things about being an uh ai machine learning list is you get to see all of these great things, you know, and get exposure. So I'd say kudos to everyone here at the table and all your teams and the people behind the scenes who made all these features happen. It took an incredible amount of work. So, yeah, kudos to you all. But great stuff. All right. Well, thank you for that question. Next one is from Gerald A. Let's see. Gerald Dash, go to capital A, like he's an A student? No, anyway, sorry. They're all A students. Thank you for your question. Games will be anonymous. And the question is, why, again, going for hard-coded schemas instead of choosing a dynamic approach like GPT or Claude are taking? I'm thinking about Markdown describes skills that can reference app entities. Wouldn't this be much more flexible and solve a lot of the other questions mentioned? It's a great question. That is a great question. I'm loving all the schema questions. Keep them coming, by the way. The cool thing about what we're building here and the ecosystem that we're going for is that we're striving for consistency and privacy. Apple's take on Apple intelligence is that it's personal intelligence and private to you and the people who use it. And so with schemas, we're able to effectively guarantee an experience that is holistic but also standardized across the platform. I think one of the things that is cool about it is, if you're using Siri with an app that has adopted our schemas in one of the domains, like I'll just pick messaging, for instance, when people interact with Siri in the messages domain, they'll get an experience that will naturally transition to other apps that adopt schemas from that domain as well. This way you get a consistent behavior and interaction with the system in a way that's like, once you learn how to interact with Siri in a specific domain, you will just feel natural integrating with all these other kinds of apps. And that's what's really cool about the experience. And also, you know, schemas provide sort of like a security benefit as well, which is like, you know, in certain situations, if you're sending money to somebody, for instance, you probably want to confirm before you're actually just having, you know, Siri send money to someone. And so with schemas, you get all these security features built right in so that you get a consistent, safe experience on our platforms. I think Antonio does a great job in his talk describing this around our kind of new, we have a new kind of entity ownership API. You know, for certain, you know, items in your app, certain entities, they might be shared with other users. You know, a calendar might be your own calendar or event, but it might be shared with folks. And by using these schemes that we know about, that we can reason about, you might get different behavior. Where if it's just your event, maybe it gets deleted. But if it's shared with other people, then maybe we want to add that confirmation. So it's these kind of things that we really feel like is this powerful, where the scheme is give us this power. It's all those edge cases that we've talked about, right? Yeah. We want to provide, for a lot of folks, this will be their first time integrating with these types of systems, building this way. So we always want to be as safe as possible as we're building these APIs. Yeah, that's how you build trust and safety, right? That's key to it. Neither of you mentioned localization, but I know a lot of the reasons we do this is to kind of integrate in support for localizing this in other languages. I know we haven't talked explicitly about plans, but I don't think it was surprising when you didn't know we're trying to get this out to different. Yeah, that's the power of the API, right? Like Apple has done the heavy lifting in terms of providing all of the model training, the natural language strings. And so when you're talking about localization, Apple can scale all of these integrations across many different locales in a way that you develop or you simply adopt our API and we take care of all that heavy lifting for you. Yeah, we will make we will try to make this better. You know, we're going to continue to make this better. Like that's that's our that's the promise there. Yeah. No, great stuff. Great stuff. All right. Well, Gerald-A, thank you for that question. Next question is from Nia Murphy. Does the evaluations framework have a way to see if my tools are being called? I know that one. Yeah, I think we talked about this a little bit earlier. You can supply the transcript from your language model session that you've configured and then use a tool call evaluator in your evaluator process. And actually, then you just provide in your data samples the assertions you want. We have an expectation system where you could say, you know, I want my tools to be called. Were they called? Were they called in the right order? Were they sent the right properties? Were the values that were sent to those properties? Like, you could go pretty deep with this. It's a pretty sophisticated assertion system. But, you know, really, you supply that at the data sample, like the scenario level. And then our evaluator takes care of the rest. It looks at the transcript. It finds out all that information and just gives you a report. Yeah, that's really awesome. And again, having worked with developers at workshops where they're implementing some of their first tools, you're like, wait a minute, it should have been called this way, and it wasn't. And it was kind of a source of frustration, frankly. Yeah, authentication is a big one. If you need to access something that requires an authentication, you want to know that. And the model won't always know that, or the model won't know, oh, I have to call this tool before this. Totally. And the importance of really giving the LLM good guidance on when that specific tool should be used. Dynamic profiles. Dynamic profiles. We haven't mentioned that yet today. Other great stuff. So, yeah, that's a great feature. And, yeah, thank you, Nia Murphy, for that question. All right, moving on to the next one here. Protonster. I like that, Protonster. I don't know. Yeah, very cool. vision of a person with a cape, you know. Next year, anonymous. Too much time is being spent on the names. How does the Apple Intelligence Framework mitigate semantic entropy when passing context between on-device models and larger server-based models? That lives up to the username. It's funny you just mentioned dynamic profiles. That's Foundation Models Framework, I think, is what it's alluding here. But, yeah, it's a great question. Like, we have the system language model that has a 4K context size, and then we have the PCC model that has a 32K. So when you're using dynamic profiles, which quickly, a high overview of that is you can, with the same transcript, switch between specific profiles. Maybe somebody is specific to accounting, and they do all those things, but you want to have another profile that's doing something health-related. You can switch between those. It's like agents, right? Yeah, exactly. And in order to kind of keep the context sharing, we have modifiers in this declarative style, like history transform to say, hey, I switched to this different profile. Maybe I want to throw out some tool calls. Maybe I want to change something and keep those balanced. And then another great thing that we added this year is the foundation models utilities package that has ways where you can do like summarization and maybe compact the transcript. if you're going to a smaller size. So there's all these tools. Another great way to test that was with the evaluations framework. We marry well together and be able to see what works best with your use cases and tailor that. Awesome. Great. Thank you for that question. Okay, moving on. Freescott is the username. Do I need to use the evaluations framework with foundation models? Oh, that's a great question. As I mentioned earlier, I think when we were talking about academic research, you can obviously use a language model. You don't have to use the foundation models framework to do a lot of the evaluation. We just talked about tool call evaluator, which is one example where if you want to use the tool call evaluator, you do need to use foundation models because we need that transcript. We're actually working to kind of do a more generic transform from other models, such as if they're using, you know, model provider, they'll get that transform. But by and large, no, you don't. You can evaluate any stochastic system. Yeah. With that. Yeah, that's great. Great stuff. Thank you for that question. Okay. Next one from Dravenin. Message. The question is, I have multiple apps that will all donate entities via index entity to the Spotlight Semantic Index. Can one of my apps, Foundation Model Sessions, retrieve another of my apps donated content via Spotlight Search Tool, same developer, shared app group, or is retrieval strictly sandbox to the donor? Yes, retrieval is limited to your app sandbox only. So you wouldn't be able to. Since they do mention app groups, like you could do interesting things, presumably, where like two apps from the same developer can be in the same app group and they can share files, right, on this. They can share some data. Not like literally through that sandbox, but like you probably like find ways to maybe share things. Yeah, so the way we think about that in App Intense is with the transferable protocol. So that is kind of the best way for doing this kind of interop between different file formats, essentially. Transferable works by a kind of progressive, from highest fidelity to lowest fidelity data format. So if you were both to share a similar data format, you can use Transferable to go between apps. And the cool thing about that API is that you're essentially opting into the data transfer. So it's done in a private and secure way, because you're essentially saying, I want to share maybe a small projection of my app, but not all the contents of my app. So you, the developer, is always in control of what gets shared with whom. That's great. Awesome. Great question. Thank you for that. Okay, next question. Again, from looks like John. That's an easy username. Yeah, Jay. Here we go. Schemas. Are schema-bound entities interchangeable? can an app's file entity be turned into a photo entity and vice versa? I suspect this is possible through Transferable. You're right on. Plus one there. You are higher, John. That's the cool thing about Transferable is that you get to specify all the different data transformations that your app can support. So, like, for instance, in our sample app that's available that you can download today, unicorn chat, we adopt transferable in the app in a way where you can have messages that could be, you know, text, but they can also contain attachments. And so the app supports transferability, exporting text, but they can also export files, you can also export photos. And so depending on the content, you can even provide transferability in a dynamic way, based on sort of the payload type. And that allows you to export your contents in a way that can integrate with different apps based on the capabilities that an app supports. So when you export a message as a photo, it means that this photo can now be imported into the Photos app, but it can also be imported into any other third-party photo app that adopts our photo schemas. Yeah, there's one more note there. There is a file entity protocol that works for all app entities. It's kind of designed for when you need these file-based formats. So read the documentation for more information there. Well, that's great. You know, I hear unicorn shits, like, moving up on the top app list. There we go. You used my page models in that, right? Yeah, you did. It combines our technologies. No, that's great. Well, thank you for that question, John. Okay, next question from Zulfi Shah, I believe. That's how we say that. In my app, I adopt app intents and app entities for various concepts in my app. Some of these might map to app schemas, though not perfectly. For example, I manage a list of contacts, and a close match would be schema from.messages,.messages person. Is that acceptable to participate in new Siri? Yeah, absolutely. I mean, the answer is definitely yes. The cool thing about this API is that you can pick and choose the functionality that your app supports and its capabilities based on the app schemas that we support. So, you know, you mentioned some kind of contact integration. We have schemas for a message person, for instance, where you can export contacts from your app to support some messaging features. In fact, Unicorn Chat uses this. And you can check out the sample code to see how we model content in the app using this schema to expose this information to Siri and Apple intelligence. But, you know, if you can't find any schema that sort of fits into your app's functionality, we have the more generic system search capability so that you can integrate with Apple intelligence and expose your app's search capabilities. It's great for apps like e-commerce, food ordering, and many other types. But then you can always use app shortcuts as well as another way to essentially integrate your app intents in a way that can be more generic. Now, of course, because app shortcuts are custom, it means that you have to provide sample phrases that match sort of your app's experiences and the scenarios that you're thinking about. And that's the cool thing about app schemas. When you integrate with a schema, Apple is doing the heavy lifting, so you don't have to provide those phrases. That's great. James, anything you want to add on? I think you covered it. I'm glad people are kind of asking this question, though. It's come up a few times. So I think it's really cool that people are thinking about this. Yeah. This is good stuff. I can't wait to see what people build with these integrations and these APIs. Very excited. Yeah. Thank you for that question. All right. Almost out of time here, so let's see if we can get one more in here. From R.Grad Harker Schron, apologies for stepping all over your username there. We already know that with image processing addition to foundation models, the token limit is still 4,096. Do we need to be aware of any constraints to our other inputs like prompt or instructions in the case for the model to function at its best? I mean, with the on-device model, you have 4096, right? With the image input support, I mean, it doesn't really change anything. If you put an image in there, you can measure this with instruments with the support we had this year, but I think it comes down to like about 200 tokens or so for an image typically. But in terms of like your instructions or anything else, you can take advantage of the full 4096. If you want to give it 4,000 tokens input and generate a really small output, that will work just as well as giving it a small input and generating large output. And then with private cloud compute, just for clarity, you get 32K context size, right? Yeah, that's nice. Much larger. Yeah, much, much larger. I think too, in 26.4, we also shipped token counting APIs. So you can play around with that to see what's working. But also, depending on your use case, you kind of want to hill climb. Yeah. You're alluding over here. And you might have a very specific use case and tailoring that prompt and making it brief maybe with that smaller context size and pulling in evaluations to do that climbing. It can also help with that. Yeah, the last thing I'll say with the undervisal model, we get a lot of questions about, oh, it's not as capable because it's smaller. And it's like it's actually you just have to be more precise with your instructions. And sometimes it's finding the right synonym to use. So you might try different variations of words. And to your point, hill climbing is a perfect way to just kind of benchmark that and see which variation of the prompt works best. But you could get a lot out of the advice model if you're willing to put some time kind of massaging your prompt. Put examples in your prompt as well, right? Yeah. Oh, yeah. Give it examples. You know, give it as much information as you can. Yeah. Well, great stuff. Well, unfortunately, that's about all we have time for today for this group lab. We're really thankful for you taking the time of your busy schedules to join us. We really hope you found this to be really helpful and beneficial. You know, we also want to say thanks to our panelists. Thank you all. I know it's been a really busy week and the release and everything else you have going. So thank you. And, yeah, just want to say thanks again to all of you. As we mentioned earlier, if you go to developer.apple.com slash forms, please post your questions, your code questions, and so on. We're there to help you any way we can. And, of course, feedback assistant, right, if you have ideas. And if you find a bug. Okay. It'll be a first, but still. Yeah, exactly. But by all means, take advantage of the resources. Some amazing videos this year, a lot of great technical documentation. and yeah, we really appreciate all your feedback and insights. If you'd like to see more of these kind of group labs, or see me throw a book, right? Yeah, you can see me throw more books and other things. Please let us know that as well. We really want to help you be successful and thank you for being part of the Apple community, the Apple family. We really appreciate all your time and great work in seeing the innovative things, apps and new technologies you come up with. So thank you. And yeah, with that, thanks again for joining us. Have a great rest of your WWDC. And we can't wait to see what you build. Yes, absolutely. Thank you all. *sad music*