Apple Intelligence

What are the most useful Apple resources for learning and applying new APIs and design changes?

• WWDC videos, documentation, sample apps, and migration guides are primary resources.
• The documentation includes a dedicated sample section and technical articles.
• Sessions often demonstrate sample apps, showing how they are built and used.
• Documentation for App Intents and Foundation Models provides usage guidance and benefits.
• Previous WWDC materials are valuable for foundational knowledge.

How do I integrate with Siri when no app schema fits my entities?

• Utilize existing app schemas by picking and choosing actions that best match the app's capabilities.
• Combine schemas with custom App Intents for functionality outside predefined schemas.
• App Intents make actions available in Shortcuts, Spotlight, and more.
• For new Siri AI integration, adopting an app schema is required.
• Consider the system.search schema for search-related functionality (e.g., e-commerce, food ordering) to allow Siri integration with the app's search.
• The system.open schema allows Siri to open donated entities directly in the app.
• If no schemas fit, App Shortcuts can be used by defining memorable phrases for specific actions.

Should I use the closest schema match or skip schemas when the fit is imperfect?

• It's recommended to use the closest related schemas.
• Developers can combine schemas from different domains if the app has diverse capabilities (e.g., calendar and reminders).
• If no schema perfectly fits, consider using App Shortcuts for specific actions.
• For new Siri AI integration, adopting an app schema is necessary.

How should app entities and actions participate in agentic Siri?

• Review available schemas and implement those that align with the app's capabilities.
• Conforming to an entity schema and donating it to Spotlight provides benefits for Siri AI interaction.
• Developers can use Siri AI to ask questions about content or take actions on it using any view annotation API.
• Consider the platform's evolution and how schemas provide a standardized, consistent experience.

Can an app act as its own orchestrator and drive other apps, or does cross-app action still have to route through Siri Shortcuts?

• A system orchestrator is available that can take action from App Intents across different apps.
• Privacy and safety are key; data indexed from the app is available to the system, not other apps directly.
• The Transferable API allows developers to control how data is shared between apps.
• Transferable supports bidirectionality, letting developers choose which data to export and import.

Can third-party apps participate in multi-turn Siri conversations?

• Yes, apps can participate in multi-turn conversations with Siri.
• App schemas enable apps to take actions, ask questions about entities, and handle follow-ups.
• The underlying large language model processes input for more natural user interactions.
• App schemas offer flexibility in how users phrase requests, allowing the model to parse information and select the correct action.

Can one app mix schemas from different domains?

• Yes, it is acceptable to mix and match schemas from different domains.
• The intention is for developers to select schemas that best match their app's capabilities.
• Schemas, App Shortcuts, and in-app search can work together to create holistic user experiences.
• Developers should leverage these components based on their understanding of their users and app functionality.

Is there a practical time work budget for Foundation Models during one background wake? If no fixed number, how should apps safely chunk work?

• For on-device Foundation Models running in the foreground, there are no rate limits.
• When running in the background, throttling may occur if the system is under heavy load.
• Apps should catch rate limited errors to know when to retry requests.

What exact error types or codes indicate Foundation Model throttling?

• Throttling can be indicated by errors like system busy, thermal pressure, or quota exhaustion.
• Distinguish retriable rate-limit errors (retry later) from terminal quota exhaustion.
• The Language Model Protocol includes Language Model Error for common issues like rate limiting and refusals.
• For the PCC model, specific errors related to quota usage and quota exhaustion are available.

How to avoid language model session.generation error.guardrail violation?

• Use the model judge evaluator in the Evaluations Framework with safety rubrics.
• Developers can use a larger model (like PCC) to evaluate prompts that trigger guardrail violations on smaller models.
• Guardrail violations are designed to prevent unsafe responses.

In iOS 27, can the Foundation Models Framework reason over video or work with a video model, or is multimodal input limited to still images?

• The Foundation Models Framework supports models with vision capabilities.
• The PCC model can reason over images with options for light, moderate, and deep reasoning.
• The framework is extensible; custom modalities like video can be supported via a custom segment API.
• The Foundation Models Framework is going open source, enabling server-side Swift preprocessing tools.
• For video processing, consider using other Apple frameworks to extract transcripts or audio descriptions and feed that text to the models.

Do App Intent schemas give an advantage over generic intents for Siri AI discovery?

• Schemas are the primary way Siri AI reasons about actions and entities in the application.
• Using schemas provides a standardized and consistent experience across apps.
• Intent donation allows users' actions within the app to be learned by Siri.
• App Shortcuts allow developers to specify phrases for app intents.
• Adopting an app schema means Apple has pre-trained models and sample phrases, reducing developer effort.

Can Image Playground generate photorealistic images inside an app?

• Yes, developers can use the Image Playground API to generate photorealistic images.
• Use ImagePlaygroundStyle.all as a parameter for photorealistic generation.
• This functionality runs on Private Cloud Compute (PCC), leveraging its powerful model.

What's the difference between indexed entities and defined entities (like app schema entities)?

• Schemas define the structure and properties of content (e.g., a book's title, author).
• Indexed Entity is an API to index that content into the system's semantic index for retrieval by Siri AI.
• Adopting an app schema and conforming to the Indexed Entity protocol provides the best Siri AI experience.
• Indexed Entity can also be used for custom entities and donated to Spotlight, enabling deep linking via system.open intents.
• App schemas simplify Spotlight integration by handling indexing keys automatically.

Can the Evaluations Framework be used for academic research?

• Yes, the Evaluations Framework can be used for academic research.
• It's designed for evaluating any stochastic system, including traditional machine learning models.
• The framework allows for scenario-based metric collection and the creation of app-specific or scenario-specific benchmarks.
• Features like synthetic data generation and Xcode integration with visuals can support research.
• Reports can be saved and exported as JSON for further analysis.

Does Siri support screen context on CarPlay?

• On-screen context is supported via NSUserActivity integration and the new view annotations API.
• CarPlay screen context and grid-style actions are not supported — file Feedback Assistant if developers need them.
• Safety is a priority; keep driver attention on the road when using CarPlay.

Are we going to be able to use App Intents with HomePod, since currently there is not a beta for HomePod?

• The new Siri AI is not available on HomePod.
• Existing App Shortcuts have worked on HomePod for a while.

Can we expect watchOS responses to App Intents to match up one-to-one with iOS?

• No — dialogue is not one-to-one across devices. Siri adapts responses by context (e.g., verbose on AirPods, concise on iPad).
• Developers are encouraged to test experiences across all supported devices, including watchOS and AirPods.
• App Intents can optionally provide dialogue that can be adapted for different contexts.
• The App Intents testing framework is available for testing.
• Customizing the Siri experience, including intent dialogue, is supported.

Does Image Playground now require an internet connection for PCC, or does it fall back to the previous model gracefully?

• Image Playground always requires an internet connection as it uses PCC (Private Cloud Compute).
• It will not fall back to an on-device model.

What are notable highlights from the Apple Intelligence updates?

• Phone call intelligence: Features like extracting confirmation numbers during calls.
• App Intents testing: A new way to unit test App Intents, including UI and Spotlight integrations, for CI pipelines.
• Foundation Models Framework: Integration with various language providers, core AI, and the ability to bring any model from the web.
• Evaluations Framework: Testing intelligence-based features, model judges, quantitative/qualitative metrics, and tool call evaluation.
• Private Cloud Compute (PCC): Easy access to server models with simplified authentication (one line of code).
• New Siri AI: Advancements in Siri, new APIs for developers, and a deeply integrated ecosystem for natural language interactions and actions across apps.

Why does Apple use hard-coded App Intent schemas instead of dynamic skill descriptions?

• Schemas provide consistency and privacy, aligning with Apple's focus on personal and private intelligence.
• Schemas ensure a standardized, holistic experience across the platform, making interactions natural across apps in the same domain.
• Schemas offer security benefits, such as built-in confirmation steps for sensitive actions (e.g., sending money).
• Schemas enable differentiated behavior based on entity ownership (e.g., shared vs. personal calendar events).
• Schemas facilitate localization by allowing Apple to handle model training and natural language strings across locales.

Can the Evaluations Framework verify tool calls?

• Yes, developers can supply a transcript from the Language Model Session and use a tool call evaluator.
• The Evaluations Framework allows developers to define assertions for tool calls, including order, properties, and values.
• This system can be used at the scenario level to report on tool execution.
• Features like dynamic profiles and Xcode and Instruments integration further aid in visualizing and verifying tool calls in real time.

How does the Apple Intelligence Framework mitigate semantic entropy when passing context between on-device models and larger server-based models?

• The Foundation Models Framework uses dynamic profiles to manage context across models with different context sizes (e.g., 4K for on-device, 32K for PCC).
• History transform modifiers can be used to manage context when switching profiles or incorporating tool calls.
• The Foundation Models Utilities package provides tools like summarization to compact transcripts for smaller context sizes.
• The Evaluations Framework can be used to test and optimize context management strategies.

Do I need to use the Evaluations Framework with Foundation Models?

• No, developers do not necessarily need to use the Evaluations Framework with Foundation Models.
• The Evaluations Framework can be used to evaluate any stochastic system.
• However, specific evaluators like the tool call evaluator require Foundation Models to access the transcript.
• Apple is working on generic transforms for other models to enable broader use of these evaluators.

Can one app retrieve another app's donated Indexed Entity content?

• Retrieval is limited to the app's sandbox only.
• For inter-app content sharing, use the Transferable protocol.
• Transferable allows for progressive data sharing from highest to lowest fidelity.
• Developers control what data is shared and with whom, ensuring privacy and security.

Schemas: Are schema-bound entities interchangeable? Can an app's file entity be turned into a photo entity and vice versa?

• Yes, schema-bound entities can be interchangeable, often through the Transferable protocol.
• The Transferable protocol allows developers to specify data transformations the app supports.
• For example, a message entity containing an attachment can be exported as a file or a photo, allowing import into relevant apps (e.g., Photos app).
• A file entity protocol exists for file-based formats.

Is an imperfect schema match acceptable for new Siri integration?

• Yes, it is acceptable to use schemas that are a close match.
• Developers can pick and choose functionality based on available app schemas.
• For concepts that don't fit a schema, use the generic system.search capability for integration with Apple Intelligence.
• App Shortcuts can also be used for custom integrations, requiring developers to provide sample phrases.
• Adopting schemas can let developers delete code — the platform handles more of the integration logic.
• Adopting an app schema means Apple handles phrase training and provides a consistent experience.
• Intent donation allows Siri to learn from user actions (e.g., "message Dan") over time.

How do image inputs affect the Foundation Models 4,096 token budget?

• For the on-device model, the context size is 4096 tokens. Image input typically consumes around 200 tokens.
• Developers can use the full 4096 tokens for instructions and prompts.
• With Private Cloud Compute (PCC), the context size is 32K tokens.
• Token counting APIs are available to measure usage.
• For optimal performance, especially with smaller models, tailor prompts, use synonyms, and include examples.