Apple Intelligence Group Lab

Engineers from the App Intents, Foundation Models, and Apple Intelligence teams introduce themselves and frame the session, which covers App Intents and the new app schemas, the agentic Siri, Foundation Models on device and via Private Cloud Compute, Image Playground, and the Evaluations framework.

Lean on WWDC videos, the documentation (including its dedicated samples section), and technical articles. App Intents had roughly five sessions this year with several sample apps you can study to see how features are built and used, alongside a significant documentation push.

New this year, app schemas are Apple-designed and fine-tuned to plug into system experiences, and you pick only the ones that match your capabilities. Adopt the schemas that genuinely fit — even a partial match like timers is worth conforming to — rather than forcing an ill-fitting type; you can also combine schemas from multiple domains.

Review the schemas and adopt the parts that fit — conform to an entity schema, donate, and interact with Spotlight — and you gain benefits even if not every intent maps cleanly to your app. Even where an intent doesn't fit, you can still let users ask questions about your content or take action on it through the new view annotation API. Start integrating now and expand coverage as the platform evolves.

New this year, a system orchestrator can take actions from many apps' App Intents across the system — apps don't drive each other directly. This routing is deliberate for privacy and safety: by exposing App Intents you make your actions available to the system-level orchestrator, which coordinates them while keeping Apple Intelligence personal and private to the user.

Yes — this is exactly what schemas enable. Users can take actions, ask questions about your entities, and do follow-ups in a multi-turn conversation. Try it in the betas with your own app: because it's your content, Siri has a deep understanding of it and users can ask natural questions and refine.

Absolutely. Pick and choose schemas from any domains that match your app's capabilities — calendar and reminders actions can coexist and work together for holistic experiences. It's not an either/or choice: schemas, App Shortcuts, and in-app search can all be combined in the same app.

In the foreground there's no rate limiting on the on-device model — requests are effectively unlimited. In the background, the system applies throttling when under heavy load. There's no fixed budget number; instead, catch the rate-limited error, which is how the system signals it's under load, and chunk or defer work in response.

This year's LanguageModel protocol defines a common LanguageModelError covering cases like rate limiting and refusals. For the Private Cloud Compute model specifically, there's a quota-usage property to check how close you are to the limit, plus model-specific errors thrown for that model — so you can distinguish retryable conditions from terminal ones.

Use the Evaluations framework's model-judge evaluator to build safety rubrics (e.g. a 1–4 scale) and systematically assess which prompts trip the guardrails. If the on-device model is over-triggering, you can also route the check through the more capable PCC model to understand what's flagged and refine your prompts.

Models declare their capabilities through the LanguageModel protocol, and vision is one of them — so a model that supports it can accept image input (both the on-device and PCC models added image input this year). Apple's PCC model also exposes light, moderate, and deep reasoning levels. There's no built-in video input, but the framework is extensible: you can use the custom segment API to support custom modalities like video, or preprocess a video into a text transcript (for example with the Vision framework) and feed that to the on-device model.

"Advantage" isn't quite the framing — schemas are simply how Apple Intelligence reasons about your app's actions and entities. To influence Siri where no schema fits, use the new APIs: intent donation and related tools that tell Siri what's happening in your app, so your actions and entities remain discoverable.

Yes. The Image Playground API gained significant additions this year — use the ImagePlaygroundStyle.all parameter to access the photorealistic style. It now runs on Private Cloud Compute, giving you a more powerful model. A dedicated WWDC video and documentation walk through the API.

They're complementary, not alternatives. A schema defines the shape and properties of your content (a book's title, author, contents). IndexedEntity is a separate API that indexes that content into the system's semantic index so Siri can retrieve it for requests like "read a book by this author." Adopt a schema and also conform to IndexedEntity to make content both well-modeled and findable.

Yes. The Evaluations framework was designed to evaluate not just language models but any stochastic system, so it applies to traditional ML models like linear regression or classifiers too. What matters is what you're measuring; the framework's methodology can support research metrics, not only product evaluation.

Understanding on-screen context is one of the three pillars of the new Siri — it can reason about what's visible and act on it, using the existing user activity API plus the new view annotations API to expose your entities on screen. On CarPlay specifically, the panel didn't confirm this support and emphasized keeping the driver focused on the road, encouraging developers to file a feedback request for CarPlay support with appropriate safety considerations.

The new Siri is available on iPhone, iPad, Mac, and visionOS — not on HomePod. However, existing App Shortcuts have worked on HomePod for a while, so you still have a path to invoke your app's actions there.

Aim for parity, but test each platform's experience yourself rather than assuming they're identical. The new Siri spans many devices, so when adopting these APIs, verify your experiences across everything you support — including AirPods, an easily overlooked but common place users invoke Siri for quick actions.

Image Playground currently always requires an internet connection to generate images, because it uses Private Cloud Compute (a server-based model). It does not fall back to on-device generation.

Panelists shared personal favorites, with a standout being the intelligence features surfaced during phone calls — for example pulling a flight confirmation number out of Mail while you're on a call, so you don't have to hunt for details mid-conversation.

The schema approach is a deliberate trade for consistency and privacy. Apple's take is personal intelligence kept private to the user, and fixed schemas let the system guarantee a holistic, predictable, and safe experience across apps — properties that a fully open, dynamically described skill system would make harder to ensure.

Yes — supply the transcript from your configured LanguageModelSession and use the tool-call evaluator. Its expectation/assertion system lets you assert whether tools were called, in the right order, with the right properties and values — a fairly deep and sophisticated set of checks.

This relates to Foundation Models' dynamic profiles and the different context windows — the on-device system model has a 4K context while the PCC model has 32K. The framework manages how context is carried across those models so information is preserved when moving between the smaller on-device and larger server-based models.

No — the Evaluations framework can assess any stochastic system and doesn't require Foundation Models. The main exception is the tool-call evaluator, which needs a Foundation Models transcript today (a more generic transform for other model providers is in progress). Otherwise you can evaluate with any language model.

Retrieval is limited to your app's own sandbox, so a session in one app can't pull another app's donated content through the Spotlight search tool. Because apps from the same developer can share an app group, you may be able to share underlying data or files between them by other means — but the semantic-index retrieval itself stays sandboxed per app.

Yes, Transferable is the right tool. It lets you specify the data transformations your app supports between entity types. The Unicorn Chat sample app (available to download) adopts Transferable so a message can be text or contain attachments — a working example of interchanging content types.

Yes. Pick the app schemas that best match your capabilities even when imperfect — the messagePerson schema lets you export contacts to support messaging features (Unicorn Chat demonstrates this). If truly nothing fits, fall back to the more generic system types to still expose your content to Siri and Apple Intelligence.

The on-device model's 4096-token budget is unchanged by image support. An image consumes roughly 200 tokens (measurable with Instruments), leaving the rest for your instructions and prompt. You can use the full budget however you like — a large input with small output works as well as the reverse.