Coding Intelligence, Machine Learning & AI Group Lab

Engineers from the machine learning and AI frameworks teams introduce themselves and outline the session, which spans Apple's ML stack — the Foundation Models framework, Core ML, MLX, and Core AI — plus agentic coding in Xcode, the Evaluations framework, on-device versus Private Cloud Compute models, and bringing your own LLM provider.

These technologies form a layered suite you can enter at different levels. At the top is the Foundation Models framework — start there for language-model use cases, try the system language model, and use Evaluations to confirm it meets your need. For custom neural-network models such as diffusion or image segmentation, or models you train or download yourself, drop to Core AI, which comes with SLAs and guarantees for building applications; going forward, anything new involving neural networks should move to Core AI. Core ML remains but is now focused on traditional ML like decision trees. Below that, MLX is the lowest level — powerful and flexible, and the place for on-device training and distributed workloads across multiple machines (for example running very large models across Macs, not on the phone). Choose the highest level that meets your need.

The on-device context is 4096 tokens and is a shared budget — if you feed in 4000 tokens, the response can use the remaining ~96. The Private Cloud Compute model offers 32K, also shared. For a larger context window use PCC, and for deeper reasoning use the reasoning-capable PCC model.

Yes, calls can run in a background task, but if the OS is busy it may rate-limit you — catch the rate-limited error from the system language model and retry later. On macOS you're fine in the foreground, and the Private Cloud Compute model provides another path when on-device throttling is a concern.

This question bundled two. On the waitlist: it applies only to Siri — it does not apply to the Private Cloud Compute language model or to the on-device things Siri does. On whether the beta includes the AFM core advanced (~20B) model: yes, it's included and is used for the voice features. For anything deeper on Apple Intelligence, the dedicated Apple Intelligence group lab is the place to go.

The framework lets you compose across the on-device model, Private Cloud Compute, and your own provider. Once a third-party provider is in the loop, its own data-handling terms apply and you're responsible for knowing and surfacing those boundaries. The panel pointed to the agentic-experiences-with-Foundation-Models session for how to structure such flows.

This spans several speech frameworks that weren't represented on the panel, so there's no definitive answer. Conceptually there are two layers: OS-level personalization Siri may apply system-wide, and whatever you do within your own app. Speech recognizers commonly pair with a personalization component — for example a small model fine-tuned on examples like your contact list — so building your own is certainly possible. If the language you need is already supported by on-device speech recognition, it's likely good enough to use as-is. For a definitive answer, ask on the developer forums, where engineers are responding throughout the week.

Agents learn primarily through search and documentation rather than training. Give the agent good tools to search and find things in your codebase, and write down what it should know — project conventions, architecture notes, an AGENTS.md — so it can retrieve and apply that context. Teaching the agent how to find and record knowledge is more effective than expecting a small local model to internalize a complex codebase.

Work in layers. Start small with unit tests by building components that can be tested independently, then have the agent enumerate the cases so unit coverage is thorough. Build up from there to UI-level automation. Structuring code for testability first makes both hand-written and agent-generated tests far more reliable.

The Vision framework has significant updates this year — see the "What's new in image understanding" session, which covers capabilities like segmentation models. Choosing between Vision and Foundation Models: use Vision for well-understood, repeatable tasks like detecting a specific object, since it's optimized, efficient, and easy to test; step up to Foundation Models when the task varies each time or needs semantic or natural-language understanding. Foundation Models is also gaining Vision-powered tools this year, including a barcode reader and an OCR tool. Natural Language specifics weren't covered.

Use the APIs added this year for managing context, and keep prompts and tool definitions lean — every tool definition and instruction consumes the shared budget. Trim unnecessary context, summarize or chunk long inputs, and only include the tools relevant to the task so the model has room to produce a high-quality response within the on-device window.

Partly. When initializing the system language model's guardrails you can opt into permissive content transformation, so the model won't error out on emotionally intense first-person input like journal entries — though it may still decline in natural language to elaborate. For error handling, distinguish two separate things: a refusal error is the model's own aligned response declining to answer (seen with guided generation), while a guardrail error comes from a separate moderation model that inspects input and output; you can catch each separately and fall back gracefully. These apply only to Apple's models. Guardrails were substantially improved this year, so false positives should be much lower — file feedback if you still hit them.

Evaluation shouldn't be an afterthought bolted on once a feature exists — it should come first, because the evaluation set is the living specification of what your feature is supposed to do. Defining how you'll measure success up front clarifies the feature itself and keeps quality anchored as models and prompts change.

No — sharing arbitrary models across apps isn't possible because it's genuinely complex: it becomes hard to manage contention when multiple apps want the resource simultaneously. That said, the system provides model caching for the frameworks Apple ships, so shared system models (like the Foundation Models system language model) don't each cost you separate storage.