---
name: core-ai-fast-first-token
description: >-
  Reduce Core AI time to first token and confirm the compute unit: measure specialization/load/inference, cache and pre-specialize, ahead-of-time .aimodelc per deviceArchitectureName, prefill vs decode functions, detect silent Neural Engine fallback, .cpuOnly safe path, runtime memory checks. Swift.
---
# Core AI Fast First Token and the Right Compute Unit

Make the first answer from your Core AI model arrive fast, on the compute unit you planned, without silent slow or wrong runs.

Course home: https://edgefde.dev/courses/core-ai-fast-first-token/

## When to use

Use when a Core AI model's first answer is slow, differs a lot across devices, or you suspect it is not running on the Neural Engine you planned.

## Environment

Xcode 27, iOS 27 or macOS 27, target links CoreAI.framework, Metal Toolchain (`xcodebuild -downloadComponent MetalToolchain`). Python 3.11+ via uv in the apple/coreai-models checkout: `uv venv`, `uv sync`, `uv run coreai.llm.export ...`. For a fresh project: `uv init --python 3.11`, `uv add torch coreai-torch`.

## Recipe

1. **Where the first-token wait goes.** Split first-token time into specialization, load, setup, and inference, with Apple's tools in the lab and logs in the field. Brief: `01-where-first-token-time-goes.llms.md`.
2. **Specialize before the user waits.** Make sure specialization happens before the first user request, from the cache or ahead-of-time compiled files. Brief: `02-specialize-before-the-user-waits.llms.md`.
3. **Use prefill for long prompts.** Cut first-token time on long prompts by loading and using the prefill function, and set the right options for dynamic decode. Brief: `03-prefill-and-decode-functions.llms.md`.
4. **Catch silent Neural Engine fallback.** Detect when a Core AI model meant for the Neural Engine does not run there, or runs there with wrong results. Brief: `04-catch-silent-compute-fallback.llms.md`.
5. **CPU-only safe path and memory.** Add an ordered compute plan with a CPU-only fallback and budget memory by measured function loads. Brief: `05-cpu-only-safe-path-and-memory.llms.md`.

## Acceptance checks

- An Instruments Core AI trace shows at most one Specialization event on a second launch (zero if cached).
- The app checks AIModelCache.default before loading and shows a Preparing state only on a cache miss.
- Long prompts run through the prefill function, short ones through decode, with times logged for both.
- A build check counts Neural Engine regions in each .aimodelc and fails when the count is zero for a model meant for the Neural Engine.
- A .cpuOnly path loads and answers on every supported device.

## Key concepts

- First-token time has parts: specialization, load, setup, then prefill inference. Measure each one.
- Specialization is the slow part on first run. Cache it, run it early, or move most of it to build time with coreai-build.
- LLM exports can have a separate prefill function next to the decode function. Use it for long prompts.
- The Neural Engine can be skipped or give wrong numbers with no error. Check compute units and compare outputs.
- SpecializationOptions.cpuOnly is a safe path. Runtime memory comes from loaded functions, not file size.

## Takeaways

- You can show a 'Preparing' state only when specialization is really needed.
- You can ship per-architecture .aimodelc files and pick the right one at runtime.
- You can prove which compute units ran your model, and fall back when the plan fails.

## Sources

- Apple Core AI docs: [Managing model specialization and caching](https://developer.apple.com/documentation/coreai/managing-model-specialization-and-caching), [Compiling Core AI models ahead of time](https://developer.apple.com/documentation/coreai/compiling-core-ai-models-ahead-of-time), [Analyzing model runtime performance with Instruments](https://developer.apple.com/documentation/coreai/analyzing-model-runtime-performance-with-instruments), [debug gauge](https://developer.apple.com/documentation/coreai/monitoring-model-performance-with-the-debug-gauge), [Validating inference correctness against a reference run](https://developer.apple.com/documentation/coreai/validating-inference-correctness-against-a-reference-run)
- Apple API reference: [AIModel](https://developer.apple.com/documentation/coreai/aimodel), [InferenceFunction](https://developer.apple.com/documentation/coreai/inferencefunction), [SpecializationOptions](https://developer.apple.com/documentation/coreai/specializationoptions), [ComputeUnitKind](https://developer.apple.com/documentation/coreai/computeunitkind), [AIModelCache](https://developer.apple.com/documentation/coreai/aimodelcache)
- apple/coreai-models: [neural_engine_rules.md](https://github.com/apple/coreai-models/blob/main/skills/skills/model-authoring/references/neural_engine_rules.md), [common_issues.md](https://github.com/apple/coreai-models/blob/main/skills/skills/model-authoring/references/common_issues.md), [gemma4/export.py](https://github.com/apple/coreai-models/blob/main/models/gemma4/export.py)
- [coreai-models #205](https://github.com/apple/coreai-models/issues/205) (Neural Engine compile fails but coreai-build exits 0 and writes a GPU-only .aimodelc), [coreai-models #279](https://github.com/apple/coreai-models/issues/279) (per-block int quantization gives wrong output on the Neural Engine)