Core AI Fast First Token and the Right Compute Unit

Make the first answer from your Core AI model arrive fast, on the compute unit you planned, without silent slow or wrong runs.

Specialization and caching, ahead-of-time .aimodelc, prefill vs decode functions, Neural Engine fallback checks, a .cpuOnly safe path, and runtime memory.

Measure where first-token time goes, specialize before the user waits, use the prefill function for long prompts, detect when the Neural Engine was not used, and fall back to a CPU-only path.

Who this is for

You already load a .aimodel with Core AI and get answers. The first answer is slow, or it is fast on one device and slow on another, or you are not sure the Neural Engine is used at all. This course uses Apple's Core AI documentation and the apple/coreai-models repo. Every API name here was checked against developer.apple.com on 15 Oct 2026. Where we could not verify something, we say so.

New to Core AI? Start with PyTorch to Core AI in Xcode and From Metal to Core AI Model. Not sure Core AI is the right stack? See Core AI vs Core ML vs MLX.

Lessons

Sources we checked

Apple Core AI docs: Managing model specialization and caching, Compiling Core AI models ahead of time, Analyzing model runtime performance with Instruments, debug gauge, Validating inference correctness against a reference run
Apple API reference: AIModel, InferenceFunction, SpecializationOptions, ComputeUnitKind, AIModelCache
apple/coreai-models: neural_engine_rules.md, common_issues.md, gemma4/export.py
coreai-models #205 (Neural Engine compile fails but coreai-build exits 0 and writes a GPU-only .aimodelc), coreai-models #279 (per-block int quantization gives wrong output on the Neural Engine)

Key concepts

First-token time has parts: specialization, load, setup, then prefill inference. Measure each one.
Specialization is the slow part on first run. Cache it, run it early, or move most of it to build time with coreai-build.
LLM exports can have a separate prefill function next to the decode function. Use it for long prompts.
The Neural Engine can be skipped or give wrong numbers with no error. Check compute units and compare outputs.
SpecializationOptions.cpuOnly is a safe path. Runtime memory comes from loaded functions, not file size.

Takeaways

You can show a 'Preparing' state only when specialization is really needed.
You can ship per-architecture .aimodelc files and pick the right one at runtime.
You can prove which compute units ran your model, and fall back when the plan fails.

Agent briefs

FAQ

Why is the first Core AI load so slow?

The first load usually includes specialization, which turns the portable .aimodel into code for this device and OS. Check AIModelCache.default first, call AIModel.specialize after download, or ship .aimodelc files from coreai-build.

What is the difference between .aimodel and .aimodelc?

.aimodel is the portable model. .aimodelc is the ahead-of-time compiled asset from xcrun coreai-build compile, one per device architecture. Pick it with AIModel.deviceArchitectureName. It still needs a little on-device specialization.

How do I know if my Core AI model runs on the Neural Engine?

Profile with the Core AI Instruments template, which shows which compute units run your model. In CI, check coreai-build output for Neural Engine regions, and compare outputs against a CPU-only run.

How do I force Core AI to use only the CPU?

Load or specialize with SpecializationOptions.cpuOnly. Use the same options for specialize and load, because the options are part of the cache key.

Does a bigger .aimodel file mean more memory at runtime?

Not directly. Apple says AIModel does not own weights or buffers; the functions you load do. Measure memory around loadFunction(named:) on the device.