CPU-only safe path and memory

The safe path

SpecializationOptions has two presets: .default (all available compute units) and .cpuOnly. Apple's guide gives one use for .cpuOnly: a small model running in the background, so it does not compete with foreground GPU work. It is also the safe fallback when your Neural Engine or GPU checks fail on a device.

import CoreAI

enum ComputePlan: String { case neuralEngine, gpu, cpuOnly }

func options(for plan: ComputePlan) -> SpecializationOptions {
    switch plan {
    case .neuralEngine: return SpecializationOptions(preferredComputeUnitKind: .neuralEngine)
    case .gpu:          return SpecializationOptions(preferredComputeUnitKind: .gpu)
    case .cpuOnly:      return .cpuOnly
    }
}

/// Try plans in order. Skip units the device does not have.
func loadWithFallback(_ url: URL, plans: [ComputePlan], functionName: String = "main") async throws -> (InferenceFunction, ComputePlan) {
    let available = ComputeUnitKind.availableKinds
    var lastError: Error?
    for plan in plans {
        if plan == .neuralEngine && !available.contains(.neuralEngine) { continue }
        if plan == .gpu && !available.contains(.gpu) { continue }
        do {
            let model = try await AIModel(contentsOf: url, options: options(for: plan))
            if let fn = try model.loadFunction(named: functionName) {
                print("compute plan:", plan.rawValue)
                return (fn, plan)
            }
        } catch {
            lastError = error
            print("plan \(plan.rawValue) failed:", error)
        }
    }
    throw lastError ?? CocoaError(.featureUnsupported)
}

Call it with [.neuralEngine, .cpuOnly] or [.gpu, .cpuOnly]. Keep a remote switch per device model that forces .cpuOnly, so you can react to a bad result like #279 without an app update. Each plan is a separate specialization and cache entry, so a fallback costs a slow first load once.

Apple also warns that the default is the best choice in most cases. Override it only when your measurements say so.

Runtime memory is not file size

Apple's AIModel page says the model instance is lightweight and does not own weights or intermediate buffers. Those belong to the functions you load. InferenceFunction is Sendable and "automatically allocates additional intermediate buffers as needed to support concurrency". So:

• Memory goes up when you call loadFunction(named:), not when you create the AIModel.
• Each loaded function (prefill, decode, each context bucket you load) adds its own resources.
• Running one function from several tasks at once can add more buffers.

Measure it on the device:

import CoreAI
import os

func measureFunctionLoad(_ model: AIModel, name: String) throws -> (InferenceFunction?, Int) {
    let before = os_proc_available_memory()
    let function = try model.loadFunction(named: name)
    let after = os_proc_available_memory()
    let used = Int(before) - Int(after)
    print("loadFunction(\(name)) used about \(used / 1_048_576) MB")
    return (function, used)
}

This is an estimate, because other work changes the number too. Run it on a quiet app and repeat it. Use the result as the measured footprint in the memory gate from Fit the model in the memory budget, and add the KV cache or state buffers you pass in.

File formats: Core AI vs Core ML

Do not mix the two up in docs or code reviews. A .mlmodelc is not a Core AI file. For when to pick each stack, see Core AI vs Core ML vs MLX.

Sources

• Apple: SpecializationOptions
• Apple: AIModel
• Apple: InferenceFunction