Catch silent Neural Engine fallback

Two silent failures

The repo's common_issues.md also lists "Neural Engine model compiles but runs on CPU", with the fix: compile with xcrun coreai-build compile model.aimodel --preferred-compute neural-engine.

Check 1: count Neural Engine regions after the build

The reporter of #205 counted Neural Engine regions in the compiled bundle. This is not a documented Apple API, it is a file-name check from that issue. It is still a useful CI guard while there is no exit code:

#!/usr/bin/env bash
set -euo pipefail
xcrun coreai-build compile MyModel.aimodel --platform iOS \
  --min-deployment-version 27.0 --preferred-compute neural-engine \
  --output compiled/ 2>&1 | tee coreai-build.log

if grep -q "ANECCompileOffline() failed" coreai-build.log; then
  echo "Neural Engine compile failed (see coreai-build.log)"; exit 1
fi
for bundle in compiled/*.aimodelc; do
  n=$(find "$bundle" -name '*ANE_region*' | wc -l | tr -d ' ')
  echo "$bundle: $n Neural Engine regions"
  if [ "$n" -eq 0 ]; then echo "no Neural Engine regions in $bundle"; exit 1; fi
done

Check xcrun coreai-build compile --help for the exact value names on your toolchain.

Check 2: read the trace

Apple says the Core AI Instruments template shows "which compute units run your model". Record one inference on device and confirm the work sits on the Neural Engine track you expect. Do this for each new export and each OS major version. Write the result in the model's release notes.

Check 3: compare the numbers per compute unit

Running on the Neural Engine is not enough. #279 ran there and was wrong. Compare outputs from the same inputs across compute units:

import CoreAI

/// Run the same input with different specialization options and return the outputs for comparison.
func runOn(_ options: SpecializationOptions, modelURL: URL, inputName: String, input: NDArray, outputName: String) async throws -> NDArray? {
    let model = try await AIModel(contentsOf: modelURL, options: options)
    guard let function = try model.loadFunction(named: "main") else { return nil }
    var outputs = try await function.run(inputs: [inputName: input])
    return outputs.remove(outputName)?.ndArray
}

func cosine(_ a: [Float], _ b: [Float]) -> Float {
    var dot: Float = 0, na: Float = 0, nb: Float = 0
    for i in 0..<min(a.count, b.count) { dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i] }
    return dot / (na.squareRoot() * nb.squareRoot() + 1e-12)
}

// let ne  = try await runOn(SpecializationOptions(preferredComputeUnitKind: .neuralEngine), ...)
// let cpu = try await runOn(.cpuOnly, ...)
// Read both with .view() into [Float] for your scalar type, then compare with cosine(_:_:).

Apple documents init(preferredComputeUnitKind:) as "maximizes use of the specified compute unit kind, falling back to other allowed compute units for incompatible operations". So a preference is not a guarantee. That is why check 2 matters.

For a deeper look, Apple's Core AI Debugger comparison lets you pick the compute unit for a session and compare every operation against a PyTorch reference run, with a similarity score per operation pair. Use it when check 3 shows a gap and you need to find the layer.

Sources

• Apple: Validating inference correctness against a reference run
• Apple: SpecializationOptions init(preferredComputeUnitKind:)