---
name: Test AI behavior before you ship
description: >-
  Teach an Edge FDE to grade Foundation Models behavior on iOS 27 and macOS 27
  with Apple's Evaluations framework, including trajectory grading and CI gates.
---
# Test AI behavior before you ship

Use this course when a prompt change can break a feature without failing a unit test. An eval tests whether what the agent did is right. You encode "right" in software. Evals are evolved unit tests. Apple's Evaluations framework is the official harness in the 27 SDK. It is a developer tool you run in tests, usually on a Mac, not a runtime you ship inside the app.

Cite [Alex Lieberman (@businessbarista) on evals](https://x.com/businessbarista/status/2097478065723162959) once on the course home. Do not repeat the link in every lesson. Do not turn the course into Harbor or LangSmith marketing. A one-line mention is enough if environments or suites come up.

## End-to-end agent recipe

1. Start with [lesson 01](01-equality-tests-fail.llms.md). Easy mode: evals as evolved unit tests. Encode "right." Grade steps and side effects, not string equality.
2. Use [lesson 02](02-evaluations-framework.llms.md) for `Evaluation`, `ModelSample`, `Metric`, and `.evaluates`. Map agent + system + tasks + verifier onto Evaluation, samples, and evaluators. Rubrics and `ModelJudgeEvaluator` for non-binary "good."
3. Use [lesson 03](03-grade-the-trajectory.llms.md) for `TrajectoryExpectation` and `ToolCallEvaluator`. Grade the path: right tools, pause before send.
4. Use [lesson 04](04-gate-ci.llms.md) to fail CI when a mean metric drops under a threshold. Hard mode: eval suite, fixtures not live vault, re-run when models or prompts change.
5. Use [lesson 05](05-grow-the-set.llms.md) to grow samples with `SampleGenerator` and to keep raw user text out of the default set. God mode: production to reviewed fixtures, traces as receipts, harness first.
6. Keep these links exact: [Stream tokens and measure speed](../on-device-streaming-metrics/), [Trace every step of an AI call](../on-device-llm-tracer/), [Version prompts like product copy](../on-device-prompt-registry/), and [Turn feedback into a better local model](../on-device-data-flywheel/).
7. Cite [Meet the Evaluations framework](https://developer.apple.com/videos/play/wwdc2026/298/), [Create robust evaluations for agentic apps](https://developer.apple.com/videos/play/wwdc2026/299/), and [Evaluating tool-calling behavior](https://developer.apple.com/documentation/evaluations/evaluating-tool-calling-behavior).
8. Use the human HTML pages for the reader. Use the `.llms.md` briefs for agent steps and acceptance checks.

## Constraints

- Plain spoken English for a non-native reader.
- Literal sentences. No flourish, no quizzes.
- Do not use em dashes.
- Use no Mermaid. Build progressive `.flow` diagrams in HTML.
- Do not invent evaluator names. Use Evaluation, ModelSample, Metric, ToolCallEvaluator, TrajectoryExpectation, ModelJudgeEvaluator, SampleGenerator, and `.evaluates`.
- Teach in Swift Testing with Evaluations. Do not use Python eval scripts for the Foundation Models harness.
- Mark call sketches as sketches. Confirm them in the Xcode 27 Evaluations docs.
- iOS 27, macOS 27, and Xcode 27 only.

## Key concepts

- String equality is the wrong test. An eval encodes what the team thinks is right. Evals are evolved unit tests.
- Grade behavior with Apple's Evaluations framework on Xcode 27 Mac. Recipe: agent + system + tasks + verifier.
- `TrajectoryExpectation` and `ToolCallEvaluator` cover tool paths and side effects.
- Gate CI with a mean metric threshold the team can say out loud. That number is the suite's encoded "right."

## Takeaways

- You can grade Foundation Models behavior in CI instead of matching full strings.
- You can fail a pull request when mean `ToolsAllPass` drops under the team threshold.
- You can grow a reviewed sample set without putting raw user text in the default repo.
- You can turn a production miss into a fixture, then fix the harness before you fine-tune.

## Acceptance checks

- Every human lesson HTML has **Key concepts** and **Takeaways** sections before lesson-nav. The course home has the same two headings for the whole course.
- Five human lessons exist with the requested basenames.
- Five matching `.llms.md` briefs exist and link to their human HTML pages.
- Every human lesson has its Agent brief link and sibling-course links.
- The course index lists all five lessons, the course SKILL.md, and all five briefs.
- The course home has a short What you'll learn section (Easy, Hard, God) and cites the Business Barista eval notes once.
- Diagrams are progressive HTML `.flow` elements. There is no Mermaid, no em dash, and no quiz.

## Environment setup

No runtime environment is required. The short code samples are sketches of public APIs, not a project to run. Evaluations run as tests in Xcode 27, typically on a Mac.