EDGE FDE
IntermediateOn-Device AI Eval Harness for CI
Catch regressions in CI so a prompt change does not quietly break the product.
Evaluations harness with trajectory grading and Swift Testing CI gates.
Grade behavior, not string equality. Fail the pull request when ToolsAllPass drops. iOS 27 and macOS 27 only.
What you'll learn
- Easy: an eval tests whether what the agent did is right. You encode "right" in software. Evals are evolved unit tests: agent, system, tasks, and a verifier.
- Hard: the environment is where the agent works and you measure it. An eval suite is tasks (prompt + verifier + environment). Re-run when a model or prompt changes. Use fixtures, not the live user vault.
- God: real use becomes reviewed fixtures. Evals find failures so you can change the agent. Fix the harness first. Traces are receipts for the wrong step.
Notes that shaped this framing: Alex Lieberman (@businessbarista) on evals, from a call with Viv Trivedy (@Vtrivedy10). The rest of the course stays on Apple Evaluations in Swift.
Lessons
Key concepts
- String equality is the wrong test. An eval encodes what the team thinks is right. Evals are evolved unit tests.
- Grade behavior with Apple's Evaluations framework on Xcode 27 Mac. Recipe: agent + system + tasks + verifier.
TrajectoryExpectationandToolCallEvaluatorcover tool paths and side effects.- Gate CI with a mean metric threshold the team can say out loud. That number is the suite's encoded "right."
Takeaways
- You can grade Foundation Models behavior in CI instead of matching full strings.
- You can fail a pull request when mean
ToolsAllPassdrops under the team threshold. - You can grow a reviewed sample set without putting raw user text in the default repo.
- You can turn a production miss into a fixture, then fix the harness before you fine-tune.
Agent briefs
FAQ
Why not test LLM output with string equality?
String equality is the wrong test for language models. Grade behavior instead. An eval encodes what the team thinks is right.
Which framework does the course use?
Apple's Evaluations framework on Xcode 27 on a Mac, with Swift Testing CI gates. iOS 27 and macOS 27 only.
How do I grade tool calls?
TrajectoryExpectation and ToolCallEvaluator cover tool paths and side effects, so you grade what the agent did, not just the final text.
How does CI fail a pull request?
Gate CI on a mean metric threshold the team can say out loud. The pull request fails when mean ToolsAllPass drops below it.
Do evals need real user text?
No. Use fixtures, not the live user vault. Grow a reviewed sample set without putting raw user text in the default repo.
Related courses
Stream tokens and measure speed · Trace every step of an AI call · Version prompts like product copy · Turn feedback into a better local model · AI Full-Stack Operator · Test Local Decision Models (/v1/systemone)