EDGE FDE
IntermediateTest AI behavior before you ship
Catch regressions in CI so a prompt change does not quietly break the product.
Evaluations harness with trajectory grading and Swift Testing CI gates.
Grade behavior, not string equality. Fail the pull request when ToolsAllPass drops. iOS 27 and macOS 27 only.
What you'll learn
- Easy: an eval tests whether what the agent did is right. You encode "right" in software. Evals are evolved unit tests: agent, system, tasks, and a verifier.
- Hard: the environment is where the agent works and you measure it. An eval suite is tasks (prompt + verifier + environment). Re-run when a model or prompt changes. Use fixtures, not the live user vault.
- God: real use becomes reviewed fixtures. Evals find failures so you can change the agent. Fix the harness first. Traces are receipts for the wrong step.
Notes that shaped this framing: Alex Lieberman (@businessbarista) on evals, from a call with Viv Trivedy (@Vtrivedy10). The rest of the course stays on Apple Evaluations in Swift.
Lessons
Key concepts
- String equality is the wrong test. An eval encodes what the team thinks is right. Evals are evolved unit tests.
- Grade behavior with Apple's Evaluations framework on Xcode 27 Mac. Recipe: agent + system + tasks + verifier.
TrajectoryExpectationandToolCallEvaluatorcover tool paths and side effects.- Gate CI with a mean metric threshold the team can say out loud. That number is the suite's encoded "right."
Takeaways
- You can grade Foundation Models behavior in CI instead of matching full strings.
- You can fail a pull request when mean
ToolsAllPassdrops under the team threshold. - You can grow a reviewed sample set without putting raw user text in the default repo.
- You can turn a production miss into a fixture, then fix the harness before you fine-tune.
Agent briefs
Related courses
Stream tokens and measure speed · Trace every step of an AI call · Version prompts like product copy · Turn feedback into a better local model