Gate CI on metrics

Session 298 calls the threshold an optimization target. You pick a number the team can defend, such as mean ToolsAllPass at or above 0.9, and you fail the test when a prompt change drops it. That number is the team's encoded "right."

An eval suite is the collection of those tasks. Each task is an input prompt, a verifier, and an environment. The environment is the place the agent works and you measure it. Agents do real work, not only text in and text out, so you need a stand-in for the user's files and tools. On device that means fixtures and a fake vault, not live user data. When a new model or prompt ships, re-run the suite and weigh cost, speed, and quality. Some teams use Harbor-style environment abstractions, or a UI such as LangSmith for people who judge outputs without living in code. This course stays on Apple Evaluations in Xcode.

@Test(.evaluates(LightControlEvaluation()))
func lightControl() async throws {
    let result = EvaluationContext.current.result
    let evaluation = LightControlEvaluation()
    let strict = result.aggregateValue(.mean(of: evaluation.toolsAllPass))
    #expect(strict > 0.9)
}

Run these tests on a Mac agent that has the 27 SDK. They are slower than unit tests. Keep them in a test plan you still run on every pull request, not once a quarter. If the on-device model is missing on the CI host, skip with a clear reason. Do not mark a skip as a pass.

When the gate fails, open the evaluation report. Read the worst samples. If TTFT also moved, check Stream tokens and measure speed. If a profile version flipped, check Version prompts like product copy and roll back.

Do not gate on a judge score you have not calibrated. Gate first on code metrics and ToolsAllPass. Add a judge threshold after you have read the rationales.

Next, grow the set without copying user chats.