---
title: "Gate CI on metrics"
lesson_id: "04"
---

# Gate CI on metrics

- **Lesson ID:** 04
- **Goal:** Fail CI with #expect on an aggregate metric. State where the tests run and how a skip must look.
- **Human lesson:** [04-gate-ci.html](04-gate-ci.html)

## Prerequisites

- Lesson 03, or tool metrics you trust.

## Inputs, outputs, and artifacts

- **Inputs:** Aggregate metrics from an Evaluation.
- **Outputs:** A red pull request when the mean drops.
- **Artifacts:** Progressive `.flow` diagrams and a gate sketch.

## Agent build steps

1. Show #expect on mean ToolsAllPass. Call the threshold the team's encoded "right."
2. Define an eval suite as tasks. Each task is a prompt, a verifier, and an environment of fixtures, not the live vault.
3. Say re-run the suite when a model or prompt changes and weigh cost, speed, and quality.
4. Optional one line: Harbor-style env abstractions or a LangSmith-style judge UI. Stay on Apple Evaluations.
5. Say tests run on a Mac with the 27 SDK.
6. Say a skip is not a pass.
7. Link streaming metrics and prompt registry for the failure follow-up.

## Constraints

Keep the language simple, friendly, and literal. iOS 27 and macOS 27 only. No quizzes, Mermaid, or em dashes. Do not invent API names. Mark sketches as sketches and tell the reader to confirm them in the Xcode 27 SDK.

## Key concepts

- The threshold is the team's encoded "right," for example mean `ToolsAllPass` at 0.9.
- An eval suite is tasks. Each task is a prompt, a verifier, and an environment of fixtures, not the live vault.
- Run the eval on every pull request on a Mac agent with the 27 SDK.
- A skip because the model is missing is not a pass.
- Gate first on code metrics and `ToolsAllPass`.

## Takeaways

- A prompt change that drops `ToolsAllPass` under the team number is a red build.
- Re-run the suite when a model or prompt changes. Weigh cost, speed, and quality.
- Open the evaluation report and read the worst samples when the gate fails.
- Check streaming metrics for TTFT moves and the prompt registry for version flips.

## Acceptance checks

- A numeric gate is present.
- CI host limits are stated.
- Sibling links, progressive `.flow` diagrams, and an Agent brief link are present.
- [ ] Proceed to [lesson 05 brief](05-grow-the-set.llms.md).