Tools and traces help. They do not choose the bar. A person has to write what good looks like. That writing is the highest leverage work on an agent. It lives in the control layer: golden sets, guardrails, and approvals.
A golden set is a list of cases the team agrees on. Each row has an input, the tools that must run or must not run, and a short rubric. "Cites the finance table" is a rubric. "Stops when the metric is not in the pack" is a rubric. If two reviewers would grade the same row differently, the bar is not written yet.
Approvals sit next to the golden set. Some actions need a named person before the agent continues: sending a customer email, posting a number to a board deck, changing a production record. The owner is a role you can page, not a group chat. When the suite disagrees with a stakeholder, that owner decides which side the harness will encode.
Teach the agent when to stop and ask. Low confidence, missing evidence, a tool error, or a request outside the pack are stop rules. A stop is a successful control outcome. A guessed KPI is not. The research-agent checks from lesson 01 belong here: confirm who asked, stay on approved data, cite, and halt when the pack is empty.
You will shrink how often a person sits in the loop. You do that by writing more of the bar into goldens and guardrails, not by hoping the next model "just knows." Overnight improvement ideas only work if the suite already measures bad behavior.
The model gets the hype. The harness gets the results.
For the Apple-side eval types, keep Test AI behavior before you ship nearby. For the hop log, keep Trace every step of an AI call.
Key concepts
- The highest leverage is a human writing what good looks like.
- The control layer holds the golden set, guardrails, and approvals.
- A named owner decides when the suite and a stakeholder disagree.
- Stop-and-ask is a success when evidence is missing or confidence is low.
- You shrink human review by encoding more of the bar, not by skipping the bar.
Takeaways
- Write rubrics two reviewers can grade the same way.
- Give approvals a person you can page, not a shared inbox.
- Treat a clean stop as a pass. Treat a guessed number as a fail.
- Remember the closer: the model gets the hype. The harness gets the results.