The loop that improves the harness

The harness is the moat. A new model can help. It will not save a loose pack, a vague tool, or a missing check. Improve the harness from real work. Fine-tune later, and only when the suite already fails for a reason the weights must learn.

flowchart LR
  P[Production run] --> B[Bad outcome]
  B --> T[Read traces]
  T --> G[Add golden case]
  G --> H[Change harness]
  H --> E[Re-run eval]
  E --> P
      

Walk it once. A production run gives a bad outcome: a wrong KPI, a missing cite, a tool that never fired. Do not open a training job. Open the traces. Traces are receipts. They show which hop failed: pack, tool, model, or guardrail. Trace every step of an AI call is how you name those hops.

Turn the miss into a golden case. Write the input, the expected tool path, and a short rubric: the answer must cite evidence, or it must stop. That row is now a test. Test AI behavior before you ship is the Apple-side eval shape for the same idea.

Change one harness piece. Tighten the prompt or the instructions. Narrow the tool definition. Add a guardrail. Or change the model choice in runtime config. Re-run the eval. If the new row passes and the old rows still pass, you improved the harness. If you cannot say which change did it, you changed too much.

The secret-margin question from lesson 03 is already a golden case. The harness should stop. If a later model starts guessing a margin, the verifier stays red until you fix the pack or the stop rule. That is harness engineering. It is cheaper than fine-tuning, and you can ship it this week.

Next, put a human on the control layer. That is the highest leverage step.