Grow the set without user text

Twenty hand-written samples will miss phrasings. Session 299 shows SampleGenerator and generable ModelSample and TrajectoryExpectation types, so you can ask a model to propose more rows. Review every generated row. Keep the rejects.

import Evaluations

// Illustrative. Confirm SampleGenerator in WWDC26 299 and the installed docs.
// Generate drafts from seed ModelSample rows. Keep only rows a person reviewed.
let generator = SampleGenerator()
let drafts: [ModelSample] = [] // call the documented generate API here
let kept = drafts.filter { rowWasReviewed($0) }

Default rule: do not put raw user transcripts in the repo or in CI. User text can hold names, addresses, and health notes. If you need real phrasing, get consent, strip identifiers, and store the row on a machine you control. Turn feedback into a better local model covers that loop.

The long loop is real use, then production data, then reviewed evals, then a better agent. Evals exist to find failures. Then you change the agent so that failure does not repeat. Prefer a harness, prompt, or tool fix first. Fine-tune later for a specific task when you have that infra.

A thumbs-down from the app can become a fixture without the original sentence. Store the feature, the prompt version, the tool path you expected, and a synthetic rewrite. That still catches the regression. Traces are the log of actions and tool calls. Use them as receipts: see a bad outcome, read the trace, find the wrong step, add a sample. Trace every step of an AI call is that log.

Humans updating verifiers and rubrics is often the bottleneck. The highest leverage is writing down what good looks like. Shrink the human-in-the-loop over time. Overnight improvement proposals only work if the suite already measures bad behavior.

Pin prompt versions on every sample. When you roll back a prompt, the same eval should go green. When you ship a new prompt, the eval must run before the app release.

Primary sources: Evaluations, WWDC26 298, and WWDC26 299.