Grade the tool trajectory

A good final sentence can still be a bad agent. The model may have skipped a lookup, or called a write tool on a read-only question. A trajectory expectation lists the tool calls you want in the transcript. That is how you grade the path: did it call the right tools, and did it stop before a send?

Think of a notes agent that should log the meeting, write a follow-up draft, and pause before send. The eval asks whether those tools ran and whether the send tool stayed unused. You are checking side effects and steps, not only the last sentence.

import Evaluations

let toolsAllPass = Metric("ToolsAllPass")
let toolsPercentagePass = Metric("ToolsPercentagePass")

struct LightControlEvaluation: Evaluation {
    var dataset = samples

    var evaluators: Evaluators {
        ToolCallEvaluator(allPass: toolsAllPass, percentagePass: toolsPercentagePass)
    }

    func aggregateMetrics(using aggregator: inout MetricsAggregator) {
        aggregator.computeMean(of: toolsAllPass)
        aggregator.computeMean(of: toolsPercentagePass)
    }
}

Apple documents ordered expectations, unordered expectations, disallowed tools, and group steps. Argument matchers include exact, key-only, range, contains, and natural-language matches. Confirm the enum cases in the docs. The sketch above matches the public ToolCallEvaluator overview.

Write one sample from a real bug. If a trace showed a missing searchBooks call, that prompt and the expected tool become a row. Trace every step of an AI call is how you found it.

Next, fail the build when the mean drops.