← All courses ← Course home

Lesson 01 of 05

Equality tests fail on language models

A unit test wants the same string every time. A language model will not give you that. You need a grade, not an equals.

Agent brief (llms.md)

A normal test says the function returns 4. A language model can return two good answers that do not match. If you assert equality, the build is red for a good reply, or green for one memorized sentence that later rots.

Input2 + 2
AssertEquals 4
Input2 + 2
AssertEquals 4
ModelMany good strings

You still need a test. You change the question. Ask how often the answer is good enough, and whether the model called the right tools. That is an evaluation. An eval tests whether what the agent did is right. "Right" is what you and your team encode into software. Evals are not a new category. They are an evolved version of the unit tests you already write. Apple shipped this as the Evaluations framework in the 27 release.

Do not only check the final string. Check the steps and side effects. On a phone that can mean a meeting was logged, a follow-up draft was written, and the agent paused before send. Two good sentences can still hide a skipped write.

import Testing

@Test
func refundCopyMatchesOneString() {
    let reply = "You can refund order 1842 within 30 days."
    // This fails on a second good sentence. Do not ship this assert.
    #expect(reply == "Refunds are allowed for 30 days after delivery.")
}

Run evaluations in Xcode 27 with Swift Testing, usually on a Mac. They are not a runtime feature you ship to the phone. The phone still runs the session. The Mac grades it before you ship.

A prompt version from Version prompts like product copy is an input to the eval, not a comment in a chat. A wrong tool path you saw in Trace every step of an AI call becomes a sample in the set.

Rule. Do not assert the full model string. Grade the behavior.

Next, meet the official types.

Key concepts

  • Unit tests expect one exact output. Models produce many good strings.
  • Equality goes red on good replies or green on one memorized sentence that rots.
  • An eval encodes what the team thinks is right. It is an evolved unit test, not string equality.
  • Evaluation asks how often the answer is good enough and whether tools and side effects were correct.
  • Run Evaluations in Xcode 27 Swift Testing on a Mac, not in the shipped app.

Takeaways

  • Do not assert the full model string. Grade the behavior. Encode what "right" looks like.
  • The Mac grades before you ship. The phone still runs the session.
  • Many good strings beat one frozen golden sentence.