---
title: "Equality tests fail on language models"
lesson_id: "01"
---

# Equality tests fail on language models

- **Lesson ID:** 01
- **Goal:** Explain why string equality is the wrong assert. An eval encodes what the team thinks is right. Evals are evolved unit tests. Evaluations is the 27 SDK harness.
- **Human lesson:** [01-equality-tests-fail.html](01-equality-tests-fail.html)

## Prerequisites

- A Foundation Models feature the reader can describe in one sentence.

## Inputs, outputs, and artifacts

- **Inputs:** A flaky equality test.
- **Outputs:** The idea of a graded metric.
- **Artifacts:** Progressive `.flow` diagrams and a Swift Testing equality fail.

## Agent build steps

1. Show a Swift Testing #expect string equality that fails on a second good reply.
2. Say an eval tests whether what the agent did is right, and that "right" is encoded in software.
3. Say evals are evolved unit tests. Check steps and side effects, not only the final string.
4. Name Evaluations as the 27 developer-tool harness.
5. Say it runs in tests, usually on a Mac.
6. Link prompt registry and tracer.

## Constraints

Keep the language simple, friendly, and literal. iOS 27 and macOS 27 only. No quizzes, Mermaid, or em dashes. Do not invent API names. Mark sketches as sketches and tell the reader to confirm them in the Xcode 27 SDK.

## Key concepts

- Unit tests expect one exact output. Models produce many good strings.
- Equality goes red on good replies or green on one memorized sentence that rots.
- An eval encodes what the team thinks is right. It is an evolved unit test, not string equality.
- Evaluation asks how often the answer is good enough and whether tools and side effects were correct.
- Run Evaluations in Xcode 27 Swift Testing on a Mac, not in the shipped app.

## Takeaways

- Do not assert the full model string. Grade the behavior. Encode what "right" looks like.
- The Mac grades before you ship. The phone still runs the session.
- Many good strings beat one frozen golden sentence.

## Acceptance checks

- Equality is rejected as the main assert.
- Evaluations is named as official.
- Sibling links, progressive `.flow` diagrams, and an Agent brief link are present.
- [ ] Proceed to [lesson 02 brief](02-evaluations-framework.llms.md).