# Evaluate the clothing classifier

- **Lesson ID:** 07
- **Goal:** Look at ClothingCNN mistakes, compare Mac versus Core AI labels, and apply a yes/no top-label-acceptable rule.
- **Human lesson:** [07-evaluate-classifier.html](07-evaluate-classifier.html)

## Prerequisites

- `clothing.pt`, `ClothingCNN` in `train.py`, and a runnable SwiftUI app from lessons 01 through 06.
- A Mac with MPS or CPU. No CUDA.

## Inputs, outputs, and artifacts

- **Inputs:** Fashion-MNIST test split, `clothing.pt`, bundled sample, optional camera photos, Core AI top labels.
- **Outputs:** Held-out accuracy, 10-class confusion matrix, open-code notes, Mac-vs-Core-AI JSONL, yes/no accept rule.
- **Artifacts:** `evals/traces`, `evals/open-codes`, printed matrix, parity rows for `fashion-sample`.

## Agent build steps

1. Recreate the `uv` env with `torch` and `torchvision`. Confirm `torch.backends.mps.is_available()`. Never add CUDA.
2. Import `ClothingCNN` from `train`, load `clothing.pt`, run the Fashion-MNIST test split, print accuracy and the confusion matrix.
3. Open-code misclassified test images and at least one picked photo. Write short notes in your own words, then group them into a few problem types (invert, crop, Shirt/T-shirt, preprocess mismatch).
4. Compare Mac logits or top labels to Core AI on the bundled sample. Write JSONL. If labels differ, reopen conversion and Swift image prep before blaming the net.
5. Write a yes/no "top label acceptable" rule. Do not use a 1-5 score as the release check.
6. Leave Instruments, gauge, and specialization in lesson 06.

## Constraints

Cite https://youtu.be/BsWxPI9UM4c once. Paraphrase only. Mac MPS/CPU only; no CUDA. Core AI, not Core ML. Keep the float32 `[1,1,28,28]` `image` to `[1,10]` `logits` contract and the fixed class order. No em dashes.

## Key concepts

- Print held-out Fashion-MNIST accuracy and a 10x10 confusion matrix with `ToTensor()` only.
- Shirt versus `T-shirt/top` is the confusion cell that often breaks real photos.
- Open-code misclassified images, then group them into a few problem types.
- Compare Mac and Core AI top labels on the same sample. The release check is yes or no, not a 1-5 score.

## Takeaways

- High test accuracy does not mean the camera path is done.
- If Mac and Core AI top labels differ, fix conversion or Swift prep before you retrain.
- Keep Instruments in lesson 06. Do not mix timing into the confusion matrix.

## Acceptance checks

- [ ] Held-out accuracy and a 10-by-10 confusion matrix print from `ClothingCNN`.
- [ ] Open codes exist for misclassified test images and one camera photo.
- [ ] JSONL shows Mac versus Core AI top labels on the bundled sample.
- [ ] A written yes/no rule defines an acceptable top label.
- [ ] No Instruments work is duplicated from lesson 06.

## Environment setup

```bash
cd clothing-coreai
uv add torch torchvision
uv run python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__); print('MPS:', torch.backends.mps.is_available())"
uv run python evaluate_classifier.py
```