Evaluate the clothing classifier

Why this check matters

A Fashion-MNIST accuracy number can look fine. But Shirt and T-shirt/top can still swap. A picked photo can be inverted. Or Core AI can disagree with the Mac weights. Before you call the SwiftUI label done, read the confusion matrix (which pairs get mixed up). Open-code the photos that fail: write short notes in your own words, then group those notes into a few problem types. Also check that the Mac and the device give the same top label.

Lesson 06 already measured load, specialization, and inference. This lesson does not repeat Instruments. It asks whether the label is right.

One step at a time

Environment setup

Use the same uv project as training. Apple Silicon MPS when available, otherwise CPU. Do not add CUDA.

cd clothing-coreai
uv add torch torchvision
uv run python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__); print('MPS:', torch.backends.mps.is_available())"
mkdir -p evals/traces evals/open-codes
uv run python evaluate_classifier.py

Held-out accuracy and confusion matrix

Reload ClothingCNN and clothing.pt. Run the official Fashion-MNIST test split with ToTensor() only. Print accuracy and a 10-by-10 confusion matrix. Shirt versus T-shirt/top is the pair that often gets mixed up. If that cell is large, a higher overall accuracy will not fix the app.

import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

from train import ClothingCNN

CLASSES = [
    "T-shirt/top", "Trouser", "Pullover", "Dress", "Coat",
    "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot",
]

if torch.backends.mps.is_available():
    device = torch.device("mps")
else:
    device = torch.device("cpu")

model = ClothingCNN()
model.load_state_dict(torch.load("clothing.pt", map_location="cpu", weights_only=True))
model.to(device).eval()

test = datasets.FashionMNIST("data", train=False, download=True, transform=transforms.ToTensor())
loader = DataLoader(test, batch_size=256)
matrix = torch.zeros(10, 10, dtype=torch.int64)

with torch.no_grad():
    for images, labels in loader:
        pred = model(images.to(device)).argmax(1).cpu()
        for y, y_hat in zip(labels, pred):
            matrix[y, y_hat] += 1

correct = int(matrix.diag().sum())
total = int(matrix.sum())
print(f"accuracy={correct / total:.3f} on {device}")
print(CLASSES)
print(matrix)

Open-code the misclassified photos

Accuracy does not tell you why a picked photo failed. Export the worst cells and a few real camera shots. For each one, write a short note in your own words. Then group those notes into a few problem types, such as inverted grayscale, a cropped torso, Shirt versus T-shirt mix-ups, a preprocess mismatch, or a class the camera should not have asked for.

{
  "id": "photo-coat-12",
  "source": "device",
  "true_or_expected": "Coat",
  "mac_top": "Coat",
  "coreai_top": "Pullover",
  "open_codes": [
    "sleeves fill the 28x28 frame",
    "Swift resize looks darker than ToTensor"
  ]
}

Mac versus Core AI parity

Parity here means the Mac and the device give the same top label. Run the same bundled sample and a few test tensors through PyTorch on the Mac and through the Core AI app. Write both top labels (and the ten logits if you can export them) to JSONL. A name or shape bug in conversion often shows up as the same swap every time, not as random noise.

import json

def parity_row(sample_id, mac_logits, coreai_logits, classes=CLASSES):
    mac_top = classes[int(mac_logits.argmax())]
    coreai_top = classes[int(coreai_logits.argmax())]
    return {
        "id": sample_id,
        "mac_top": mac_top,
        "coreai_top": coreai_top,
        "label_match": mac_top == coreai_top,
        "max_abs_logit_delta": float((mac_logits - coreai_logits).abs().max()),
    }

print(json.dumps(parity_row("fashion-sample", mac, coreai)))

If the labels match and the logits are close, conversion worked. If labels differ, stop judging "model quality" and reopen lesson 03 and the Swift image prep. Do not retrain this small classifier to hide a tensor contract bug.

Binary: top label acceptable

For the SwiftUI path, the release question is a simple yes or no. Is the displayed top label acceptable for this image? Shirt versus T-shirt/top may be acceptable in a demo if you write that down. Coat versus Sandal is not. Write that rule down. A 1-5 confidence score is extra. The real check is whether a person would accept the label.

Instruments stays in lesson 06

Gauge colors, specialization, and later inference stay in Profile the clothing inference path. Do not mix milliseconds into the confusion matrix. A fast wrong label is still a fail.

When this goes wrong

• Shipping on test accuracy alone. The camera photo is what users see.
• Ignoring the Shirt / T-shirt cell. That cell is the usual real-world miss.
• Calling a preprocess bug a model miss. Compare Mac and Core AI on the same tensor first.
• Measuring again instead of reading mistakes. Lesson 06 already covered Instruments.
• Adding CUDA to chase a higher number. This course is MPS or CPU.

Done when

• Held-out Fashion-MNIST accuracy and a printed confusion matrix exist for ClothingCNN.
• Open codes exist for misclassified test images and at least one camera photo.
• A JSONL compare shows Mac versus Core AI top labels on the bundled sample.
• A written yes/no rule defines when the top label is acceptable.
• Lesson 08 still confirms weights, .aimodel, prepare-once, and the real-device profile.

Keep learning

The same look-then-score path on voice is Evaluate the writing style. On generations it is Evaluate generations.