Evaluate generations

Why this matters when the model runs on a phone

A personalized generator can miss the subject, drift off the style you wrote down, smear artifacts, or ignore the prompt while still looking visually polished. Those failures do not show up in one hand-picked seed. If you put the model on a phone, collect a small golden set. Generate it on the Mac preview, after merge, and on the device. Then read the images before anyone invents a score.

This lesson does not relax merge-then-export. You still bake the LoRA into the base (merge the LoRA into the base and export one model file) and ship the merged .aimodel folder. Evaluation judges that baked resource. It does not attach an adapter at runtime.

The steps, one box at a time

Environment setup

Use the same Mac uv project. MPS or CPU. No CUDA. Keep the device app from lesson 05 so the third generation pass is the baked resource, not a notebook preview.

cd diffusion-lora-coreai
uv run python -c "import torch; print(torch.__version__); print('MPS:', torch.backends.mps.is_available())"
mkdir -p evals/golden evals/pre evals/merged evals/device evals/open-codes
uv run python generate_eval_grid.py --stage pre --out evals/pre
uv run python generate_eval_grid.py --stage merged --out evals/merged

Build a golden set

Write eight to twenty contracts you can pass or fail. Each row is a prompt, a seed, and one success sentence from lesson 02. Include subject shots, style shots, a negative (the subject must not appear), and one prompt the model liked to ignore during training.

{
  "id": "mug-window-04",
  "prompt": "the blue enamel mug on a sunlit windowsill, product photo",
  "seed": 1042,
  "stage": "device",
  "success": "The mug is recognizable as the training subject. The sill is secondary.",
  "runtime_lora_swap": false
}

Generate the grid three times: base or pre-merge preview, merged Mac export, and on-device pipeline. Same prompt. Same seed. If merge looks right and the device looks like the base, the resource folder or tokenizer is the first suspect.

Open-code the pictures

Sit with the grid. Open-code means you write what failed in plain language before you name a category.

{
  "id": "mug-window-04",
  "stage": "device",
  "open_codes": [
    "handle is missing",
    "enamel color is close",
    "extra mug in the background"
  ]
}

Axial buckets and one taste owner

Axial means you group those notes into a few shared buckets. Four buckets cover this course:

• Subject missing. The subject you wrote down is gone or swapped.
• Style miss. The subject is present but the look is generic.
• Artifacts. Melted edges, extra limbs, text soup, broken geometry.
• Prompt ignore. The model drew its favorite scene instead of the ask.

Pick one person who knows the product well to decide what good and bad look like. Reviewers can challenge a bucket. They should not each invent their own list. If two people cannot put the same image in the same bucket, the definition is still too fuzzy.

Yes-or-no checks. CLIP is second.

Turn each bucket into a yes-or-no. "Is the training mug present?" is a rubric. "Quality 1-5" is not. Code can catch some artifacts (empty file, tiny dimensions, near-duplicate of the base). Subject and style still need a human or a yes-or-no judge aligned to that human.

RUBRIC = """
Answer PASS or FAIL only for this one question:
Does the image show the contracted subject from the success sentence?

Success sentence: {success}
Prompt: {prompt}
If the subject is absent, swapped, or unrecognizable, FAIL.
"""

CLIP similarity to a reference photo can be a secondary alarm. It is not the eval. A high CLIP score can still miss the style contract. A low score can flag a crop or a color shift that a person would still accept. Use it to sort the queue, then read the pictures.

Instruments stays separate

Elapsed time, the extra time to get the model ready, and memory belong to the on-device pipeline lesson and to Xcode Instruments. Do not mix speed into the quality check. A fast wrong image is still a fail. A slow correct image is a performance bug, not a LoRA miss.

When this goes wrong

• Scoring a hand-picked collage instead of the golden set. Hand-picked pictures are for a demo, not a test.
• Letting CLIP or FID be the release check. Those numbers do not know your subject contract.
• Averaging three reviewers into a 3.4. Fix the bucket or pick one person to decide.
• Judging only the Mac preview. The product is the baked device resource.
• Calling the adapter the thing you evaluated. You evaluated the merged export, or you skipped the real path.

Done when

• A golden set exists with prompts, seeds, and success sentences.
• Pre-merge, merged, and device grids are saved for the same rows.
• Open codes collapse into the four buckets, owned by one shared list.
• Each bucket has a yes-or-no pass or fail. CLIP is optional and secondary.
• Lesson 08 still confirms the notes that let someone rebuild, merge-then-export, and the device measurement.

Keep learning

The same error-analysis path on writing style is Evaluate the writing style. On labels it is Evaluate the clothing classifier.

PyTorch to Core AI in Xcode · From the metal to the model · Core AI models, typed · Model architectures in plain English · LLM LoRA for your writing style