---
name: diffusion-lora-coreai lesson 07
description: >-
  Agent recipe for lesson 07: error-analyze diffusion generations, open-code
  failures, axial-code buckets, then apply binary rubrics. CLIP is secondary.
---
# Evaluate generations

**Lesson id:** `07`  
**One-line goal:** Judge the baked diffusion artifact by reading a golden-set grid before anyone writes a score.

## Why this matters on a phone

A nice example image is not enough to decide if you can ship. Collect pre-merge, merged, and device images for the same prompts. The product is the baked resource.

## Prerequisites

Lessons 01 through 05 on one project. Merged export folder intact. On-device pipeline from lesson 05 can generate one image.

## Inputs

Subject or style contract, holdout prompts, fixed seeds, pre-merge previews, merged Mac images, device images.

## Outputs and artifacts

Golden-set JSON, image grids for `pre`, `merged`, and `device`, open-code notes (plain words on each image), four axial buckets (shared groups), yes-or-no checks, optional CLIP ranks.

## Executable steps

1. Recreate the `uv` env. Confirm MPS or CPU. Do not add CUDA.
2. Write 8 to 20 golden rows: prompt, seed, success sentence. Include a negative and a prompt-ignore case.
3. Generate the grid at pre-merge, after merge, and on device. Same prompt and seed. `runtime_lora_swap: false`.
4. Open-code each image before naming buckets. Write what you see in plain words.
5. Collapse notes into subject missing, style miss, artifacts, and prompt ignore. Pick one person who knows the product to own the list of buckets.
6. Write a yes-or-no PASS/FAIL check per bucket. Use CLIP only to sort the queue.
7. Keep Instruments and elapsed time out of the quality check.

## Constraints

Cite https://youtu.be/BsWxPI9UM4c once. Paraphrase only. No CUDA. Merge the LoRA, then export; do not evaluate a hot-swapped adapter. No quizzes, Mermaid, or em dashes. Yes-or-no checks, not 1-5 scores.

## Failure modes

Hand-picked collage, CLIP or FID as the release check, averaged Likert scores, Mac-only grids, evaluating the adapter instead of the merged export.

## Key concepts

- Look before you score: open-code, then axial buckets, then yes or no checks.
- Build a golden set of fixed prompt, seed, and success-sentence contracts.
- Generate three grids: pre-merge, merged Mac, and on-device.
- CLIP is a secondary alarm. Speed belongs in Instruments, not in quality eval.

## Takeaways

- One taste owner decides the buckets. Do not average 1-5 scores.
- Judge the baked device resource, not the adapter.
- If the device grid looks like the base, check the resource folder first.

## Acceptance checks

- [ ] Golden set has prompts, seeds, and success sentences.
- [ ] Pre-merge, merged, and device grids exist for the same rows.
- [ ] Open codes collapse into the four buckets under one shared list.
- [ ] Each bucket is yes-or-no. CLIP is optional and secondary.
- [ ] No runtime LoRA swap is claimed.

**Human lesson:** [Evaluate generations](07-evaluate-generations.html)