---
name: llm-lora-writing-style lesson 07
description: >-
  Agent recipe for lesson 07: read writing-style traces, write short notes
  (open-code), group them into axial buckets, then automate code checks and a
  binary judge.
---
# Evaluate the writing style

**Lesson id:** `07`  
**One-line goal:** Judge the baked writing-style artifact by reading traces first, then automating only the buckets those traces justify.

## On-device job

Evaluation is part of the delivery. Mac merge traces and device chat traces both matter. A nice demo prompt is not a release check.

## Prerequisites

Lessons 01 through 06 on one project. Merged model exported. Tokenizer sidecar present. On-device chat from lesson 06 can return a local reply.

## Inputs

Review prompts, Mac merged-model replies, device replies, writing secret-scan list, author voice samples.

## Outputs and artifacts

`evals/traces/mac.jsonl`, `evals/traces/device.jsonl`, open-code notes, four axial buckets, code-check script, binary voice rubric, confusion matrix, held-out prompt list.

## Executable steps

1. Recreate the `uv` env. Confirm MPS or CPU. Do not add CUDA.
2. Collect the same prompts on the Mac merged model and on the device chat. Write JSONL. Record `runtime_lora_swap: false`.
3. Open-code 20 to 50 traces before naming a score. Open coding means short notes in your own words.
4. Group notes into voice drift, secret leakage, refusal, and formatting. Those groups are the axial buckets.
5. Automate secrets and format with code. Add a binary PASS/FAIL voice judge only after the buckets are stable.
6. Align the judge with a human confusion matrix, not an agreement percent.
7. Freeze a held-out slice and rerun checks after every merge-then-export.

## Constraints

Cite https://youtu.be/BsWxPI9UM4c once. Paraphrase only. No CUDA. Merge the LoRA, then export; do not judge a hot-swapped adapter. No quizzes, Mermaid, or em dashes. Binary judges, not 1-5 scores. One short paragraph may point at [Test AI behavior before you ship](../on-device-eval-harness/) for the unit-test-to-eval idea on Foundation Models agents.

## Failure modes

Tests before traces, too many buckets, LLM secret scanning, agreement percent that hides false passes, Mac-only traces, treating a tokenizer mismatch as a training bug.

## Key concepts

- Read traces first: Mac merged model and device chat for the same review prompts.
- Open-code, then four buckets: voice drift, secret leakage, refusal, formatting.
- Automate secrets and format with checks. Voice gets a binary judge.
- Judge the baked model, not the adapter.

## Takeaways

- If Mac and device disagree, check the bake or the tokenizer sidecar first.
- BLEU, embeddings, and 1-5 scores are not the eval.
- Block a bake on secret leaks or a voice fail rate over your limit.

## Acceptance checks

- [ ] Mac and device JSONL exist for the same prompts, both from the baked merged artifact.
- [ ] Open codes and the four axial buckets are written down.
- [ ] Secret leakage is a code check. Voice uses a binary judge with a confusion matrix.
- [ ] Held-out prompts rerun after the next bake.
- [ ] No runtime LoRA swap is claimed.

**Human lesson:** [Evaluate the writing style](07-evaluate-writing-style.html)