Evaluate the writing style

Why this matters on the device

A baked writing model can still sound like the original base model. It can leak a secret from your writing. It can refuse a simple rewrite. Or it can break the format the app needs. Those problems can hide behind a nice demo prompt. Treat checking as part of what you deliver: traces from the Mac merge and from the device, a short list of problem types you can explain, and a check that can run again after the next bake.

This lesson does not replace merge-then-export. You still merge the LoRA into the base, export the ordinary graph, and keep the tokenizer sidecar. Evaluation judges that baked artifact. It does not invent a runtime adapter slot.

The steps, one box at a time

Environment setup

Stay in the same Mac uv project you used to train and merge. MPS or CPU only. Do not add CUDA.

cd writing-style-lora
uv run python -c "import torch; print(torch.__version__); print('MPS:', torch.backends.mps.is_available())"
mkdir -p evals/traces evals/open-codes evals/held-out
uv run python collect_traces.py --source mac --out evals/traces/mac.jsonl
uv run python collect_traces.py --source device --out evals/traces/device.jsonl

Collect traces from Mac and device

A trace is one prompt, one reply, and the extra fields that let you replay it. Collect the same review prompts on the merged Mac model and on the baked on-device chat from lesson 06. If those two disagree, the bake or the tokenizer sidecar is the first suspect, not the judge.

{
  "id": "rewrite-email-03",
  "source": "device",
  "platform": "iOS",
  "prompt": "Rewrite this note so it sounds like me. Keep the ask.",
  "reply": "Hey, can we move Tuesday? I am slammed after lunch.",
  "base_revision": "qwen3-0.6b",
  "artifact": "writing-style.aimodel",
  "tokenizer": "tokenizer",
  "runtime_lora_swap": false,
  "latency_ms": 840
}

Open coding

Open coding means writing short notes on each reply, in your own words. Read twenty to fifty traces before you name a score. For each reply, write what you see. Do not force the note into a bucket yet. A good open code is specific: what drifted, what leaked, what the app could not parse.

{
  "id": "rewrite-email-03",
  "open_codes": [
    "voice is closer to a generic assistant than to my short notes",
    "ends with a corporate sign-off I never use",
    "no secret leaked"
  ]
}

Axial buckets

Axial coding means grouping those notes into a few problem types you can count. For this course, start with four:

• Voice drift. The reply is fluent but not yours.
• Secret leakage. A name, key, address, or private fact from your writing appears.
• Refusal. The model declines a rewrite it should attempt.
• Formatting. Extra markdown, a fake citation, or a structure the chat UI cannot show cleanly.

Pick one person who knows the product well to decide what good and bad look like. If two reviewers disagree, the bucket is still too vague. Do not average the disagreement away.

Automate: code checks, then a binary judge

First, automate the cheap, exact problems. Secrets and format belong to code. Voice is harder. Only after the axial buckets are stable should you add a binary LLM judge (a yes-or-no check). Use it only for the problem that code cannot see.

import json
import re
from pathlib import Path

SECRET = re.compile(
    r"(sk-[A-Za-z0-9]{16,}|AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]+PRIVATE KEY-----)",
    re.I,
)
FORMAT_NOISE = re.compile(r"(?m)^(#{1,6}\s|```|Sources?:)", re.I)

def code_checks(reply: str) -> dict[str, bool]:
    return {
        "secret_leakage": bool(SECRET.search(reply)),
        "formatting": bool(FORMAT_NOISE.search(reply)),
    }

for path in Path("evals/traces").glob("*.jsonl"):
    for line in path.read_text().splitlines():
        row = json.loads(line)
        print(row["id"], row["source"], code_checks(row["reply"]))

A judge rubric should ask one yes-or-no question. "Sounds like me" is one question. "Sounds like me and is safe and is formatted" is three questions packed into one score.

RUBRIC = """
You are checking one writing-style reply against the author's voice.
Answer PASS or FAIL only.

PASS if the reply could pass as this author's short note: direct, lightly informal, no corporate closer.
FAIL if the reply sounds like a generic assistant, invents a sign-off, or ignores the rewrite ask.

Author samples:
{voice_samples}

Prompt: {prompt}
Reply: {reply}
"""

Align with a confusion matrix, not agreement percent

Set aside a held-out slice (prompts you did not use to tune the judge). Label them yourself. Run the judge. Build a two-by-two table, called a confusion matrix: human pass or fail versus judge pass or fail. A 90% agreement number can hide a judge that never fails anyone. You care about false passes on voice drift, and false fails on good short notes.

If the judge and the human disagree in a patterned way, fix the rubric or the bucket. Do not add a second judge to cover up the first.

Held-out set and CI

Keep a slice of prompts the trainer never saw. After every merge-and-export, run code checks on Mac traces and device traces. Then run the aligned binary judge on the voice bucket. Block the bake if any secret leaks. Also block it if the voice fail rate is worse than the limit you chose in advance. After the app is live, sample real device traces into the same JSONL shape. Open-code a batch each week. Do not let the live stream change the buckets by itself.

Misconceptions

• A BLEU or embedding score is the eval. Those numbers can move while the voice is still wrong.
• A 1-5 judge is more scientific. Reviewers will not share a meaning for 3. Binary is easier to align.
• Mac traces are enough. The product is the device chat. Collect both.
• You can skip merge-then-export if the judge likes the adapter. Core AI still has no documented hot-swap LoRA. Judge the baked model.

When this goes wrong

• Writing tests before reading traces. You will measure the wrong thing with confidence.
• Too many axial buckets. If every trace is its own category, you cannot automate.
• Judging secrets with an LLM. Use the regex (or a real secret scanner) first.
• Shipping a judge that was never compared to a human table. Agreement percent will flatter it.
• Treating a failed device trace as a training bug. Check tokenizer sidecar and platform preset first.

Done when

• Mac and device JSONL exist for the same review prompts, both from the baked merged artifact.
• Open codes and four axial buckets are written down, with secret leakage scanned by code.
• A binary voice judge has a confusion matrix against a human held-out slice.
• The next bake can rerun the checks in CI. Lesson 08 still confirms the full delivery.

Keep learning

An eval is still an evolved unit test: you encode what "right" looks like in software. For Foundation Models agents, tool paths, and CI gates, that harness is Test AI behavior before you ship.

The same look-then-score path on images is Evaluate generations. On labels it is Evaluate the clothing classifier.

PyTorch to Core AI in Xcode · From the metal to the model · Core AI models, typed · Model architectures in plain English · Diffusion LoRA to Core AI on device