---
title: "A labelled set and option order bias"
lesson_id: "02"
---

# A labelled set and option order bias

- **Lesson ID:** 02
- **Goal:** Measure accuracy and order bias on your own labelled data.
- **Human lesson:** [02-labelled-set-and-order-bias.html](02-labelled-set-and-order-bias.html)

## Prerequisites

- Lesson 01 server running on port 8080.

## Inputs, outputs, and artifacts

- **Inputs:** `tickets.jsonl` with text and label.
- **Outputs:** Per-ticket picks, accuracy, flip count, latency.
- **Artifacts:** `tickets.jsonl`, `eval_decision.py`, saved output.

## Agent build steps

1. Save `tickets.jsonl` and replace it with your own labelled examples, including hard ones.
2. Save `eval_decision.py`.
3. Run `uv run python eval_decision.py --url http://127.0.0.1:8080`.
4. List the flipped and low-confidence tickets.
5. Rewrite overlapping option descriptions and run again.

## Constraints

Keep the language simple, friendly, and literal. No quizzes, no em dashes. Brand Edge FDE only. Use small local models. Do not invent flags, endpoints, or issue numbers. 

## Key concepts

- A labelled set tells you accuracy on your own data.
- Shuffling options shows order bias.
- Wrong answers and flips often have low confidence.
- Hard cases sit between two labels.

## Takeaways

- Keep a labelled JSONL set with hard cases.
- Run every ticket in at least two option orders.
- Treat flips as uncertain and route them to a human.

## Acceptance checks

- Each ticket is asked in at least 2 orders.
- Accuracy and flips are reported.
- [ ] Proceed to [lesson 03 brief](03-compare-quantizations.llms.md).