---
title: "Measure speculative decoding first"
lesson_id: "03"
---

# Measure speculative decoding first

- **Lesson ID:** 03
- **Goal:** Decide with numbers whether speculative decoding helps on this machine.
- **Human lesson:** [03-measure-speculative-decoding.html](03-measure-speculative-decoding.html)

## Prerequisites

- llama.cpp installed.
- A target model and a draft model with the same tokenizer.

## Inputs, outputs, and artifacts

- **Inputs:** Three or more prompts like your real traffic.
- **Outputs:** A table of generation speed, prompt speed, and acceptance, off vs on.
- **Artifacts:** `spec_ab.py` output and the server log line.

## Agent build steps

1. Start the off server on port 8092 and the on server on port 8093 as shown.
2. Run `uv run python spec_ab.py 8092 8093`.
3. Grep the on server log for `draft acceptance`.
4. Compare generation and prompt speed per prompt.
5. On Macs, use a llama.cpp build that includes PR #29869. On CUDA, check acceptance against issue #26750.
6. Keep it on only where it is faster.

## Constraints

Keep the language simple, friendly, and literal. No quizzes, no em dashes. Brand Edge FDE only. Use small local models. Do not invent flags, endpoints, or issue numbers. 

## Key concepts

- A draft guesses tokens and the big model checks them.
- High acceptance does not always mean faster. The draft costs time too.
- Acceptance can differ by backend (llama.cpp #26750).
- Mac Metal needed PR #29869 to make small batches fast.

## Takeaways

- Run the same prompts with it off and on, on your own hardware.
- Confirm the `draft acceptance` log line is there.
- Keep it only where it wins, and watch prompt processing speed too.

## Acceptance checks

- The table has off and on rows for the same prompts.
- The acceptance log line is quoted.
- PR #29869 and issue #26750 are linked.
- [ ] Proceed to [lesson 04 brief](04-new-models-and-upgrades.llms.md).