Measure speculative decoding first

How it works, in short

A small draft model (or a draft head, as in MTP) guesses the next few tokens. The big model checks them in one step. If most guesses are right, you get several tokens for the cost of one check. If many are wrong, you pay for the guesses and the checks.

Run an A/B test

Start the same model twice, once without and once with a draft model. llama.cpp uses --spec-type and -md. --spec-draft-n-max sets how many tokens to guess (default 3).

# Off
llama-server -hf Qwen/Qwen3-1.7B-GGUF:Q8_0 --port 8092 -c 4096 -np 1
# On: Qwen3 0.6B drafts for Qwen3 1.7B (same tokenizer)
llama-server -hf Qwen/Qwen3-1.7B-GGUF:Q8_0 -hfd Qwen/Qwen3-0.6B-GGUF:Q8_0 \
  --spec-type draft-simple --spec-draft-n-max 3 --port 8093 -c 4096 -np 1

"""A/B test: the same prompts against a server without and with speculative decoding."""
import json
import sys
import urllib.request

PROMPTS = [
    "Write a Python function that checks if a number is prime. Code only.",
    "Tell me a short story about a cat who learns to fly.",
    "List the numbers from 1 to 40 separated by commas.",
]


def run(port):
    url = f"http://127.0.0.1:{port}/v1/chat/completions"
    for prompt in PROMPTS:
        body = {"messages": [{"role": "user", "content": prompt}], "max_tokens": 200,
                "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}
        req = urllib.request.Request(url, json.dumps(body).encode(), {"Content-Type": "application/json"})
        t = json.load(urllib.request.urlopen(req))["timings"]
        drafted, accepted = t.get("draft_n"), t.get("draft_n_accepted")
        rate = f"{accepted / drafted:.0%}" if drafted else "off"
        print(f"{prompt[:28]:28} | gen {t['predicted_per_second']:5.1f} tok/s | "
              f"prompt {t['prompt_per_second']:6.1f} tok/s | accept {rate}")


if __name__ == "__main__":
    for port in sys.argv[1:] or ["8092", "8093"]:
        print(f"--- port {port}")
        run(port)

Results on the test box (8 CPU cores, build b11624):

--- port 8092
Write a Python function that | gen  16.3 tok/s | prompt    7.0 tok/s | accept off
Tell me a short story about  | gen  19.0 tok/s | prompt  130.0 tok/s | accept off
List the numbers from 1 to 4 | gen  16.8 tok/s | prompt  157.1 tok/s | accept off
--- port 8093
Write a Python function that | gen  14.9 tok/s | prompt   81.4 tok/s | accept 88%
Tell me a short story about  | gen  16.6 tok/s | prompt  166.1 tok/s | accept 51%
List the numbers from 1 to 4 | gen  20.5 tok/s | prompt  125.3 tok/s | accept 98%

Only the counting prompt got faster. The code prompt was slower even with 88% of guesses accepted, because on this CPU the draft model's own time ate the gain. The story prompt had 51% acceptance and was slower. Your GPU will give different numbers. That is why you run the test. (The first prompt speed on each server includes warm-up, so run each prompt twice and use the second number.)

The server log also prints the rate, so you can confirm speculative decoding is really on:

slot print_timing: id  0 | task 0 | draft acceptance = 0.87805 (  144 accepted /   164 generated), mean len =  3.62

If you never see this line, speculative decoding is not running, whatever your flags say.

Known traps

• llama.cpp PR #29869 (merged October 2026): on Apple GPUs without the tensor API (M1 to M4), Metal ran the small batches of speculative decoding with slow kernels, so DFlash on an M3 Ultra was slower than normal decoding. Use a build with this PR on a Mac.
• llama.cpp #26750: with the same GGUF and build, MTP acceptance was 35.8% on CUDA and about 92% on Vulkan. On CUDA it turned a speedup into a 32% slowdown. Acceptance can depend on the backend.
• Prompt processing speed: also compare the prompt column. Big agent prompts spend most time there, and the draft model adds work.
• Many requests at once: a busy server already fills its batches, so extra guesses help less. Test with your real concurrency.

Rules

• Start with --spec-draft-n-max 3.
• Keep it on only for prompt types where generation is faster and prompt processing is not much slower.
• Log the acceptance rate in production. A drop means a model, prompt, or backend change.