Compare quantizations | Decision Models

Environment setup

Use the same local-lab folder from lesson 01 (uv, Python 3.11+, and uv add numpy). If you skipped it, do the setup in lesson 01 first.

Install llama.cpp. On macOS: brew install llama.cpp. On Linux or Windows, download a release from github.com/ggml-org/llama.cpp/releases. Check it with llama-server --version.

Run two servers

llama-server -hf ggml-org/Kev-0.8B-GGUF:Q8_0 --port 8095 -np 1 -c 4096
llama-server -hf ggml-org/Kev-0.8B-GGUF:BF16 --port 8096 -np 1 -c 4096

uv run python eval_decision.py --url http://127.0.0.1:8095 > q8.txt
uv run python eval_decision.py --url http://127.0.0.1:8096 > bf16.txt

On the test box (CPU only):

Quant	Accuracy	Flips	Auto at 0.5	p50 ms	p95 ms
Q8_0	15/16	1/16	14/16, all right	207	283
BF16	15/16	1/16	14/16, all right	320	381

Per-ticket confidence moved by about 0.01 (t13 was 0.46 on Q8_0 and 0.47 on BF16). For this model and this set, Q8_0 is the better choice: same answers, faster. That result is for Kev-0.8B only.

It is not always that safe

llama.cpp #30064: Clef Q8_0 gave probabilities close to uniform (France 0.54 for "Paris is the capital of France"), while BF16 gave 0.99. The pick was still right, so only a confidence check would show the problem.
Ollama #18865: in Ollama, Clef models set the batch size to the context size, so the default num_ctx 16384 needed a 3.3 to 3.8 GB compute buffer and failed to load. The reported workaround is a derived model with PARAMETER num_ctx 4096.
Ollama #18769: clef-flash failed every /v1/systemone call with "non-finite logit" on CUDA in Ollama 0.35.1. #18865 says this was fixed in 0.40.1.

Rule: compare confidence, not only accuracy. A quant can keep the right pick and still lose the confidence you use for routing.

Key concepts

Quantization can change probabilities even when the pick stays right.
Run the same labelled set on each quant.
Compare accuracy, flips, confidence, and latency.
Some decision model quants and runtimes have known bugs.

Takeaways

Pick the smallest quant whose confidences match the bigger one on your set.
Check confidence spread, not only accuracy.
Set num_ctx small for Clef in Ollama if it fails to load.