Thresholds, human fallback, latency

Pick the threshold

for t in 0.5 0.7 0.9; do
  uv run python eval_decision.py --url http://127.0.0.1:8080 --threshold $t --orders 1 | grep auto-handled
done

Kev-0.8B Q8_0 on the test box:

(This run was on a server with -np 2. Compared with the -np 1 run in lesson 02, one confidence moved just above 0.5. Small numeric changes like this are normal, which is one more reason not to set the threshold right at the edge.)

Here 0.7 is a good choice: no wrong automatic actions and only 2 of 16 tickets for a person. With 16 tickets this is a sketch, not proof. Use a few hundred real examples, and pick the lowest threshold where the automatic answers are right as often as your product needs.

THRESHOLD = 0.7

def route(answer: dict) -> str:
    if answer["confidence"] >= THRESHOLD:
        return answer["choice"]
    return 'human'

Measure latency on your hardware

eval_decision.py prints p50 and p95 for every call. On the test box Kev-0.8B Q8_0 was about 207 ms p50 and 283 ms p95 on CPU. Measure on the machine that will run the pipeline, with the batch sizes and slots you will use.

Watch the slot count

llama.cpp #30482: when one request has two questions whose prompts share a prefix and the server has more than one slot, /v1/systemone returns HTTP 500. The test box reproduced it on build b11624 with -np 2:

{"error":{"code":500,"message":"failed to get embeddings, the question and its options must fit in one batch","type":"server_error"}}

With -np 1 the same request works. Until it is fixed, run decision models with -np 1, or send questions that share instructions and options in separate requests. The README also says that for some models (laya, clef, vllm-d3, lfm2-d1-omni) the whole prompt must fit in --ubatch-size.

Before you ship

• Labelled set, accuracy, and order flips (lesson 02).
• Quant chosen by confidence, not only accuracy (lesson 03).
• Threshold from data, with a human queue for the rest.
• p50 and p95 latency on the target machine.
• Log every answer with its confidence, so you can grow the test set from real mistakes.