EDGE FDE
IntermediateTest Local Decision Models (/v1/systemone)
Let a small local model make routine decisions only when it is sure, and send the rest to a person.
A labelled test set, an order-bias check, a quantization comparison, a confidence threshold with human fallback, and latency numbers.
Call /v1/systemone on llama-server or Ollama, measure accuracy, order flips, and confidence on your own data, and pick a quant and threshold with evidence.
Lessons
Key concepts
- Decision models return probabilities and confidence, not text.
- Probabilities are not calibrated for your data.
- Option order can change the answer on hard cases.
- Quantization can change confidence even when picks stay right.
- Multi-slot servers can fail on shared-prefix questions (llama.cpp #30482).
Takeaways
- Test on your own labelled set before you connect the model to anything.
- Shuffle options and treat flips as uncertain.
- Choose the quant by confidence and latency.
- Act only above a threshold you picked from data. Send the rest to a person.
Agent briefs
FAQ
What is a decision model?
A model that answers typed questions about a state with probabilities instead of text. Question types are choice, score, and noul (yes or no).
How do I call a decision model locally?
POST a state and questions to /v1/systemone on llama-server, or on Ollama 0.35.0 or newer with a model field. Check output_modalities for decisions first.
Are decision model probabilities calibrated?
Not for your data. The llama-server README says they are scaled with temperatures in the model file and are not guaranteed to be calibrated. Test on your own labelled set.
Why shuffle the option order?
On hard cases the pick can change with the order. A flip means the model is unsure, so route that item to a person.
How do I pick a confidence threshold?
Run your labelled set at several thresholds and pick the lowest one where the automatic answers are right as often as you need. Send everything below it to a human.
Related courses
Test AI behavior before you ship · The agent harness (and how to improve it) · Reliable Local Agents (llama.cpp + Ollama) · Local LLM Runtime Traps (Ollama + llama.cpp) · On-Device Search with EmbeddingGemma 2