← All courses ← Course home

Lesson 01 of 04

What a decision model returns

A decision model does not write text. You give it a state and typed questions, and it returns a choice, a score, or a yes probability, with probabilities for every option.

Agent brief (llms.md)

Environment setup

Do this once before the first runnable example. You need Python 3.11 or newer and uv.

# 1. Install uv (skip if you have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Make a project folder with Python 3.11+
mkdir local-lab && cd local-lab
uv init --bare --python 3.11
uv venv

# 3. Add the course packages
uv add numpy

# 4. Optional: PyTorch, only if you also want to run models directly in Python.
#    On Apple Silicon this wheel uses MPS. On other machines it uses the CPU.
uv add torch
uv run python -c "import torch; print('mps' if torch.backends.mps.is_available() else 'cpu')"

Install llama.cpp. On macOS: brew install llama.cpp. On Linux or Windows, download a release from github.com/ggml-org/llama.cpp/releases. Check it with llama-server --version.

Install Ollama from ollama.com/download. Check it with ollama --version.

Run every script with uv run python script.py. The examples were checked on llama.cpp build b11624 and Ollama 0.40.4 on a Linux CPU box. Your numbers will be different on your machine. That is the point: measure on your own hardware.

Where the API comes from

llama-server has POST /v1/systemone, described in the llama-server README as a TypeSafe-compatible System One API. Ollama added the same endpoint in 0.35.0, and 0.40.4 updated its llama.cpp with native decision model support and added ollama run nimble --ask in the terminal.

Start a small decision model

llama-server -hf ggml-org/Kev-0.8B-GGUF:Q8_0 --port 8080 -np 1 -c 4096

Kev-0.8B is small enough for a laptop CPU. A decision model shows decisions in its model info, so your app can check before it calls the endpoint:

curl -s localhost:8080/v1/models | jq '.data[0].architecture'
# {"input_modalities": ["text"], "output_modalities": ["decisions"]}

Three question types

curl -s localhost:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
  "questions": {
    "route": {"type": "choice", "instructions": "Which team should handle this?",
              "criteria": {"billing": null, "shipping": null, "technical": null}},
    "angry": {"type": "noul", "instructions": "Is the customer angry?"},
    "urgency": {"type": "score", "instructions": "How urgent is this?",
                "criteria": ["can wait", "this week", "today", "right now"]}
  }
}' | jq

Kev-0.8B Q8_0 on the test box (llama.cpp b11624), shortened:

{
  "answers": {
    "route": {"type": "choice", "choice": "billing",
              "probabilities": {"billing": 0.793, "shipping": 0.123, "technical": 0.083},
              "confidence": 0.690},
    "angry": {"type": "noul", "noul": 0.652},
    "urgency": {"type": "score", "score": 1.174,
                "probabilities": {"0": 0.371, "1": 0.282, "2": 0.148, "3": 0.199},
                "confidence": 0.0}
  },
  "usage": {"input_tokens": 106, "output_tokens": 0}
}

Look at urgency: the score is 1.17 but the confidence is 0.0. The probabilities are spread out. A pipeline that only reads score would act on a guess. Always read confidence too.

The README also says: the probabilities are scaled with temperatures stored in the model file, and are not guaranteed to be calibrated for your data. That is why the next lessons test the model on your own labelled examples.

With Ollama the request is the same, plus a "model" field, sent to http://localhost:11434/v1/systemone.

Key concepts

  • /v1/systemone takes a state and named questions.
  • Types are choice, score, and noul.
  • Answers include probabilities and a 0 to 1 confidence.
  • Probabilities are not calibrated for your data.

Takeaways

  • Check output_modalities for decisions before calling.
  • Read confidence with every answer.
  • Plan to test on your own labelled data.