Keep the prompt cache warm

# 1. Install uv (skip if you have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Make a project folder with Python 3.11+
mkdir local-lab && cd local-lab
uv init --bare --python 3.11
uv venv

# 3. Add the course packages
uv add numpy

# 4. Optional: PyTorch, only if you also want to run models directly in Python.
#    On Apple Silicon this wheel uses MPS. On other machines it uses the CPU.
uv add torch
uv run python -c "import torch; print('mps' if torch.backends.mps.is_available() else 'cpu')"

What the cache does

llama-server keeps the tokens of the last prompt in each slot. When a new request starts with the same tokens, the server reuses them. It only reads the new part. This is called the prompt cache, or prefix cache. It is on by default (--cache-prompt).

The match is from the first token forward. If one token near the top changes, everything after it must be read again. A timestamp in the first line of the system prompt is enough to break it.

Start a server

llama-server -hf Qwen/Qwen3-1.7B-GGUF:Q8_0 --port 8080 -c 8192 -np 1

Every chat response from llama-server has a timings object. Two fields matter here: cache_n is how many prompt tokens came from the cache, and prompt_n is how many tokens the server had to read.

Measure it

This script sends the same question three ways: with a stable system prompt, with a timestamp at the top, and with the timestamp moved into the user turn.

"""Show the prompt cache working (or not) on llama-server, using the timings it returns."""
import datetime
import json
import time
import urllib.request

URL = "http://127.0.0.1:8080/v1/chat/completions"
RULES = "You are a support agent for a notes app. " + "Follow the house rules. " * 300


def ask(system, question):
    body = {
        "messages": [{"role": "system", "content": system}, {"role": "user", "content": question}],
        "max_tokens": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})
    start = time.perf_counter()
    r = json.load(urllib.request.urlopen(req))
    t = r["timings"]
    print(f"  wall={time.perf_counter() - start:5.2f}s  cache_n={t['cache_n']:5d}  "
          f"prompt_n={t['prompt_n']:5d}  prompt_ms={t['prompt_ms']:7.0f}")


print("A. stable system prompt, changing part last")
for i in range(3):
    ask(RULES, f"Question {i}: how do I export a note?")

print("B. timestamp at the top of the system prompt")
for i in range(3):
    stamp = datetime.datetime.now().isoformat()
    ask(f"Current time: {stamp}\n" + RULES, f"Question {i}: how do I export a note?")

print("C. same stable prompt, timestamp moved into the user turn")
for i in range(3):
    stamp = datetime.datetime.now().isoformat()
    ask(RULES, f"(Current time: {stamp}) Question {i}: how do I export a note?")

Output on the test box (8 CPU cores, Qwen3 1.7B Q8_0):

A. stable system prompt, changing part last
  wall=11.14s  cache_n=    0  prompt_n= 1539  prompt_ms=   9966
  wall= 1.52s  cache_n= 1521  prompt_n=   18  prompt_ms=    256
  wall= 1.55s  cache_n= 1521  prompt_n=   18  prompt_ms=    237
B. timestamp at the top of the system prompt
  wall= 9.83s  cache_n=    3  prompt_n= 1567  prompt_ms=   7027
  wall= 8.17s  cache_n=   24  prompt_n= 1546  prompt_ms=   7078
  wall=10.00s  cache_n=   24  prompt_n= 1546  prompt_ms=   8484
C. same stable prompt, timestamp moved into the user turn
  wall= 2.21s  cache_n= 1519  prompt_n=   51  prompt_ms=    582
  wall= 1.12s  cache_n= 1541  prompt_n=   29  prompt_ms=    290
  wall= 1.31s  cache_n= 1541  prompt_n=   29  prompt_ms=    297

Case B reads about 1,550 tokens every time. Cases A and C read only the new 18 to 51 tokens. The time to the first token drops from about 7 to 10 seconds to under 1 second. Same model, same question. Only the order changed.

Check the server log

The log tells you the same thing. Look for these lines:

slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.988 (> 0.100 thold), f_keep = 0.976
slot print_timing: id  0 | task 21 | prompt eval time =     255.88 ms /    18 tokens

A high f_keep and a small token count on prompt eval time mean the cache worked. A token count close to your full prompt on every turn means it did not.

Common things that break the cache

• A date, time, or random id in the system prompt. Move it to the end, or into the user turn.
• A file tree or "current state" block near the top that changes every turn. Put it after the history.
• A tool list that changes order or content between turns. Build it once and keep it the same.
• A chat template that removes old reasoning text from earlier turns. The history then changes, so the cache breaks after that point.

Hybrid models need a recent build

Models that mix attention with recurrent or sliding-window layers (Qwen 3.5 and 3.6, Gemma 4) do not use the cache in the same simple way. llama.cpp keeps "context checkpoints" for them. Several bugs caused full re-reads on every turn: llama.cpp #20225 (Qwen 3.5, closed), llama.cpp #21912 (Gemma 4 and Qwen 3.5 in coding agents, closed), and llama.cpp #24055 (checkpoints always invalidated with --checkpoint-min-step, still open). If your log shows a full re-read on every turn with a hybrid model, update llama.cpp first, then run the script above again.