Set the context and size the memory

Three ways to set it

• Server-wide: OLLAMA_CONTEXT_LENGTH=16384 ollama serve. Works for every endpoint.
• Per model: a Modelfile with PARAMETER num_ctx. Works for every endpoint, because the app asks for the new model name.
• Per request: "options": {"num_ctx": 16384} on native /api/chat. Does not work on /v1/chat/completions.

# Per model
cat > Modelfile <<'MF'
FROM qwen2.5:0.5b
PARAMETER num_ctx 16384
MF
ollama create qwen2.5-16k -f Modelfile
uv run python ctx_guard.py qwen2.5-16k
# tokens in the full prompt: 9515
# tokens the app path read: 9515
# PASS

# Server-wide
OLLAMA_CONTEXT_LENGTH=16384 ollama serve
uv run python ctx_guard.py qwen2.5:0.5b
# PASS

Both fixes passed on the test box (Ollama 0.40.4). ollama ps then showed 16384 in the CONTEXT column.

Size the memory

A bigger context needs more memory for the KV cache. For a normal attention model the cache is about:

KV bytes = 2 (K and V) x layers x kv_heads x head_dim x context x 2 bytes (f16)

This script reads those numbers from the GGUF file:

"""Print the architecture stored in a GGUF file and estimate KV cache memory."""
import sys

from gguf import GGUFReader


def field(reader, key):
    f = reader.fields.get(key)
    if f is None:
        return None
    value = f.parts[f.data[0]]
    if f.types[0].name == "STRING":
        return bytes(value).decode()
    return value.tolist()[0]


def main():
    path = sys.argv[1]
    ctx = int(sys.argv[2]) if len(sys.argv) > 2 else 8192
    reader = GGUFReader(path)
    arch = field(reader, "general.architecture")
    layers = field(reader, f"{arch}.block_count")
    heads = field(reader, f"{arch}.attention.head_count")
    kv_heads = field(reader, f"{arch}.attention.head_count_kv") or heads
    head_dim = field(reader, f"{arch}.attention.key_length")
    if head_dim is None:
        head_dim = field(reader, f"{arch}.embedding_length") // heads
    print("architecture:", arch)
    print("name:", field(reader, "general.name"))
    print("trained context_length:", field(reader, f"{arch}.context_length"))
    print(f"layers={layers} kv_heads={kv_heads} head_dim={head_dim}")
    # K and V, every layer, f16 = 2 bytes. Hybrid and sliding-window models use less.
    kv_bytes = 2 * layers * kv_heads * head_dim * ctx * 2
    print(f"KV cache at {ctx} tokens (f16, full attention): about {kv_bytes / 2**20:.0f} MiB")


if __name__ == "__main__":
    main()

$ uv run python gguf_arch.py Qwen3-1.7B-Q8_0.gguf 8192
architecture: qwen3
name: Qwen3 1.7B Instruct
trained context_length: 40960
layers=28 kv_heads=8 head_dim=128
KV cache at 8192 tokens (f16, full attention): about 896 MiB

This is an estimate. Hybrid models (recurrent or sliding-window layers) use less, a quantized KV cache uses less, and the runtime adds compute buffers. In Ollama the memory grows with OLLAMA_NUM_PARALLEL times the context, as the Ollama FAQ says. Use the number to choose a context that leaves room, then confirm with ollama ps that the model is still 100% GPU and not split to CPU.