Ollama's context length trap

# 1. Install uv (skip if you have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Make a project folder with Python 3.11+
mkdir local-lab && cd local-lab
uv init --bare --python 3.11
uv venv

# 3. Add the course packages
uv add numpy gguf

# 4. Optional: PyTorch, only if you also want to run models directly in Python.
#    On Apple Silicon this wheel uses MPS. On other machines it uses the CPU.
uv add torch
uv run python -c "import torch; print('mps' if torch.backends.mps.is_available() else 'cpu')"

The default is small

The Ollama context length doc says the default depends on GPU memory: under 24 GiB of VRAM you get 4k tokens, 24 to 48 GiB gets 32k, and 48 GiB or more gets 256k. Most laptops are in the first group. Agents, coding tools, and RAG prompts are often longer than 4,096 tokens.

Check what is loaded with ollama ps. The CONTEXT column is the real number:

NAME          ID              SIZE      PROCESSOR          CONTEXT    RUNNER      UNTIL
q05:latest    be14bd33fca1    597 MB    60%/40% CPU/GPU    4096       llamacpp    4 minutes from now

On the test box the model was the Qwen2.5 0.5B GGUF imported as q05. With ollama pull qwen2.5:0.5b you get the same model under that name.

The OpenAI endpoint ignores num_ctx

Many apps talk to Ollama through /v1/chat/completions. The OpenAI API has no field for context size. The Ollama OpenAI compatibility doc says to use a Modelfile instead. Ollama #5356 asked for num_ctx there, and Ollama #16814 reports num_ctx being ignored on that endpoint in 0.30.7. On the test box (Ollama 0.40.4) a request with "num_ctx": 16384 still loaded 4,096.

What happens when the prompt is too long

On Ollama 0.40.4 with the llama.cpp runner, a 9,515-token prompt got an error:

HTTP 400 {"error":{"message":"request (9515 tokens) exceeds the available context size (4096 tokens), try increasing it"}}

That is the good case: you see it. Older versions and other runners have cut the start of the prompt and returned a normal reply. The start is where the system prompt is, so the model just behaves worse. Do not assume which one you have. Test it.

A CI check

This script counts the real prompt tokens once with native /api/chat and a large num_ctx. Then it sends the same messages the way your app does and compares usage.prompt_tokens. It fails on an error or on fewer tokens.

"""CI check: did Ollama read the whole prompt, or did it cut or reject it?"""
import json
import sys
import urllib.error
import urllib.request

HOST = "http://127.0.0.1:11434"
MODEL = sys.argv[1] if len(sys.argv) > 1 else "q05"


def post(path, body):
    req = urllib.request.Request(HOST + path, json.dumps(body).encode(), {"Content-Type": "application/json"})
    try:
        return 200, json.load(urllib.request.urlopen(req))
    except urllib.error.HTTPError as err:
        return err.code, {"error": err.read().decode()[:300]}


def main():
    long_doc = " ".join(f"Line {i}: the quick brown fox jumps over the lazy dog." for i in range(600))
    messages = [
        {"role": "system", "content": "Always answer in French."},
        {"role": "user", "content": long_doc + "\nWhat is 2+2?"},
    ]
    # 1. Ground truth: ask native /api/chat with a big context so nothing is cut.
    code, ref = post("/api/chat", {"model": MODEL, "messages": messages, "stream": False,
                                   "options": {"num_ctx": 32768, "num_predict": 1}})
    if code != 200:
        sys.exit(f"reference call failed: {code} {ref}")
    sent = ref["prompt_eval_count"]
    print("tokens in the full prompt:", sent)

    # 2. The path your app really uses. Here: the OpenAI-style endpoint.
    code, app = post("/v1/chat/completions", {"model": MODEL, "messages": messages, "max_tokens": 1})
    if code != 200:
        sys.exit(f"FAIL: app path returned HTTP {code}: {app['error']}")
    read = app["usage"]["prompt_tokens"]
    print("tokens the app path read:", read)
    if read < sent:
        sys.exit(f"FAIL: prompt was cut, {sent - read} tokens lost")
    print("PASS")


if __name__ == "__main__":
    main()

ollama pull qwen2.5:0.5b
uv run python ctx_guard.py qwen2.5:0.5b
# tokens in the full prompt: 9515
# FAIL: app path returned HTTP 400: ... exceeds the available context size (4096 tokens)

Lesson 02 fixes it. Run the same script after the fix and it prints PASS.