---
name: Local LLM Runtime Traps (Ollama + llama.cpp)
description: >-
  Catch quiet local runtime problems in Ollama and llama.cpp: context truncation, context and KV sizing, speculative decoding that does not help, unknown architectures, and upgrade regressions.
---
# Local LLM Runtime Traps (Ollama + llama.cpp)

Use this course when a local model gets worse with no error, or breaks after an update. The scripts run against Ollama on port 11434 and llama-server on ports 8092 and 8093 with small Qwen models.

## End-to-end agent recipe

1. Use [lesson 01](01-ollama-context-length.llms.md). Run `ctx_guard.py` against the model your app uses and record whether the app path errors, cuts, or passes.
2. Use [lesson 02](02-set-context-and-size-memory.llms.md). Set the context with a Modelfile or `OLLAMA_CONTEXT_LENGTH`, estimate KV memory with `gguf_arch.py`, and re-run `ctx_guard.py` until it passes.
3. Use [lesson 03](03-measure-speculative-decoding.llms.md). Run `spec_ab.py` against an off server and an on server, confirm the `draft acceptance` log line, and keep speculative decoding only for prompts where it wins.
4. Use [lesson 04](04-new-models-and-upgrades.llms.md). Read the GGUF architecture, test new models on upstream llama-server, pin runtime and model versions, and run `smoke_test.py` plus `ctx_guard.py` before every upgrade.
5. Keep these links exact: [Reliable Local Agents (llama.cpp + Ollama)](../local-agent-reliability/), [Stream tokens and measure speed](../on-device-streaming-metrics/), [Test AI behavior before you ship](../on-device-eval-harness/), [The agent harness (and how to improve it)](../agent-harness-improve-loop/).
6. Use the human HTML pages for the reader. Use the `.llms.md` briefs for agent steps and acceptance checks.

## Constraints

- Plain spoken English for a non-native reader. Literal sentences. No quizzes. No em dashes.
- Brand Edge FDE only. Do not name a course author.
- Use small local models. Do not require a GPU.
- Say which runtime version a result came from. Behavior changes between versions.
- Environment setup comes before the first runnable example.

## Key concepts

- Ollama's default context is 4k under 24 GiB VRAM, and the OpenAI endpoint ignores `num_ctx`.
- A too-long prompt is cut or rejected, depending on version.
- KV memory grows with context and parallel slots.
- Speculative decoding can be slower. Acceptance depends on prompts and backend.
- Runtimes must know a model's architecture, and upgrades can regress.

## Takeaways

- Compare tokens sent with tokens read in CI.
- Set context with a Modelfile or `OLLAMA_CONTEXT_LENGTH`.
- A/B speculative decoding on your hardware.
- Pin runtime and model versions and smoke test every upgrade.

## Sources

- Ollama context length doc: https://github.com/ollama/ollama/blob/main/docs/context-length.mdx
- Ollama FAQ: https://github.com/ollama/ollama/blob/main/docs/faq.mdx
- Ollama issues: #5356, #16814, #18698, #18856, #18898, #18909, #18968 at https://github.com/ollama/ollama/issues
- llama.cpp PR #29869 and issue #26750 at https://github.com/ggml-org/llama.cpp