EDGE FDE
IntermediateLocal LLM Runtime Traps (Ollama + llama.cpp)
Stop quiet runtime problems from making your local model worse without any error.
A CI context check, context and KV memory sizing, a speculative decoding A/B test, and a pin-and-smoke-test upgrade path.
Catch cut or rejected prompts in Ollama, set the context where your endpoint sees it, prove speculative decoding helps before you turn it on, and upgrade runtimes safely.
Lessons
Key concepts
- Ollama's default context is 4k under 24 GiB VRAM, and the OpenAI endpoint ignores
num_ctx. - A too-long prompt is cut or rejected, depending on version.
- KV memory grows with context and parallel slots.
- Speculative decoding can be slower. Acceptance depends on prompts and backend.
- Runtimes must know a model's architecture, and upgrades can regress.
Takeaways
- Compare tokens sent with tokens read in CI.
- Set context with a Modelfile or
OLLAMA_CONTEXT_LENGTH. - A/B speculative decoding on your hardware.
- Pin runtime and model versions and smoke test every upgrade.
Agent briefs
FAQ
What is Ollama's default context length?
It depends on VRAM. Under 24 GiB you get 4k tokens, 24 to 48 GiB gets 32k, and 48 GiB or more gets 256k. Check the CONTEXT column in ollama ps.
Why does num_ctx not work on Ollama's OpenAI endpoint?
The OpenAI API has no context size field, so /v1/chat/completions ignores it. Use a Modelfile with PARAMETER num_ctx or set OLLAMA_CONTEXT_LENGTH.
How do I know if my prompt was cut?
Compare the tokens you sent with prompt_eval_count or usage.prompt_tokens. Fail CI if the model read fewer tokens or returned an error.
Does speculative decoding always make a local model faster?
No. Run the same prompts with it off and on, read the draft acceptance rate, and keep it only where generation is faster on your hardware.
Why does a new model say unknown model architecture?
Your runtime does not have code for the architecture in the GGUF file yet. Read general.architecture, try upstream llama-server, and pin versions.
Related courses
Reliable Local Agents (llama.cpp + Ollama) · Stream tokens and measure speed · Test AI behavior before you ship · The agent harness (and how to improve it) · Test Local Decision Models (/v1/systemone) · On-Device Search with EmbeddingGemma 2