EDGE FDE
IntermediateReliable Local Agents (llama.cpp + Ollama)
Make local agents fast and stop one bad reply from breaking the whole chat.
Prompt cache order, safe tool calls, a reply parser for thinking models, JSON vs tools, and sizing one server for several agents.
Measure prompt cache hits, validate tool calls before they run, catch answers in the wrong field, and size llama-server or Ollama for several agents.
Lessons
Key concepts
- The prompt cache only reuses tokens that match from the start.
- A broken tool call in history can break every later turn.
- Answers can land in
reasoning_contentor vanish with a normalfinish_reason. - JSON mode and tools fight over the output shape.
- Slots split or share the context. A full context returns an error.
Takeaways
- Put stable text first and changing text last, then check
cache_n. - Validate, retry, and keep bad turns out of history.
- Parse every reply with one strict parser.
- Send tools or a schema, not both.
- Measure total and per-agent speed on your own machine.
Agent briefs
FAQ
Why is my local agent slow on every turn?
Usually the model re-reads the whole prompt because something near the top changes, like a timestamp or tool list. Put stable text first and check timings.cache_n in the llama-server response.
How do I stop one bad tool call from breaking the chat?
Parse and validate the tool call before you run it. If it is broken, log it, do not add it to the history, and retry.
Why does my reasoning model return an empty reply?
Some builds put the answer or tool call in reasoning_content, or drop tool_calls while finish_reason says tool_calls. Use one strict parser that raises on these cases.
Can I use JSON mode and tools together?
Not reliably. Ollama issue 18944 shows tool calls turning into text with format json. Send tools or a schema in one request, not both.
How many agents can share one llama-server?
Measure it. More slots raise total speed but lower per-agent speed, and the context is split or shared. Move to vLLM for many users or several GPUs.
Related courses
The agent harness (and how to improve it) · Test AI behavior before you ship · Trace every step of an AI call · Stream tokens and measure speed · Local LLM Runtime Traps (Ollama + llama.cpp) · Test Local Decision Models (/v1/systemone)