---
name: Reliable Local Agents (llama.cpp + Ollama)
description: >-
  Make local agent loops on llama.cpp and Ollama fast and reliable: prompt cache order, validated tool calls, a strict reply parser for thinking models, JSON vs tools, and several agents on one server.
---
# Reliable Local Agents (llama.cpp + Ollama)

Use this course when a local agent is slow on every turn, breaks after one bad tool call, or returns empty replies. Every script runs against a local llama-server on port 8080 or Ollama on port 11434 with a small Qwen3 model.

## End-to-end agent recipe

1. Use [lesson 01](01-keep-the-prompt-cache-warm.llms.md). Order the prompt stable-first, run `cache_check.py`, and compare `timings.cache_n` and `prompt_n` across the three cases.
2. Use [lesson 02](02-thinking-output-in-the-wrong-field.llms.md). Use `safe_parse.read_reply` on every reply and test thinking on and off with the curl request.
3. Use [lesson 03](03-tool-calls-that-do-not-break.llms.md). Write a tool loop that validates arguments, retries bad turns without saving them, and caps tokens and steps.
4. Use [lesson 04](04-json-mode-and-tools.llms.md). Split steps into tool steps and data steps, and keep the format-vs-tools test in CI.
5. Use [lesson 05](05-several-agents-on-one-box.llms.md). Read slot layout from the server log, run `parallel_bench.py` for 1, 2, and 4 agents, and size `-c` for the longest prompt per agent.
6. Keep these links exact: [The agent harness (and how to improve it)](../agent-harness-improve-loop/), [Test AI behavior before you ship](../on-device-eval-harness/), [Trace every step of an AI call](../on-device-llm-tracer/), [Stream tokens and measure speed](../on-device-streaming-metrics/).
7. Use the human HTML pages for the reader. Use the `.llms.md` briefs for agent steps and acceptance checks.

## Constraints

- Plain spoken English for a non-native reader. Literal sentences. No quizzes. No em dashes.
- Brand Edge FDE only. Do not name a course author.
- Use small local models (Qwen3 0.6B or 1.7B). Do not require a GPU.
- Only claim behavior that a linked issue or a run on your machine shows. Say which build you used.
- Environment setup comes before the first runnable example.

## Key concepts

- The prompt cache only reuses tokens that match from the start.
- A broken tool call in history can break every later turn.
- Answers can land in `reasoning_content` or vanish with a normal `finish_reason`.
- JSON mode and tools fight over the output shape.
- Slots split or share the context. A full context returns an error.

## Takeaways

- Put stable text first and changing text last, then check `cache_n`.
- Validate, retry, and keep bad turns out of history.
- Parse every reply with one strict parser.
- Send tools or a schema, not both.
- Measure total and per-agent speed on your own machine.

## Sources

- llama.cpp issues: #20225, #21912, #24055, #22948, #25923, #30056, #30321, #30246, #30256, #30276 at https://github.com/ggml-org/llama.cpp/issues
- Ollama issues: #17778, #18938, #18944 at https://github.com/ollama/ollama/issues
- mlx-lm issue #1946 at https://github.com/ml-explore/mlx-lm/issues/1946
- Ollama FAQ: https://github.com/ollama/ollama/blob/main/docs/faq.mdx