Several agents on one box

How llama-server shares context

On build b11624 the test box showed three cases in the startup log:

# default (-np auto): 4 slots, one shared pool
llama-server -hf Qwen/Qwen3-0.6B-GGUF:Q8_0
load_model: initializing, n_slots = 4, n_ctx_slot = 40960, kv_unified = 'true'

# -np 4 -c 8192: the context is split, 2048 tokens per slot
load_model: initializing, n_slots = 4, n_ctx_slot = 2048, kv_unified = 'false'

# -np 4 -c 8192 -kvu: one shared pool of 8192 for all slots
load_model: initializing, n_slots = 4, n_ctx_slot = 8192, kv_unified = 'true'

Split: each agent gets -c divided by -np. A long agent prompt fails even when other slots are empty. Shared: any agent can use the whole pool, but when the pool is full new requests are rejected. Nothing is cut to make room. The test box returned:

400 {"error":{"code":400,"message":"request (14298 tokens) exceeds the available context size (8192 tokens), try increasing it","type":"exceed_context_size_error"}}

Always read n_slots, n_ctx_slot, and kv_unified from your own log.

Measure total speed and speed per agent

"""Measure total and per-agent speed when several agents share one llama-server."""
import json
import sys
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor

URL = "http://127.0.0.1:8080/v1/chat/completions"


def one(i):
    body = {"messages": [{"role": "user", "content": f"Agent {i}: write two sentences about rivers."}],
            "max_tokens": 64, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}
    req = urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})
    start = time.perf_counter()
    r = json.load(urllib.request.urlopen(req))
    return r["usage"]["completion_tokens"], time.perf_counter() - start


for agents in [int(a) for a in sys.argv[1:]] or [1, 2, 4]:
    start = time.perf_counter()
    with ThreadPoolExecutor(agents) as pool:
        results = list(pool.map(one, range(agents)))
    wall = time.perf_counter() - start
    tokens = sum(t for t, _ in results)
    per_agent = sum(t / s for t, s in results) / agents
    print(f"agents={agents} total={tokens / wall:.1f} tok/s per-agent={per_agent:.1f} tok/s wall={wall:.1f}s")

llama-server -hf Qwen/Qwen3-1.7B-GGUF:Q8_0 --port 8080 -c 8192 -np 4
uv run python parallel_bench.py 1 2 4

agents=1 total=31.7 tok/s per-agent=31.7 tok/s wall=1.1s
agents=2 total=45.7 tok/s per-agent=22.9 tok/s wall=1.6s
agents=4 total=77.9 tok/s per-agent=20.7 tok/s wall=1.9s

On the test box, 4 agents got about 2.5 times the total speed of 1 agent, and each agent was about one third slower. Short prompts make this look good. A long prompt from one agent is read in the same batches as the others, so all agents wait while it is read.

Known traps

• Ollama #18938: Ollama forced OLLAMA_NUM_PARALLEL to 1 for hybrid families such as qwen35, qwen35moe, qwen3next, lfm2, and nemotron_h. With one slot there is one prompt cache, so each agent turn pushes out the others' cache. The issue is closed; check the startup warning on your version.
• llama.cpp #30276: on one 16 GB GPU, a 262,144 context with 4 or more slots made decode about 5 times slower, even for a single request. Smaller totals were fine.
• Ollama memory grows with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH, as the Ollama FAQ says.

Practical sizing

• Set -c to agents times the longest prompt you expect, plus room for the answer. Then decide split or shared.
• Send very long, one-off prompts (large files, logs) to a second, smaller server so they do not block the chat agents.
• Keep each agent's system prompt stable (lesson 01). Each slot has its own cache.
• If you need many users, steady high throughput, or several GPUs, move to a serving engine built for it, such as vLLM. llama-server is good for a few agents on one machine.

Sources

• llama.cpp server README
• Ollama FAQ