# Run One Large LLM Across Several Macs with MLX: Lesson 05. Long prompts and serving many agents

Lesson id: `mlx-distributed-mac-cluster/05-long-prompts-and-many-agents`
Human lesson: https://edgefde.dev/courses/mlx-distributed-mac-cluster/05-long-prompts-and-many-agents.html

## Goal

Serve the cluster to several agents with mlx_lm.server and measure first-token time under load.

## Prerequisites

- Lessons 01 to 04 done
- A passing link test

## Inputs

- Model id
- Hostfile
- Rank 0 host name

## Outputs and artifacts

- Prefill step size results
- Running server on rank 0
- `agents_ttft.py` results for 1, 2, 4, 8 agents

## Steps

1. Run `mlx_lm benchmark` through `mlx.launch` with `-p 65536 -g 128 -b 1` for `--prefill-step-size` 2048, 4096, 8192 and record first-token time.
2. Start the server: `uv run mlx.launch --hostfile jaccl.json --backend jaccl -- ~/mlx-cluster/.venv/bin/python -m mlx_lm server --model $MODEL --host 0.0.0.0 --port 8080 --prefill-step-size <best>`.
3. From an agent machine run `uv run python agents_ttft.py --url http://<rank0>:8080/v1/chat/completions --agents N` for N in 1, 2, 4, 8.
4. Add a prompt length cap and agent timeouts based on the measured worst case.
5. Point agent tools at rank 0 with an OpenAI-compatible provider.

## Constraints

- Do not expose the server to the internet.
- Do not use adapters or draft models in distributed mode.
- Do not publish speed numbers you did not measure.

## Acceptance checks

- Server answers a chat request from another machine.
- Worst-case first-token time is recorded for each agent count.

## Key concepts

- First-token time grows with prompt length. Apple measured about 40 s at 64K tokens on four M5 Ultras.
- `--prefill-step-size` sets the chunk size for long prompts. The server default is 2048.
- Only rank 0 serves HTTP, and it binds to `127.0.0.1` unless you set `--host`.
- The mlx-lm server is not meant for production security. Keep it on a private network.

## Takeaways

- Stream answers and set agent timeouts for long prompts.
- Benchmark prefill step sizes on your own cluster before you pick one.
- Measure worst-case first-token time with several agents and one long prompt.