---
name: mlx-distributed-mac-cluster
description: >-
  Run one large LLM across several Apple silicon Macs with MLX distributed. Choose pipeline or tensor parallel, size memory per Mac, cable a Thunderbolt 5 mesh with RDMA (JACCL), test the link with mlx.launch, recover from JACCL errors, fall back to ring, and serve agents with mlx_lm.server. Python and bash.
---
# Run One Large LLM Across Several Macs with MLX

Run a frontier-class open model on your own Macs, for private code and documents, when it is too big for one machine.

Course home: https://edgefde.dev/courses/mlx-distributed-mac-cluster/

## When to use

Use when a model does not fit on one Mac, when planning a Mac Studio cluster, or when a JACCL cluster fails at startup with errno 60 or errno 22, or crashes and needs reboots.

## Environment

Two or more Apple silicon Macs with Thunderbolt 5, macOS 26.2+ for RDMA, MLX 0.30.1+. On every Mac, same path: `uv init --bare --python 3.11`, `uv venv`, `uv add mlx mlx-lm huggingface_hub`, optional `uv add torch` (MPS on Apple silicon, CPU elsewhere), `uv run ...`. No Xcode needed.

## Recipe

1. **Pick pipeline or tensor, size memory.** Decide pipeline or tensor parallel for a model, confirm the model supports it, and confirm every Mac can hold its share. Brief: `01-pipeline-or-tensor-and-memory.llms.md`.
2. **Cable the mesh and turn on RDMA.** Build a full Thunderbolt 5 mesh, enable RDMA on every Mac, and produce a checked JACCL hostfile. Brief: `02-thunderbolt-mesh-and-rdma.llms.md`.
3. **Test the link before loading weights.** Prove the cluster link works and record latency and bandwidth before loading any model. Brief: `03-test-the-link-first.llms.md`.
4. **Fix startup errors, fall back to ring.** Recover from JACCL startup failures in a fixed order and fall back to the ring backend with pipeline parallel. Brief: `04-startup-failures-and-ring-fallback.llms.md`.
5. **Long prompts and serving many agents.** Serve the cluster to several agents with mlx_lm.server and measure first-token time under load. Brief: `05-long-prompts-and-many-agents.llms.md`.

## Acceptance checks

- `check_split_support.py` and `size_shards.py` show the chosen split fits on every Mac.
- `check_hostfile.py` passes on the JACCL hostfile.
- `check_link.py` prints `ok: N ranks` through `mlx.launch` before any model load.
- `start-cluster.sh` falls back to ring with `--pipeline` when JACCL fails.
- Worst-case first-token time is measured with several agents.

## Key concepts

- Pipeline splits by depth and mainly makes a model fit. Tensor splits every layer and makes it faster, but needs JACCL.
- Every Mac must hold its own share. mlx-lm splits pipeline by layer count and tensor evenly.
- JACCL needs a full Thunderbolt 5 mesh, RDMA enabled in Recovery, and a correct hostfile.
- Test the link with a tiny `all_sum` before loading weights, and read the output, not the exit code.
- Known JACCL failures can need a reboot of every Mac. The ring backend with `--pipeline` is the fallback.

## Takeaways

- You can choose a split and prove each Mac can hold its part before you download anything.
- You can build, check, and test a cluster in minutes instead of finding out after a long weight load.
- You have a recovery order and a start script that falls back to ring.
- You can serve the cluster to agents and know the worst first-token time they will see.