# Run One Large LLM Across Several Macs with MLX: Lesson 02. Cable the mesh and turn on RDMA

Lesson id: `mlx-distributed-mac-cluster/02-thunderbolt-mesh-and-rdma`
Human lesson: https://edgefde.dev/courses/mlx-distributed-mac-cluster/02-thunderbolt-mesh-and-rdma.html

## Goal

Build a full Thunderbolt 5 mesh, enable RDMA on every Mac, and produce a checked JACCL hostfile.

## Prerequisites

- Lesson 01 done
- Passwordless ssh to every Mac
- Physical access to each Mac for Recovery

## Inputs

- Host names
- Cable count per pair

## Outputs and artifacts

- `~/mlx-cluster/jaccl.json`
- Topology picture from `--dot`
- `check_hostfile.py` passing

## Steps

1. Cable every pair of Macs with Thunderbolt 5 (same cable count for every pair).
2. On each Mac: boot to Recovery, open Terminal, run `rdma_ctl enable`, reboot.
3. On each Mac run `ibv_devices` and `ibv_devinfo`; every connected port must be PORT_ACTIVE.
4. Run `uv run mlx.distributed_config --verbose --hosts <hosts> --over thunderbolt --dot` and confirm every pair is connected.
5. Run `uv run mlx.distributed_config --verbose --hosts <hosts> --over thunderbolt --backend jaccl --auto-setup --output-hostfile ~/mlx-cluster/jaccl.json`.
6. Run `uv run python check_hostfile.py ~/mlx-cluster/jaccl.json` and fix every problem it prints.
7. Copy the hostfile to the same path on every Mac.

## Constraints

- Do not run Exo on the same Macs.
- Do not mix cable counts between pairs.
- Do not claim a port count for a Mac model without its spec page.

## Acceptance checks

- `check_hostfile.py` exits 0.
- Every Mac lists RDMA devices and PORT_ACTIVE ports.

## Key concepts

- JACCL needs a full mesh: N × (N − 1) / 2 Thunderbolt links. Ring needs only neighbours.
- RDMA is turned on per Mac in Recovery with `rdma_ctl enable`. It cannot be done over ssh.
- The hostfile lists, per rank, the RDMA device to every other rank, `null` for itself, and an IP for rank 0.
- Two cables per pair help prefill. JACCL uses the smallest cable count across all pairs.

## Takeaways

- Count ports before you buy cables.
- Check `ibv_devices` and `PORT_ACTIVE` on every Mac after every reboot.
- Generate the hostfile with `mlx.distributed_config`, then check it with a script before launch.