Cable the mesh and turn on RDMA

Pick the topology

The MLX distributed docs say the JACCL backend supports only fully connected topologies: a Thunderbolt cable must connect every pair of Macs directly. The ring backend only needs each Mac to reach its two neighbours.

The formula is N × (N − 1) / 2 links. Check how many Thunderbolt 5 ports your Mac model has on its Apple tech specs page before you buy cables. We did not verify port counts per model.

Apple used two cables between every pair. JACCL sends large messages over all cables of a pair at once. For Muse Spark Open on four M5 Ultras at a 65,536-token context, two cables per link cut the first-token time from about 50.5 s to 40.6 s. The docs add one catch: if some pairs have fewer cables than others, JACCL uses the smallest count for all pairs.

Turn on RDMA

The docs say RDMA over Thunderbolt cannot be enabled remotely, even with sudo. On every Mac:

1. Start the Mac in Recovery.
2. Open Utilities, then Terminal.
3. Run rdma_ctl enable.
4. Reboot.

Then check that RDMA devices exist. The docs show output like this for an M3 Ultra:

ibv_devices
#    device                 node GUID
#    ------              ----------------
#    rdma_en2            8096a9d9edbaac05
#    rdma_en3            8196a9d9edbaac05
#    ...
ibv_devinfo | grep -E "hca_id|state"   # each connected port should show PORT_ACTIVE

The PORT_ACTIVE check comes from user reports (mlx #3755 and the r/LocalLLM thread). In the Reddit thread, one link came up PORT_DOWN after a reboot and had to be reseated.

Network setup and the hostfile

JACCL does not send data over TCP/IP, but the docs say you still must disable the Thunderbolt Bridge and give each Thunderbolt link its own small network. JACCL also uses a TCP side channel at startup, so rank 0 needs an IP that every Mac can reach. mlx.distributed_config does all of this for you. First look at the cabling:

uv run mlx.distributed_config --verbose \
  --hosts mac1,mac2,mac3,mac4 --over thunderbolt --dot | dot -Tpng | open -f -a Preview

When the picture shows every pair connected, write the hostfile. With passwordless sudo on the Macs, --auto-setup runs the network commands. Without it, the tool prints the commands for you to run.

uv run mlx.distributed_config --verbose \
  --hosts mac1,mac2,mac3,mac4 --over thunderbolt --backend jaccl \
  --auto-setup --output-hostfile ~/mlx-cluster/jaccl.json

The docs example writes --output. In the source (mlx 0.32.x) the flag is --output-hostfile; the short form works because Python's argparse accepts a unique prefix.

A JACCL hostfile is a list with one entry per Mac, in rank order. rdma lists the RDMA device that reaches each other Mac, and null for itself. For two cables, write a list of devices for that peer. The source also accepts an object with backend, envs, and hosts keys.

[
  {"ssh": "mac1", "ips": ["192.168.10.1"], "rdma": [null, "rdma_en5", "rdma_en4", "rdma_en3"]},
  {"ssh": "mac2", "ips": [], "rdma": ["rdma_en5", null, "rdma_en3", "rdma_en4"]},
  {"ssh": "mac3", "ips": [], "rdma": ["rdma_en4", "rdma_en3", null, "rdma_en5"]},
  {"ssh": "mac4", "ips": [], "rdma": ["rdma_en3", "rdma_en4", "rdma_en5", null]}
]

Check the hostfile before you launch

mlx.launch already refuses some broken files. When we removed one link from the docs example, it stopped with The hostfile is malformed: no RDMA device is listed for m3-ultra-1 to m3-ultra-2. This script checks the same rules plus mismatched cable counts, and it gives a clean exit code for CI:

"""Check a JACCL hostfile before you launch anything.

Rules come from the MLX distributed docs and mlx/_distributed_utils/launch.py:
rank 0 needs an IP, every rank lists one rdma entry per rank, its own entry is
null, every pair is connected (full mesh), and the cable lists match.
"""
import json
import sys


def load_hosts(path):
    data = json.load(open(path))
    return data["hosts"] if isinstance(data, dict) else data


def as_list(entry):
    if entry is None:
        return []
    return entry if isinstance(entry, list) else [entry]


def check(hosts):
    problems = []
    n = len(hosts)
    if n < 2:
        problems.append("need at least 2 hosts")
    if not hosts[0].get("ips"):
        problems.append("rank 0 needs an IP that every other Mac can reach")
    for i, h in enumerate(hosts):
        rdma = h.get("rdma", [])
        if len(rdma) != n:
            problems.append(f"{h['ssh']}: rdma has {len(rdma)} entries, expected {n}")
            continue
        if rdma[i] is not None:
            problems.append(f"{h['ssh']}: its own rdma entry (position {i}) must be null")
    if problems:
        return problems
    cables = set()
    for i in range(n):
        for j in range(i + 1, n):
            a = as_list(hosts[i]["rdma"][j])
            b = as_list(hosts[j]["rdma"][i])
            if not a or not b:
                problems.append(f"no cable listed between {hosts[i]['ssh']} and {hosts[j]['ssh']}")
            elif len(a) != len(b):
                problems.append(f"{hosts[i]['ssh']} lists {len(a)} cables to {hosts[j]['ssh']}, "
                                f"but {hosts[j]['ssh']} lists {len(b)}")
            else:
                cables.add(len(a))
    if len(cables) > 1:
        problems.append(f"pairs have different cable counts {sorted(cables)}; "
                        "JACCL uses the smallest count for every pair")
    return problems


if __name__ == "__main__":
    hosts = load_hosts(sys.argv[1])
    problems = check(hosts)
    for p in problems:
        print("problem:", p)
    n = len(hosts)
    print(f"{n} hosts, {n * (n - 1) // 2} Mac-to-Mac links needed for a full mesh")
    sys.exit(1 if problems else 0)

$ uv run python check_hostfile.py jaccl.json
4 hosts, 6 Mac-to-Mac links needed for a full mesh
$ uv run python check_hostfile.py broken.json
problem: no cable listed between m3-ultra-1 and m3-ultra-2
4 hosts, 6 Mac-to-Mac links needed for a full mesh

Sources

• MLX docs: Distributed Communication (JACCL, Enabling RDMA, Defining a Mesh)
• MLX source: mlx.distributed_config
• Apple Machine Learning Research: Distributed LLM inference with MLX and RDMA