Fix startup errors, fall back to ring

What people report

Things we learned from these reports:

• In exo #1847 the cluster did not recover until every node was rebooted. The reporter asked for a way to reset RDMA state without a reboot. We did not find a documented reset API.
• In mlx #3755 the reporter narrowed the problem to one host: its connect() sat in SYN_SENT and never sent a SYN, while nc to the same address worked. In that report the ring backend over Ethernet failed too, so a fallback does not always save you.
• The Reddit poster found reports of failures on several macOS 26.x builds and one success report, and called it "not version X broke it". We could not confirm a known-good macOS build.
• A JACCL error on one Mac can be caused by another Mac running out of memory (exo #2430). Check memory first (lesson 01).

Recovery runbook

1. Stop every rank on every Mac: for h in mac1 mac2 mac3 mac4; do ssh $h 'pkill -f mlx_lm; pkill -f check_link.py'; done.
2. On every Mac run ibv_devinfo. Reseat any cable whose port is not PORT_ACTIVE.
3. Run check_link.py again (lesson 03).
4. If it still fails, reboot all Macs, not only the one that printed the error. Then run the test again.
5. If it still fails, run the cluster on the ring backend (below) and keep notes: macOS build, MLX version, exact error, which host. That is what maintainers ask for in the issues above.

Make a ring hostfile

The ring backend uses TCP sockets, so it works over Thunderbolt IP links or Ethernet. Rank i talks only to rank i − 1 and rank i + 1. The docs describe it as always available and use it for Thunderbolt rings:

uv run mlx.distributed_config --verbose \
  --hosts mac1,mac2,mac3,mac4 --over thunderbolt --backend ring \
  --auto-setup --output-hostfile ~/mlx-cluster/ring.json
# or over your normal network:
#   --over ethernet

On the ring backend, use --pipeline. Tensor parallel sends many small messages per token, Apple says very low latency is critical for it, and the MLX docs describe JACCL as necessary for things like tensor parallelism. If your model only supports tensor (lesson 01), it may still run on ring, but we could not measure how slow that is.

The MLX source also has a jaccl-ring backend option in mlx.distributed_config and mlx.launch. It is not described on the docs page and we did not test it.

Start script with fallback

This script tests JACCL first. If the ok: line is missing, it tests the ring backend and starts the server with --pipeline. It uses Python for the time limit because macOS does not ship a timeout command, and it avoids bash features that macOS's older bash lacks.

#!/usr/bin/env bash
# Test the link first, then start mlx_lm.server on the best backend that works.
set -uo pipefail

MODEL="${MODEL:?set MODEL to a Hugging Face repo or a local path}"
NODES="${NODES:-4}"
JACCL_HOSTFILE="${JACCL_HOSTFILE:-$HOME/mlx-cluster/jaccl.json}"
RING_HOSTFILE="${RING_HOSTFILE:-$HOME/mlx-cluster/ring.json}"
PY="${PY:-$HOME/mlx-cluster/.venv/bin/python}"
CHECK="${CHECK:-$HOME/mlx-cluster/check_link.py}"
PORT="${PORT:-8080}"

smoke() {  # $1 = backend, $2 = hostfile
  local out
  # macOS has no `timeout` command, so use Python for the time limit.
  # mlx.launch returns 0 even when a rank fails, so read the output instead.
  out=$("$PY" -c 'import subprocess, sys; sys.exit(subprocess.run(sys.argv[1:], timeout=120).returncode)' \
          mlx.launch --hostfile "$2" --backend "$1" \
          --env EXPECTED_WORLD_SIZE="$NODES" --env CHECK_BACKEND="$1" \
          --python "$PY" "$CHECK" 2>&1)
  echo "$out"
  grep -q "^ok: $NODES ranks" <<<"$out"
}

if smoke jaccl "$JACCL_HOSTFILE"; then
  BACKEND=jaccl; HOSTFILE="$JACCL_HOSTFILE"; SPLIT=()           # tensor parallel
elif smoke ring "$RING_HOSTFILE"; then
  echo "JACCL failed. Falling back to ring with pipeline split." >&2
  BACKEND=ring; HOSTFILE="$RING_HOSTFILE"; SPLIT=(--pipeline)
else
  echo "Both backends failed. Stop every rank, check cables, reboot all Macs." >&2
  exit 1
fi

echo "starting server: backend=$BACKEND split=${SPLIT[*]:-tensor}"
${DRY_RUN:+echo} mlx.launch --hostfile "$HOSTFILE" --backend "$BACKEND" -- \
  "$PY" -m mlx_lm server --model "$MODEL" --host 0.0.0.0 --port "$PORT" ${SPLIT[@]+"${SPLIT[@]}"}

What we could test: on our Linux box (no Macs, so no JACCL), with DRY_RUN=1, a broken JACCL hostfile and a two-rank local ring, the script fell back and printed the ring command:

mlx.launch: error: The hostfile is malformed: no RDMA device is listed for m3-ultra-1 to m3-ultra-2, ...
ok: 2 ranks, all_sum correct (3)
...
JACCL failed. Falling back to ring with pipeline split.
starting server: backend=ring split=--pipeline
mlx.launch --hostfile ring2.json --backend ring -- /path/.venv/bin/python -m mlx_lm server --model test/model --host 0.0.0.0 --port 8080 --pipeline

We did not run the JACCL success path, because that needs real Macs with RDMA. If the Python time limit stops mlx.launch, ranks on other Macs can keep running. Step 1 of the runbook cleans them up.

Sources

• MLX docs: Distributed Communication (Ring, JACCL, Tips and Tricks)
• ml-explore/mlx issue #3755
• exo-explore/exo issue #1847