# Run One Large LLM Across Several Macs with MLX: Lesson 04. Fix startup errors, fall back to ring

Lesson id: `mlx-distributed-mac-cluster/04-startup-failures-and-ring-fallback`
Human lesson: https://edgefde.dev/courses/mlx-distributed-mac-cluster/04-startup-failures-and-ring-fallback.html

## Goal

Recover from JACCL startup failures in a fixed order and fall back to the ring backend with pipeline parallel.

## Prerequisites

- Lessons 01 to 03 done
- `jaccl.json` and `ring.json` on every Mac

## Inputs

- Model id
- Number of Macs
- Paths to hostfiles, venv Python, and check_link.py

## Outputs and artifacts

- `start-cluster.sh`
- `ring.json`
- Incident notes with versions and errors

## Steps

1. Create `ring.json` with `uv run mlx.distributed_config --verbose --hosts <hosts> --over thunderbolt --backend ring --auto-setup --output-hostfile ~/mlx-cluster/ring.json`.
2. Save `start-cluster.sh` and run it with `MODEL=<model> NODES=N ./start-cluster.sh`.
3. On JACCL failure, follow the runbook: stop all ranks, check `ibv_devinfo`, rerun the test, reboot all Macs, rerun the test.
4. If JACCL still fails, let the script start the ring backend with `--pipeline`.
5. Record macOS build, MLX version, the exact error, and the host for every failure.

## Constraints

- Leave `MLX_METAL_FAST_SYNCH` unset in production.
- Do not claim a known-good macOS build.
- Do not run tensor parallel on ring without measuring it.

## Acceptance checks

- A broken JACCL hostfile makes the script fall back to ring.
- The server starts only after an `ok:` line.

## Key concepts

- `Couldn't connect (error: 60)` is a timeout at startup. `Changing queue pair to RTR failed with errno 22` is a queue pair setup failure.
- Reports show some JACCL failures clear only after rebooting every Mac. No documented reset API was found.
- `MLX_METAL_FAST_SYNCH=1` is in Apple's benchmark commands, but the MLX docs say leave it unset.
- The ring backend uses TCP and fits pipeline parallel. JACCL is what makes tensor parallel practical.

## Takeaways

- Keep a written recovery order: stop all ranks, check ports, test, reboot all, test, then fall back.
- Start the server only after a passing link test, and fall back to ring plus `--pipeline` on failure.
- Log the macOS build, MLX version, and exact error every time, so you can compare with open issues.