Run One Large LLM Across Several Macs with MLX

Run a frontier-class open model on your own Macs, for private code and documents, when it is too big for one machine.

Pipeline vs tensor parallel, memory per Mac, Thunderbolt 5 mesh, RDMA and the JACCL backend, link tests, known failures, ring fallback, and serving agents.

Choose a split, prove every Mac can hold its share, build and check a JACCL hostfile, test the link in seconds, recover or fall back when JACCL fails, and measure what agents wait.

Who this is for

You run MLX on a Mac and want to run a model that does not fit on one machine, such as the 816B Muse Spark Open model from Apple's October 2026 post. You are comfortable with Terminal, ssh, and Python. All code is Python and bash. There is no Swift or Xcode in this course.

We could not run anything on real Macs for this course. Every command was checked against the MLX docs and source, and the scripts that do not need Macs were run on a Linux box. Each lesson says what we ran and what we could not. For single-Mac memory limits see Keep On-Device LLMs Alive on Real Phones. For MLX in an iOS app see MLX Swift in Production.

Lessons

Sources we checked

Apple: Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio (15 October 2026)
MLX: Distributed Communication docs, source for mlx.launch and mlx.distributed_config (mlx 0.32.x)
mlx-lm: repository (sharded_load, pipeline split, server options) and SERVER.md
Failure reports: r/LocalLLM: 4× M3 Ultra JACCL cluster dies with errno 60, exo #1847, mlx #3755, exo #2430, exo #1843, mlx #3142

Key concepts

Pipeline splits by depth and mainly makes a model fit. Tensor splits every layer and makes it faster, but needs JACCL.
Every Mac must hold its own share. mlx-lm splits pipeline by layer count and tensor evenly.
JACCL needs a full Thunderbolt 5 mesh, RDMA enabled in Recovery, and a correct hostfile.
Test the link with a tiny all_sum before loading weights, and read the output, not the exit code.
Known JACCL failures can need a reboot of every Mac. The ring backend with --pipeline is the fallback.

Takeaways

You can choose a split and prove each Mac can hold its part before you download anything.
You can build, check, and test a cluster in minutes instead of finding out after a long weight load.
You have a recovery order and a start script that falls back to ring.
You can serve the cluster to agents and know the worst first-token time they will see.

Agent briefs

FAQ

Should I use pipeline or tensor parallel with MLX on several Macs?

Use tensor parallel when you have a working RDMA mesh (JACCL) and the model supports it; it speeds up decode. Use pipeline with --pipeline to make a model fit, or on the ring backend.

How many Thunderbolt cables does an MLX JACCL cluster need?

JACCL needs a full mesh: a cable between every pair of Macs, so N times N minus 1 divided by 2. Four Macs need 6 cables, or 12 with two cables per pair as Apple used.

How do I enable RDMA over Thunderbolt for MLX?

On each Mac, start in Recovery, open Terminal, run rdma_ctl enable, and reboot. It cannot be done over ssh. Check with ibv_devices. You need macOS 26.2 or newer.

What does [jaccl] Couldn't connect (error: 60) mean?

Error 60 is a timeout while the Macs set up the RDMA connection, before any weights load. Stop all ranks, check PORT_ACTIVE, rerun a small all_sum test, reboot every Mac, then fall back to ring.

How long is the first token for a 64K prompt on a Mac cluster?

Apple measured about 40.6 seconds for Muse Spark Open on four M5 Ultra Mac Studios with two Thunderbolt cables per pair, and about 50.5 seconds with one cable.