Cluster orchestration

A cluster is a scheduler, a network, a shared filesystem, and a way to fail. Orchestration is not a product name. It is the loop: submit, place, run, checkpoint, reclaim, retry.

Topics

• Scheduling. Gang scheduling for multi-GPU jobs. Queues with priority. Preemption if you share the cell.
• Networking. All-reduce needs a fast fabric. A mis-cabled rail shows up as a straggler, not as a clean error.
• Storage. Checkpoints are large. A shared POSIX fill-up is a real outage. Write to a path you can quota.
• Failure modes. GPU Xid errors, ECC, dead NCCL ranks, spot reclaim, node drain, silent throughput drop.
• Health. If you cannot answer "is rank 3 alive" you do not have a cluster, you have hope.

# Sketch only: one job, one log dir, one resume flag.
torchrun --nproc_per_node=8 train.py \
  --ckpt-dir /checkpoints/tiny-lm \
  --resume latest \
  --max-steps 2000

Write a runbook with four pages at most: how to submit, where logs live, how to resume, and what you do when one rank dies. Include a "do not SSH and compile CUDA in place" rule if that is how your last incident started.

Assignment

• Draw a sequence diagram: submit to scheduler, land on nodes, start ranks, first checkpoint, rank death, resume.
• List five failure modes you will actually see. For each, the signal and the action.
• Estimate checkpoint size from week 6's byte sketch. Say how often you can afford to write it.

Opinion checkpoint

Next: Semiconductor supply chain and capstone.