The Cluster Software
The hardware is accepted. Now 1,024 GPUs need software that decides who runs what, notices when a GPU starts failing, takes the bad node out, restarts the job from its last checkpoint and tells Gopala what happened, ideally while he sleeps. This part builds that layer at start-up scale: the choice between Slurm and Kubernetes, the monitoring stack, a health check that drains bad nodes on its own, a checkpoint policy derived from Meta's published failure record, and the on-call rules for a company where the on-call rota is one person and some agents.
failure_budget.py was run (output below is real); its linear scaling from Meta's cluster to Gopala's and its checkpoint costs are our assumptions. healthcheck.sh needs a real Slurm node and was not executed for this article. Incidents in the story are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 15โ20
- Chooses Slurm for training and Kubernetes for serving (section 3). Takes the provider's managed Slurm but keeps root to install his own checks.
- Installs DCGM, the DCGM exporter, Prometheus and Grafana; writes the node health check and the alert rules (sections 4โ6).
2. What the cluster software has to do
What the cluster software does
The five jobs, in the order a failure passes through them.
The layer between the hardware and the training code has five jobs:
- Schedule: give a job all of its nodes at once (a 128-node job that gets 127 nodes cannot start), queue the rest, share the cluster between the main run, experiments and evaluation.
- Watch: collect GPU, network, storage and job metrics every few seconds.
- Detect: turn signals (an ECC error, a GPU error code, a slow step time) into a decision about a node.
- Recover: drain the bad node, bring in a spare, restart from the last checkpoint.
- Tell: page a human only when a human is needed.
- The site's scheduling and observability and on-call articles describe how large labs staff these jobs.
- Gopala has to do them with one person and scripts.
3. Slurm or Kubernetes?
| Slurm | Kubernetes (+ Kueue or similar) | |
|---|---|---|
| Built for | HPC batch jobs on fixed nodes | Long-running services and containers |
| Gang scheduling | Native: a job gets all its nodes or waits | Through add-ons (Kueue, Volcano, and others) |
| Researcher experience | sbatch, srun; familiar to most ML researchers | YAML, containers, operators |
| Serving | Awkward | Natural: autoscaling, rolling updates |
| GPU management | GRES plugins, NVIDIA tools on the node | NVIDIA GPU Operator installs drivers, DCGM exporter and device plugin |
- His choice: Slurm for the training block, because gang scheduling and simple failure handling matter most for a run on all 128 nodes, and Kubernetes for the separate serving cluster, where autoscaling and rolling updates matter.
- Projects such as SchedMD's Slinky, which runs Slurm on Kubernetes, are an option for later; for now he keeps two simple systems rather than one complicated one.
- Three partitions on the training block:
main(the big run, highest priority),exp(experiments, can be pre-empted) andeval(evaluation sweeps).- Eight nodes are kept as hot spares: in service for pre-emptible jobs, available to the main run within minutes when a node is drained.
4. The Slurm configuration
- The provider runs the Slurm controller, but the parts that encode Gopala's policy are his.
- The excerpt below is the version kept in git
- it was written for this article and not run against a real controller, so treat it as a starting point and check every option against the Slurm documentation for your version.
- The excerpt below is the version kept in git
# slurm.conf (excerpt) โ the training block. Provider-managed controller, # Gopala's own partitions, health check and failure handling. ClusterName=maestro SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory GresTypes=gpu NodeName=node[001-136] Gres=gpu:h100:8 CPUs=112 RealMemory=2000000 State=UNKNOWN PartitionName=main Nodes=node[001-136] PriorityTier=10 PreemptMode=OFF MaxTime=INFINITE PartitionName=exp Nodes=node[001-136] PriorityTier=1 PreemptMode=REQUEUE Default=YES PartitionName=eval Nodes=node[001-136] PriorityTier=5 PreemptMode=REQUEUE PreemptType=preempt/partition_prio # health: cheap check every 5 min on every node; drains, never reboots HealthCheckProgram=/etc/slurm/healthcheck.sh HealthCheckInterval=300 HealthCheckNodeState=ANY # a dead node must not hold a 128-node job hostage for long SlurmdTimeout=120 KillWait=30 UnkillableStepTimeout=180 # jobs that lose a node are requeued by the wrapper, not silently retried JobRequeue=0 ReturnToService=1
- 136 nodes: 128 for the main run plus 8 hot spares, all in every partition, so spares do useful pre-emptible work until they are needed.
- Partition priority with requeue: a
mainjob pre-emptsexpandevaljobs, which are requeued and resume from their own checkpoints. - Short timeouts: a node that stops answering is declared down in about two minutes, not the much longer defaults some sites use, so a dead node cannot hold the whole job.
- No automatic requeue of the main job: the restart wrapper does it explicitly, excluding drained nodes, so a failing node cannot be handed straight back to the job that it just broke.
- Many sites use LBNL's Node Health Check (NHC), an open-source framework that Slurm can run as its health-check program, instead of a hand-written script; GPU-specific checks are often added on top, as some GPU clouds document for their own clusters.
- Gopala starts with his short script because he can read every line of it, and plans to move to NHC when the checks grow.
5. The monitoring stack
The monitoring stack
All open source. Step time is the single most useful signal.
Everything is open source and standard:
- NVIDIA DCGM on every node reads GPU health and performance counters; the DCGM exporter publishes them as Prometheus metrics (for example
DCGM_FI_DEV_GPU_TEMP,DCGM_FI_DEV_XID_ERRORS,DCGM_FI_PROF_GR_ENGINE_ACTIVE). - node exporter for CPU, memory, disk; fabric counters for link errors.
- Prometheus stores time series; Grafana draws one dashboard per question: is the job healthy, which node is worst, is the storage keeping up.
- The training code itself logs step time, loss and tokens per second. Step time is the single most useful signal: if one node slows down, every step slows down.
- Logs (kernel messages, Slurm, training) go to one searchable store with the node name on every line, so "what happened on node 57 at 03:12" is one query.
6. Which GPU errors mean what
- The driver reports GPU problems as numbered Xid events.
- Treating every Xid as "bad GPU" would drain healthy nodes; ignoring them would leave broken ones in the job.
His handling table, built from NVIDIA's Xid documentation and kept next to the health check:
| Xid | Meaning (NVIDIA) | His action |
|---|---|---|
| 13, 31, 43 | Errors usually caused by the application (bad memory access, a kernel fault) | No drain; look at the job first |
| 48 | Double-bit ECC error | Drain; reset and run the diagnostic; RMA if repeated |
| 63, 64 | Memory row-remapping event / failure | Drain; a reset applies the remap; RMA on failure |
| 74 | NVLink error | Drain; run the single-node all-reduce from Part 6 |
| 79 | GPU has fallen off the bus | Drain; power-cycle through the BMC; ticket |
| 94, 95 | Contained / uncontained ECC error | 94: restart the job; 95: drain and reset |
- The table is a living document: each new incident either confirms a row or changes it, and the change is a commit with the incident linked.
7. Alert rules: who gets woken up
- Metrics are useless without rules that decide what deserves attention.
- The rules are short and split into two severities: ticket (handled in working hours, or by automation) and page (a human, now).
- Like the Slurm file, they were written for this article and not run against a live Prometheus
- the metric names starting
DCGM_come from NVIDIA's exporter, thetrain_ones are what his training code exports.
- the metric names starting
# alerts.yml โ Prometheus alert rules. Only "page" wakes a human.
groups:
- name: maestro-training
rules:
- alert: GpuXidError
expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
labels: {severity: ticket}
annotations: {summary: "Xid {{ $value }} on {{ $labels.Hostname }} GPU {{ $labels.gpu }}"}
- alert: GpuHot
expr: DCGM_FI_DEV_GPU_TEMP > 85
for: 10m
labels: {severity: ticket}
- alert: StepTimeSlow # the whole job is slower than its own baseline
expr: train_step_seconds > 1.2 * avg_over_time(train_step_seconds[6h])
for: 30m
labels: {severity: page}
- alert: StragglerRank # one node much slower than the median
expr: max(train_rank_step_seconds) > 1.05 * quantile(0.5, train_rank_step_seconds)
for: 15m
labels: {severity: ticket}
- alert: TwoRestartsInAnHour
expr: increase(train_restarts_total[1h]) >= 2
labels: {severity: page}
- alert: NoHotSpare
expr: slurm_nodes_idle{partition="main"} < 1
for: 30m
labels: {severity: page}
- alert: LossDiverging
expr: train_loss > 1.5 * min_over_time(train_loss[2h])
for: 10m
labels: {severity: page}- Only four conditions page: the whole job slowing down, two restarts in an hour, no spare left, and the loss diverging.
- Everything else is either handled by the drain-and-restart path or waits for the morning.
8. A health check that drains bad nodes
- Slurm can run a program on every node at a fixed interval and act on the result.
- Gopala's check is cheap (seconds, not minutes), and it only drains a node: it marks it so no new job lands there and lets the scheduler and the restart logic handle the job.
- It never reboots anything on its own; a human or a separate, slower repair loop does that.
#!/bin/bash
# healthcheck.sh โ Slurm HealthCheckProgram, runs every 5 minutes on each node
# (slurm.conf: HealthCheckProgram=/etc/slurm/healthcheck.sh
# HealthCheckInterval=300 HealthCheckNodeState=ANY)
# Cheap checks only: a node that fails is drained, never rebooted from here.
H=$(hostname -s)
drain() { scontrol update nodename=$H state=drain reason="hc: $1"; exit 0; }
[ "$(nvidia-smi -L | wc -l)" -eq 8 ] || drain "gpu count"
nvidia-smi -q -d ECC | grep -q "Pending Page Blacklist *: Yes" && drain "ecc retire pending"
dmesg -T --since "-10min" | grep -qE "NVRM: Xid.*(48|63|64|74|79|94|95)" && drain "xid $(dmesg | grep -oE 'Xid[^,]*' | tail -1)"
[ "$(ibstat | grep -c 'State: Active')" -ge 8 ] || drain "ib link down"
df -P /checkpoints >/dev/null 2>&1 || drain "checkpoint fs"
T=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader | sort -n | tail -1)
[ "$T" -lt 87 ] || drain "gpu temp $T"
exit 0- The NVIDIA Xid codes it watches are documented error events from the GPU driver: for example, 48 (double-bit ECC error), 63 and 64 (memory row remapping events), 74 (NVLink error), 79 (GPU has fallen off the bus) and 94/95 (contained and uncontained ECC errors).
- Not every Xid means a bad GPU, so each code's handling is in a table he reviews, and new codes are added only after looking at NVIDIA's Xid documentation.
- Health checks catch failures.
- They do not catch slow nodes, which are just as harmful to a synchronised job.
- For that, the training code reports per-rank step time, and an alert fires when one node is consistently more than a few percent slower than the median
- Gopala then tests it with Part 6's burn-in.
9. See a run unfold
A simulated 31-day run (our model)
10. Replaying the run: spares and bad luck
Hot spares vs goodput, at two failure rates
Real output of restart_sim.py, 200 simulated runs each (our model).
- The arithmetic above uses averages.
- Real failures cluster, spares run out, and the second failure in a day hurts more than the first.
- So Gopala replays the 31-day run thousands of times with random failures and a simple model of the restart path:
- with a free hot spare, recovery takes 10 minutes
- without one, he waits four hours for the provider to swap a node
- failed nodes come back after 48 hours.
"""restart_sim.py โ replay a 31-day run through the failure and restart logic.
Monte Carlo: failures arrive at random (exponential gaps) at a given
cluster MTTF. Each failure loses the work since the last checkpoint and
costs a recovery time that depends on whether a hot spare is free;
failed nodes come back after a repair time. Compares two failure rates:
the one scaled from Llama 3 (Part 7, ~49 h) and the much worse 7.9 h
reported for 1,024-GPU jobs in Meta's research clusters (A100s).
All timings are our assumptions. Fixed seed, so the output is reproducible.
"""
import random
DAYS, CKPT_H, SAVE_H = 31, 1.0, 30 / 3600
RECOVER_SPARE_H, RECOVER_WAIT_H, REPAIR_H = 10 / 60, 4.0, 48.0
def run(mttf_h, spares, seed):
rng = random.Random(seed)
t, lost, fails, waits, back = 0.0, 0.0, 0, 0, []
end = DAYS * 24
while True:
gap = rng.expovariate(1 / mttf_h)
if t + gap >= end:
break
t += gap; fails += 1
back = [b for b in back if b > t] # repaired nodes rejoin the spare pool
lost += (t % CKPT_H) # work since last checkpoint
if len(back) < spares:
rec = RECOVER_SPARE_H
else:
rec = RECOVER_WAIT_H; waits += 1 # wait for the provider to swap a node
back.append(t + REPAIR_H)
lost += rec; t += rec
saves = end / CKPT_H * SAVE_H
return fails, waits, 1 - (lost + saves) / end
def main():
print(f"{'cluster MTTF':>12s} {'spares':>6s} {'failures':>8s} {'no spare':>8s} {'goodput':>8s}")
for mttf in (49.0, 7.9):
for spares in (0, 4, 8):
res = [run(mttf, spares, s) for s in range(200)]
f = sum(r[0] for r in res) / len(res); w = sum(r[1] for r in res) / len(res)
g = sum(r[2] for r in res) / len(res)
print(f"{mttf:10.1f} h {spares:6d} {f:8.1f} {w:8.1f} {g:8.1%}")
if __name__ == "__main__":
main()Two findings shape the design:
- At the Llama 3-derived rate (one failure about every 49 hours), four hot spares already capture almost all the benefit: goodput rises from about 91% with none to about 98%. Eight spares add almost nothing, so spares 5โ8 are insurance, not throughput.
- At the research-cluster rate (MTTF of about 7.9 hours for 1,024-GPU jobs, reported by Meta's FAIR team for A100 research clusters), even eight spares leave goodput near 85%, and with none the run would lose almost 40% of its time. In that world the bottleneck is how fast the provider repairs nodes, not how many spares he holds.
- The failures column grows with more spares at the high rate because less time is spent waiting, so more of the 31 days is exposed to failure; it is a sign the simulation is counting time correctly, not a paradox.
- Which world will he be in?
- He does not know until the main block runs, so the plan watches the first two weeks of experiments (Part 9) for the real failure rate and adjusts spares and the checkpoint interval before the main run starts.
11. How often will the main run break?
Goodput by checkpoint interval (10-minute recovery)
Real output of failure_budget.py (failure rate scaled from Llama 3; our assumptions).
- To set the checkpoint interval he needs a failure rate.
- The best public number is Meta's: 419 unexpected interruptions in 54 days on 16,384 GPUs.
- Scaling it linearly by GPU count (a simplification: his hardware, provider and software differ) gives his expectation for a 31-day run on 1,024 GPUs.
- Then the script asks what each choice of checkpoint interval and recovery time costs.
"""failure_budget.py โ how often will Maestro-1's run be interrupted?
Scales Meta's published Llama 3 405B failure record (419 unexpected
interruptions in 54 days on 16,384 H100s) to Gopala's cluster, then
asks how checkpoint interval and recovery time change goodput.
Assumes failures scale linearly with GPU count (our assumption).
"""
META = {"gpus": 16384, "days": 54, "unexpected": 419}
GPUS, DAYS = 1024, 31
def main():
rate = META["unexpected"] / META["gpus"] / META["days"] # per GPU-day
n = rate * GPUS * DAYS
print(f"rate {rate*1e3:.3f} interruptions per 1,000 GPU-days")
print(f"Maestro-1: {GPUS} GPUs x {DAYS} days -> {n:.1f} expected interruptions "
f"(one every {DAYS*24/n:.0f} h)")
print(f"{'ckpt every':>10s} {'recover':>8s} {'lost h':>7s} {'goodput':>8s}")
for ckpt_min in (15, 30, 60, 180):
for rec_min in (10, 45):
# lose half a checkpoint interval of work plus the recovery time
lost = n * (ckpt_min / 2 + rec_min) / 60
save = DAYS * 24 * 60 / ckpt_min * 0.5 / 60 # 30 s blocking save, our assumption
good = 1 - (lost + save) / (DAYS * 24)
print(f"{ckpt_min:7d} min {rec_min:5d} min {lost+save:7.1f} {good:8.1%}")
if __name__ == "__main__":
main()Reading it:
- About 15 interruptions are expected, one every two days. Each one costs the work since the last checkpoint (half an interval, on average) plus recovery.
- Checkpointing too often wastes time saving; too rarely wastes time redoing work. With these assumptions the best interval is around an hour, which gives about 97.8% goodput with fast recovery.
- Recovery time matters as much as checkpoint interval. Going from 45 to 10 minutes per restart is worth about 9 GPU-cluster-hours over the run, roughly 9,000 GPU-hours or ~$21k at $2.35. That is why the hot spares and the automated restart exist.
- Asynchronous checkpointing (saving in the background while training continues), now supported by the main training frameworks, cuts the save cost further
- Part 10 uses it.
12. Making checkpoints cheap
- The budget above assumes each save blocks training for 30 seconds.
- Whether that is realistic depends on the framework.
- PyTorch's team, working with IBM, reported that distributed checkpointing cut checkpoint times from roughly half an hour to under five minutes, and that asynchronous checkpointing (copying state to CPU memory, then writing to storage in the background) brought the visible pause down to seconds: about 6โ14 seconds for 7Bโ13B models in their tests.
- Later work in TorchTitan reported further reductions.
Three implications for Gopala:
- With asynchronous saves, the best interval moves shorter, because saving is nearly free and lost work is what remains.
- Node memory must hold a copy of the model and optimiser shards while they are written out.
- A checkpoint is only useful if it can be read back: restart time is set by read bandwidth, which Part 6 tested.
13. The serving cluster on Kubernetes
- The 64 serving GPUs (Part 4) run a different stack.
- NVIDIA's GPU Operator installs the driver, container toolkit, device plugin and DCGM exporter on every node, so GPUs appear to Kubernetes as schedulable resources and their metrics flow into the same Prometheus.
- Model servers (Part 15) run as deployments with rolling updates; an autoscaler adds replicas when queue length grows.
- The two clusters share monitoring, alert routing and the runbook, but not a scheduler, so a training emergency can never take the product down.
14. The fire drill
The recovery chain tested in the fire drill
Scenario: 41 minutes on the first drill, 11 after fixes.
- A recovery path that has never been tested does not exist.
- In week 20, during a 32-node Maestro-0 post-training job, Gopala marks one node as failed on purpose and watches the chain: health check drains the node โ job fails โ the restart wrapper requests a new allocation with a hot spare โ training resumes from the last checkpoint โ an alert summarises what happened.
- The first drill takes 41 minutes end to end: the job waits 25 minutes for Slurm to give up on the dead node.
- Shortening the job's timeout and having the wrapper exclude drained nodes explicitly brings the second drill down to 11 minutes.
- These timings are the story's, not measurements.
15. The runbook
- Each paging alert has a page in the runbook with the same five headings: what it means, how to confirm it, the first safe action, when to escalate to the provider, and how to record it.
Example, "two restarts in an hour":
- Means: the automatic path is failing, or one bad node keeps coming back.
- Confirm: list the nodes involved in both failures; check drain reasons and Xid codes.
- First action: exclude any node involved in both; restart from the last good checkpoint.
- Escalate: if a third restart follows within an hour, open a severity-1 ticket with the provider and stop the job.
- Record: incident note in git, Xid table updated if a new code appeared.
16. On call, with one person
Gopala cannot be awake for 31 days. The rules:
- Automate the common case. A node failure that the health check catches and the restart wrapper handles does not page anyone; it is in the morning summary.
- Page only on what automation cannot fix: loss diverging, step time 20% slower for 30 minutes, two restarts within an hour, storage full, no spare left.
- Agents read, humans decide. An AI agent with read-only access to logs and metrics writes the first incident summary and a suggested cause. It can open a ticket with the provider. It cannot drain, reboot or change a job.
- A second person for the main run. A contractor shares the night shifts for the 31 days, with a written runbook.
17. Seven cluster-software mistakes
- Letting the health check reboot nodes, so a flapping node keeps rejoining the job.
- Watching GPU utilisation instead of step time; a straggler keeps every GPU "busy" waiting.
- Paging on every Xid, until the on-call person stops reading pages.
- Default timeouts that let a dead node hold a 128-node job for half an hour.
- Never testing a restart until the main run needs one.
- Checkpoints that are written but never test-read.
- Changing configuration during the run without a commit.
18. Everything in git
- The Slurm configuration, health check, alert rules, dashboards, the golden image definition (Part 6) and the restart wrapper live in one repository.
- Every change is a commit with a reason, and the main run freezes it the same way the facility is frozen (Part 5).
- When something breaks in month 10, the question "what changed?" has an answer.
19. Delegation box
- The scheduler design and partitions
- The health-check rules and paging rules
- The fire drill
- First incident summaries from logs (read-only)
- Drafting provider tickets
- Writing dashboard queries and alert rules for review
- The provider's managed Slurm and their on-site staff
- Open-source tools: Slurm, Kubernetes, DCGM, Prometheus, Grafana
- A contractor sharing nights during the main run
20. Budget and risk tracker
| Item | Status after Part 7 | Note |
|---|---|---|
| Spend to date | ~$1.1M (scenario) | Training block billing since acceptance, serving cluster from month 4 |
| Cluster | 128 nodes in service, 8 hot spares | Maestro-0 post-training running |
| Time | 20 of ~78 weeks (month ~4.5) | Phase 1 complete |
| Recovery risk | Tested | Recovery in ~11 minutes in the drill (scenario) |
| Silent slowdown risk | Medium | Per-rank step-time alerts |
| Key-person risk | High | Runbook and a second on-call for the main run |
21. The series
The Maestro: 18 parts in five phases
- Phase 1 ends with a machine that runs itself most of the time: Slurm for training, Kubernetes for serving, health checks that drain bad nodes, hot spares, hourly checkpoints and a tested 11-minute recovery.
- Phase 2 starts with what the machine eats: Part 8, the data refinery.
- Follow it on Maestro Mission Control.
22. Reading the numbers
- Primary / standard
- Llama 3 405B: 419 unexpected interruptions in 54 days on 16,384 H100s; >90% effective training time (Meta). Slurm HealthCheckProgram and node drain (SchedMD docs). Xid error meanings (NVIDIA Xid documentation). DCGM exporter metric names (NVIDIA). Kueue (Kubernetes SIG Scheduling). 1,024-GPU job MTTF ~7.9 h in Meta research clusters (arXiv:2410.21680). Checkpoint times ~30 min โ <5 min with distributed checkpointing, ~6โ14 s visible pause with async for 7Bโ13B models (PyTorch blog). LBNL NHC.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Linear scaling of failures with GPU count, the simulation's recovery (10 min with a spare, 4 h without) and 48 h repair times, 30-second blocking save, recovery times, the drill timings, partitions, spares and paging rules.
- Sources
- Slurm slurm.conf (HealthCheckProgram) ยท Kueue ยท Slinky (Slurm on Kubernetes)
- NVIDIA Xid errors ยท NVIDIA DCGM exporter ยท NVIDIA GPU Operator
- Meta: The Llama 3 Herd of Models ยท Meta FAIR: Revisiting Reliability in Large-Scale ML Research Clusters ยท Prometheus
- PyTorch: reducing checkpointing times ยท LBNL Node Health Check (NHC)