Home โ€บ Blog โ€บ The Maestro ยท Part 7
The Maestro ยท Part 7 ยท Phase 1 ยท ๐Ÿ—๏ธ Infrastructure hat

The Cluster Software

The hardware is accepted. Now 1,024 GPUs need software that decides who runs what, notices when a GPU starts failing, takes the bad node out, restarts the job from its last checkpoint and tells Gopala what happened, ideally while he sleeps. This part builds that layer at start-up scale: the choice between Slurm and Kubernetes, the monitoring stack, a health check that drains bad nodes on its own, a checkpoint policy derived from Meta's published failure record, and the on-call rules for a company where the on-call rota is one person and some agents.

FL
FrontierAGI Team
๐Ÿ—๏ธ Infrastructure๐Ÿ› ๏ธ SRE๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) Slurm, Kubernetes, Kueue, the NVIDIA GPU Operator, DCGM and its exporter, Prometheus and Grafana are real, widely used tools. The Llama 3 failure numbers are Meta's. failure_budget.py was run (output below is real); its linear scaling from Meta's cluster to Gopala's and its checkpoint costs are our assumptions. healthcheck.sh needs a real Slurm node and was not executed for this article. Incidents in the story are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 15โ€“20

Weeks 15โ€“16 ยท Choosing
  • Chooses Slurm for training and Kubernetes for serving (section 3). Takes the provider's managed Slurm but keeps root to install his own checks.
Weeks 17โ€“18 ยท Watching
  • Installs DCGM, the DCGM exporter, Prometheus and Grafana; writes the node health check and the alert rules (sections 4โ€“6).
Weeks 19โ€“20 ยท Recovering
  • Writes failure_budget.py and sets the checkpoint interval (section 11). Runs a fire drill: pulls a node in the middle of a Maestro-0 job and times the recovery (section 14).

2. What the cluster software has to do

What the cluster software does

๐Ÿ“…ScheduleGang-schedule 128 nodes at once
๐Ÿ‘€WatchGPU, network, storage, step time
๐ŸšจDetectXid, ECC, slow ranks
๐Ÿ”RecoverDrain, hot spare, restart from checkpoint
๐Ÿ“ฃTellPage a human only when needed

The five jobs, in the order a failure passes through them.

The layer between the hardware and the training code has five jobs:

  1. Schedule: give a job all of its nodes at once (a 128-node job that gets 127 nodes cannot start), queue the rest, share the cluster between the main run, experiments and evaluation.
  2. Watch: collect GPU, network, storage and job metrics every few seconds.
  3. Detect: turn signals (an ECC error, a GPU error code, a slow step time) into a decision about a node.
  4. Recover: drain the bad node, bring in a spare, restart from the last checkpoint.
  5. Tell: page a human only when a human is needed.

3. Slurm or Kubernetes?

SlurmKubernetes (+ Kueue or similar)
Built forHPC batch jobs on fixed nodesLong-running services and containers
Gang schedulingNative: a job gets all its nodes or waitsThrough add-ons (Kueue, Volcano, and others)
Researcher experiencesbatch, srun; familiar to most ML researchersYAML, containers, operators
ServingAwkwardNatural: autoscaling, rolling updates
GPU managementGRES plugins, NVIDIA tools on the nodeNVIDIA GPU Operator installs drivers, DCGM exporter and device plugin

4. The Slurm configuration

slurm.conf ยท Slurm configuration excerpt ยท not executed for this article
# slurm.conf (excerpt) โ€” the training block. Provider-managed controller,
# Gopala's own partitions, health check and failure handling.
ClusterName=maestro
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
GresTypes=gpu
NodeName=node[001-136] Gres=gpu:h100:8 CPUs=112 RealMemory=2000000 State=UNKNOWN
PartitionName=main Nodes=node[001-136] PriorityTier=10 PreemptMode=OFF MaxTime=INFINITE
PartitionName=exp  Nodes=node[001-136] PriorityTier=1  PreemptMode=REQUEUE Default=YES
PartitionName=eval Nodes=node[001-136] PriorityTier=5  PreemptMode=REQUEUE
PreemptType=preempt/partition_prio
# health: cheap check every 5 min on every node; drains, never reboots
HealthCheckProgram=/etc/slurm/healthcheck.sh
HealthCheckInterval=300
HealthCheckNodeState=ANY
# a dead node must not hold a 128-node job hostage for long
SlurmdTimeout=120
KillWait=30
UnkillableStepTimeout=180
# jobs that lose a node are requeued by the wrapper, not silently retried
JobRequeue=0
ReturnToService=1
Off-the-shelf health checks
  • Many sites use LBNL's Node Health Check (NHC), an open-source framework that Slurm can run as its health-check program, instead of a hand-written script; GPU-specific checks are often added on top, as some GPU clouds document for their own clusters.
    • Gopala starts with his short script because he can read every line of it, and plans to move to NHC when the checks grow.

5. The monitoring stack

The monitoring stack

๐Ÿ“Š GrafanaOne dashboard per question
๐Ÿ—„๏ธ PrometheusTime series and alert rules
๐Ÿ“ค DCGM exporter ยท node exporterMetrics from every node
๐Ÿฉบ NVIDIA DCGMGPU health and counters
๐Ÿ‹๏ธ Training codeStep time, loss, tokens per second

All open source. Step time is the single most useful signal.

Everything is open source and standard:

6. Which GPU errors mean what

His handling table, built from NVIDIA's Xid documentation and kept next to the health check:

XidMeaning (NVIDIA)His action
13, 31, 43Errors usually caused by the application (bad memory access, a kernel fault)No drain; look at the job first
48Double-bit ECC errorDrain; reset and run the diagnostic; RMA if repeated
63, 64Memory row-remapping event / failureDrain; a reset applies the remap; RMA on failure
74NVLink errorDrain; run the single-node all-reduce from Part 6
79GPU has fallen off the busDrain; power-cycle through the BMC; ticket
94, 95Contained / uncontained ECC error94: restart the job; 95: drain and reset

7. Alert rules: who gets woken up

alerts.yml ยท Prometheus alert rules ยท not executed for this article
# alerts.yml โ€” Prometheus alert rules. Only "page" wakes a human.
groups:
- name: maestro-training
  rules:
  - alert: GpuXidError
    expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
    labels: {severity: ticket}
    annotations: {summary: "Xid {{ $value }} on {{ $labels.Hostname }} GPU {{ $labels.gpu }}"}
  - alert: GpuHot
    expr: DCGM_FI_DEV_GPU_TEMP > 85
    for: 10m
    labels: {severity: ticket}
  - alert: StepTimeSlow            # the whole job is slower than its own baseline
    expr: train_step_seconds > 1.2 * avg_over_time(train_step_seconds[6h])
    for: 30m
    labels: {severity: page}
  - alert: StragglerRank           # one node much slower than the median
    expr: max(train_rank_step_seconds) > 1.05 * quantile(0.5, train_rank_step_seconds)
    for: 15m
    labels: {severity: ticket}
  - alert: TwoRestartsInAnHour
    expr: increase(train_restarts_total[1h]) >= 2
    labels: {severity: page}
  - alert: NoHotSpare
    expr: slurm_nodes_idle{partition="main"} < 1
    for: 30m
    labels: {severity: page}
  - alert: LossDiverging
    expr: train_loss > 1.5 * min_over_time(train_loss[2h])
    for: 10m
    labels: {severity: page}

8. A health check that drains bad nodes

healthcheck.sh ยท Slurm HealthCheckProgram ยท not executed for this article
#!/bin/bash
# healthcheck.sh โ€” Slurm HealthCheckProgram, runs every 5 minutes on each node
# (slurm.conf: HealthCheckProgram=/etc/slurm/healthcheck.sh
#              HealthCheckInterval=300  HealthCheckNodeState=ANY)
# Cheap checks only: a node that fails is drained, never rebooted from here.
H=$(hostname -s)
drain() { scontrol update nodename=$H state=drain reason="hc: $1"; exit 0; }

[ "$(nvidia-smi -L | wc -l)" -eq 8 ]            || drain "gpu count"
nvidia-smi -q -d ECC | grep -q "Pending Page Blacklist *: Yes" && drain "ecc retire pending"
dmesg -T --since "-10min" | grep -qE "NVRM: Xid.*(48|63|64|74|79|94|95)" && drain "xid $(dmesg | grep -oE 'Xid[^,]*' | tail -1)"
[ "$(ibstat | grep -c 'State: Active')" -ge 8 ] || drain "ib link down"
df -P /checkpoints >/dev/null 2>&1              || drain "checkpoint fs"
T=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader | sort -n | tail -1)
[ "$T" -lt 87 ]                                 || drain "gpu temp $T"
exit 0

9. See a run unfold

A simulated 31-day run (our model)

10. Replaying the run: spares and bad luck

Hot spares vs goodput, at two failure rates

MTTF 49 h ยท 0 spares90.9%
MTTF 49 h ยท 4 spares97.7%
MTTF 49 h ยท 8 spares97.8%
MTTF 7.9 h ยท 0 spares61.5%
MTTF 7.9 h ยท 4 spares69.0%
MTTF 7.9 h ยท 8 spares85.3%

Real output of restart_sim.py, 200 simulated runs each (our model).

restart_sim.py ยท Python 3, no dependencies
"""restart_sim.py โ€” replay a 31-day run through the failure and restart logic.

Monte Carlo: failures arrive at random (exponential gaps) at a given
cluster MTTF. Each failure loses the work since the last checkpoint and
costs a recovery time that depends on whether a hot spare is free;
failed nodes come back after a repair time. Compares two failure rates:
the one scaled from Llama 3 (Part 7, ~49 h) and the much worse 7.9 h
reported for 1,024-GPU jobs in Meta's research clusters (A100s).
All timings are our assumptions. Fixed seed, so the output is reproducible.
"""
import random

DAYS, CKPT_H, SAVE_H = 31, 1.0, 30 / 3600
RECOVER_SPARE_H, RECOVER_WAIT_H, REPAIR_H = 10 / 60, 4.0, 48.0

def run(mttf_h, spares, seed):
    rng = random.Random(seed)
    t, lost, fails, waits, back = 0.0, 0.0, 0, 0, []
    end = DAYS * 24
    while True:
        gap = rng.expovariate(1 / mttf_h)
        if t + gap >= end:
            break
        t += gap; fails += 1
        back = [b for b in back if b > t]            # repaired nodes rejoin the spare pool
        lost += (t % CKPT_H)                           # work since last checkpoint
        if len(back) < spares:
            rec = RECOVER_SPARE_H
        else:
            rec = RECOVER_WAIT_H; waits += 1           # wait for the provider to swap a node
        back.append(t + REPAIR_H)
        lost += rec; t += rec
    saves = end / CKPT_H * SAVE_H
    return fails, waits, 1 - (lost + saves) / end

def main():
    print(f"{'cluster MTTF':>12s} {'spares':>6s} {'failures':>8s} {'no spare':>8s} {'goodput':>8s}")
    for mttf in (49.0, 7.9):
        for spares in (0, 4, 8):
            res = [run(mttf, spares, s) for s in range(200)]
            f = sum(r[0] for r in res) / len(res); w = sum(r[1] for r in res) / len(res)
            g = sum(r[2] for r in res) / len(res)
            print(f"{mttf:10.1f} h {spares:6d} {f:8.1f} {w:8.1f} {g:8.1%}")

if __name__ == "__main__":
    main()
$ python3 restart_sim.py cluster MTTF spares failures no spare goodput 49.0 h 0 13.7 13.7 90.9% 49.0 h 4 14.8 0.2 97.7% 49.0 h 8 14.8 0.0 97.8% 7.9 h 0 62.2 62.2 61.5% 7.9 h 4 70.0 46.5 69.0% 7.9 h 8 86.9 11.8 85.3%

Two findings shape the design:

11. How often will the main run break?

Goodput by checkpoint interval (10-minute recovery)

every 15 min96.1%
every 30 min97.5%
every 60 min97.8%
every 180 min96.4%

Real output of failure_budget.py (failure rate scaled from Llama 3; our assumptions).

failure_budget.py ยท Python 3, no dependencies
"""failure_budget.py โ€” how often will Maestro-1's run be interrupted?

Scales Meta's published Llama 3 405B failure record (419 unexpected
interruptions in 54 days on 16,384 H100s) to Gopala's cluster, then
asks how checkpoint interval and recovery time change goodput.
Assumes failures scale linearly with GPU count (our assumption).
"""
META = {"gpus": 16384, "days": 54, "unexpected": 419}
GPUS, DAYS = 1024, 31

def main():
    rate = META["unexpected"] / META["gpus"] / META["days"]   # per GPU-day
    n = rate * GPUS * DAYS
    print(f"rate {rate*1e3:.3f} interruptions per 1,000 GPU-days")
    print(f"Maestro-1: {GPUS} GPUs x {DAYS} days -> {n:.1f} expected interruptions "
          f"(one every {DAYS*24/n:.0f} h)")
    print(f"{'ckpt every':>10s} {'recover':>8s} {'lost h':>7s} {'goodput':>8s}")
    for ckpt_min in (15, 30, 60, 180):
        for rec_min in (10, 45):
            # lose half a checkpoint interval of work plus the recovery time
            lost = n * (ckpt_min / 2 + rec_min) / 60
            save = DAYS * 24 * 60 / ckpt_min * 0.5 / 60    # 30 s blocking save, our assumption
            good = 1 - (lost + save) / (DAYS * 24)
            print(f"{ckpt_min:7d} min {rec_min:5d} min {lost+save:7.1f} {good:8.1%}")

if __name__ == "__main__":
    main()
$ python3 failure_budget.py rate 0.474 interruptions per 1,000 GPU-days Maestro-1: 1024 GPUs x 31 days -> 15.0 expected interruptions (one every 49 h) ckpt every recover lost h goodput 15 min 10 min 29.2 96.1% 15 min 45 min 38.0 94.9% 30 min 10 min 18.7 97.5% 30 min 45 min 27.4 96.3% 60 min 10 min 16.2 97.8% 60 min 45 min 25.0 96.6% 180 min 10 min 27.1 96.4% 180 min 45 min 35.9 95.2%

Reading it:

12. Making checkpoints cheap

Three implications for Gopala:

13. The serving cluster on Kubernetes

14. The fire drill

The recovery chain tested in the fire drill

๐ŸฉบHealth checkDrains the node
๐Ÿ’ฅJob failsWithin ~2 min
๐Ÿ”„WrapperNew allocation, excludes drained node
๐Ÿ†•Hot spareJoins the job
โ–ถ๏ธResumeFrom last checkpoint

Scenario: 41 minutes on the first drill, 11 after fixes.

Scenario
  • The first drill takes 41 minutes end to end: the job waits 25 minutes for Slurm to give up on the dead node.
    • Shortening the job's timeout and having the wrapper exclude drained nodes explicitly brings the second drill down to 11 minutes.
    • These timings are the story's, not measurements.

15. The runbook

Example, "two restarts in an hour":

  1. Means: the automatic path is failing, or one bad node keeps coming back.
  2. Confirm: list the nodes involved in both failures; check drain reasons and Xid codes.
  3. First action: exclude any node involved in both; restart from the last good checkpoint.
  4. Escalate: if a third restart follows within an hour, open a severity-1 ticket with the provider and stop the job.
  5. Record: incident note in git, Xid table updated if a new code appeared.

16. On call, with one person

Gopala cannot be awake for 31 days. The rules:

17. Seven cluster-software mistakes

  1. Letting the health check reboot nodes, so a flapping node keeps rejoining the job.
  2. Watching GPU utilisation instead of step time; a straggler keeps every GPU "busy" waiting.
  3. Paging on every Xid, until the on-call person stops reading pages.
  4. Default timeouts that let a dead node hold a 128-node job for half an hour.
  5. Never testing a restart until the main run needs one.
  6. Checkpoints that are written but never test-read.
  7. Changing configuration during the run without a commit.

18. Everything in git

19. Delegation box

๐Ÿ™‹ Gopala himself
  • The scheduler design and partitions
  • The health-check rules and paging rules
  • The fire drill
๐Ÿค– AI agents
  • First incident summaries from logs (read-only)
  • Drafting provider tickets
  • Writing dashboard queries and alert rules for review
๐Ÿค Bought, rented or borrowed
  • The provider's managed Slurm and their on-site staff
  • Open-source tools: Slurm, Kubernetes, DCGM, Prometheus, Grafana
  • A contractor sharing nights during the main run

20. Budget and risk tracker

ItemStatus after Part 7Note
Spend to date~$1.1M (scenario)Training block billing since acceptance, serving cluster from month 4
Cluster128 nodes in service, 8 hot sparesMaestro-0 post-training running
Time20 of ~78 weeks (month ~4.5)Phase 1 complete
Recovery riskTestedRecovery in ~11 minutes in the drill (scenario)
Silent slowdown riskMediumPer-rank step-time alerts
Key-person riskHighRunbook and a second on-call for the main run

21. The series

The Maestro: 18 parts in five phases

Verdict
  • Phase 1 ends with a machine that runs itself most of the time: Slurm for training, Kubernetes for serving, health checks that drain bad nodes, hot spares, hourly checkpoints and a tested 11-minute recovery.

22. Reading the numbers

Primary / standard
Llama 3 405B: 419 unexpected interruptions in 54 days on 16,384 H100s; >90% effective training time (Meta). Slurm HealthCheckProgram and node drain (SchedMD docs). Xid error meanings (NVIDIA Xid documentation). DCGM exporter metric names (NVIDIA). Kueue (Kubernetes SIG Scheduling). 1,024-GPU job MTTF ~7.9 h in Meta research clusters (arXiv:2410.21680). Checkpoint times ~30 min โ†’ <5 min with distributed checkpointing, ~6โ€“14 s visible pause with async for 7Bโ€“13B models (PyTorch blog). LBNL NHC.
Secondary โš ๏ธ
None in this part.
Our assumptions
Linear scaling of failures with GPU count, the simulation's recovery (10 min with a spare, 4 h without) and 48 h repair times, 30-second blocking save, recovery times, the drill timings, partitions, spares and paging rules.