Day Zero: Bring-Up
The 128 nodes arrive in month 3. On paper they are identical; in practice some will have a loose cable, a GPU with failing memory, an old firmware version or a fabric port running at half speed. Any one of them can slow or stop a job that uses all 1,024 GPUs. Bring-up is the work of finding them before the training run does. This part is Gopala's day-zero runbook: what to check on each node, how to test the network in pairs and as a whole, how to read NCCL bandwidth numbers, a small parser that flags bad pairs, and the acceptance report that starts the bill.
burnin.sh needs a real GPU node and was not executed for this article. nccl_check.py was run, but on a sample log written for this article in the nccl-tests format, not output from a real cluster. Failure rates come from Meta's Llama 3 paper. Defect counts in the story are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 13–18
- The provider delivers a quarter of the block early (Part 4). Gopala runs the per-node burn-in (section 6) and parks every failing node.
- Runs pairwise and full-fabric NCCL tests (sections 5–7) as more nodes arrive; finds and reports bad links.
- The full 128 nodes pass the contract acceptance test (Part 4) after replacements; billing starts. Maestro-0 post-training begins on the first accepted nodes.
2. Why trust nothing
Why trust nothing: Llama 3 405B, 54 days
Meta, The Llama 3 Herd of Models (16,384 H100s).
- Large training runs fail mostly because of hardware.
- Meta's Llama 3 paper is the clearest public record: in a 54-day snapshot of pre-training Llama 3 405B on 16,384 H100s, there were 466 job interruptions, 419 of them unexpected.
- Faulty GPUs caused 30.1% of the unexpected ones and GPU HBM3 memory 17.2%
- network switches and cables, host maintenance, SRAM, system processors and others made up the rest.
- Meta still reached more than 90% effective training time, because it found and replaced bad parts fast.
- Gopala's cluster is 16 times smaller, so he should expect roughly 16 times fewer failures per day (Part 7 does the arithmetic), but the first weeks are the worst: parts that are going to fail early ("infant mortality") do so under the first sustained load.
- Bring-up means applying that load deliberately, before his job depends on it.
- The site's Who Runs the AI Factory:
- bring-up and firmware
- this part is the hands-on version for one person and 128 nodes.
3. The bring-up plan, week by week
- Bring-up overlaps delivery: the first nodes are tested while the rest are still being racked.
Gopala's plan, with the gate that ends each step:
| Week | Step | Gate to move on |
|---|---|---|
| 13 | BMC inventory of the first 32 nodes over Redfish; firmware drift report | All nodes on golden firmware or flagged |
| 13–14 | Golden image installed; per-node burn-in in batches of 16 | Each node passes or is parked with a reason |
| 14 | Storage mounts and a parallel write test | Stated bandwidth reached from all nodes at once |
| 14–16 | Pairwise NCCL tests as racks arrive; leaf-switch tests | No pair below 90% of the best |
| 16–17 | Full-fabric all-reduce on all accepted nodes | Within 10% of the contract figure |
| 17–18 | 72-hour run of his own training job on the full block | No hardware stop; step time stable |
| 18 | Acceptance report signed by both sides | Billing starts |
4. What gets checked, layer by layer
Bring-up, from the motherboard up
Each layer is tested only after the one below it passes.
| Layer | Check | Tool | Typical defect |
|---|---|---|---|
| Inventory | 8 GPUs, right model, memory, serial numbers recorded | nvidia-smi, BMC (Redfish API) | A GPU missing from the bus |
| Firmware | Same VBIOS, NIC firmware, BIOS and BMC on every node | nvidia-smi -q, ibstat | One node with older firmware behaving differently |
| Topology | GPUs on NVSwitch; each NIC close to its GPU on PCIe | nvidia-smi topo -m | NIC in the wrong slot halves bandwidth |
| GPU health | Memory, PCIe, NVLink and compute stress | dcgmi diag -r 3 (NVIDIA DCGM) | HBM errors, a weak GPU |
| Thermals | Sustained full load for hours without throttling | DCGM level 4 or a burn tool | Fan or airflow fault |
| Fabric links | 8 active links at 400 Gb/s | ibstat, ib_write_bw | Port trained at lower speed, dirty optic |
| Collective bandwidth | All-reduce in one node, in pairs, then all nodes | nccl-tests all_reduce_perf | A slow pair, a bad switch port |
| Storage | Write bandwidth from all nodes at once | fio, IOR | A misconfigured client mount |
5. Before the OS: inventory through the BMC
- Every server has a baseboard management controller (BMC): a small computer on the motherboard that can power the node on and off, read its sensors and report its hardware and firmware, even when no operating system is installed.
- Modern BMCs speak Redfish, the DMTF's standard REST API, so one script can inventory servers from different vendors.
Gopala uses it on day zero to compare every node against the expected configuration before spending an hour installing anything:
"""redfish_inventory.py — inventory nodes through their BMCs (DMTF Redfish)
and flag firmware drift before anything boots an OS.
Redfish is the standard REST API of server baseboard management
controllers: /redfish/v1/Systems, /Chassis, /UpdateService/FirmwareInventory.
Run with no arguments, it starts a small MOCK BMC (three fake nodes) in a
thread and inventories that, so the code path can be checked without
hardware. On a real cluster: python3 redfish_inventory.py bmc1 bmc2 ...
(add authentication and TLS verification for real BMCs).
"""
import json, sys, threading, urllib.request
from collections import Counter
from http.server import BaseHTTPRequestHandler, HTTPServer
MOCK = { # node -> (serial, BIOS, BMC firmware, GPU VBIOS, NIC firmware, health)
"node001": ("SN-1001", "1.4.2", "24.09", "96.00.99.00.01", "28.42.1000", "OK"),
"node002": ("SN-1002", "1.4.2", "24.09", "96.00.99.00.01", "28.42.1000", "OK"),
"node003": ("SN-1003", "1.3.7", "24.09", "96.00.99.00.01", "28.39.1002", "Warning"),
}
class BMC(BaseHTTPRequestHandler):
def log_message(self, *a): pass
def do_GET(self):
node, path = self.path.split("/", 2)[1], "/" + self.path.split("/", 2)[2]
sn, bios, bmc, vb, nic, health = MOCK[node]
if path == "/redfish/v1/Systems/1":
body = {"SerialNumber": sn, "BiosVersion": bios, "Status": {"Health": health},
"ProcessorSummary": {"Count": 2}, "MemorySummary": {"TotalSystemMemoryGiB": 2048}}
elif path == "/redfish/v1/UpdateService/FirmwareInventory":
body = {"Members": [{"Id": "BMC", "Version": bmc}, {"Id": "GPU_VBIOS", "Version": vb},
{"Id": "NIC", "Version": nic}]}
else:
self.send_response(404); self.end_headers(); return
data = json.dumps(body).encode()
self.send_response(200); self.send_header("Content-Type", "application/json")
self.end_headers(); self.wfile.write(data)
def get(base, path):
with urllib.request.urlopen(base + path, timeout=5) as r:
return json.load(r)
def inventory(base):
s = get(base, "/redfish/v1/Systems/1")
fw = {m["Id"]: m["Version"] for m in get(base, "/redfish/v1/UpdateService/FirmwareInventory")["Members"]}
return {"serial": s["SerialNumber"], "health": s["Status"]["Health"], "BIOS": s["BiosVersion"], **fw}
def main():
if sys.argv[1:]:
targets = {h: f"https://{h}" for h in sys.argv[1:]}
else:
srv = HTTPServer(("127.0.0.1", 0), BMC)
threading.Thread(target=srv.serve_forever, daemon=True).start()
targets = {n: f"http://127.0.0.1:{srv.server_port}/{n}" for n in MOCK}
inv = {n: inventory(b) for n, b in targets.items()}
keys = ["BIOS", "BMC", "GPU_VBIOS", "NIC"]
print(f"{'node':8s} {'serial':8s} {'health':8s} " + " ".join(f"{k:15s}" for k in keys))
for n, r in inv.items():
print(f"{n:8s} {r['serial']:8s} {r['health']:8s} " + " ".join(f"{r[k]:15s}" for k in keys))
for k in keys:
golden, _ = Counter(r[k] for r in inv.values()).most_common(1)[0]
odd = [n for n, r in inv.items() if r[k] != golden]
if odd:
print(f"DRIFT {k}: golden {golden}, different on {', '.join(odd)}")
sick = [n for n, r in inv.items() if r["health"] != "OK"]
print("hold back before OS install:", ", ".join(sorted(set(sick + [n for k in keys for n, r in inv.items()
if r[k] != Counter(x[k] for x in inv.values()).most_common(1)[0][0]]))) or "none")
if __name__ == "__main__":
main()- The run above is real but the "BMC" is a small mock server the script starts itself, with node003 deliberately given an older BIOS and NIC firmware and a warning status, so the drift logic can be seen working.
- Against real BMCs the same code reads the same paths (
/redfish/v1/Systemsand/redfish/v1/UpdateService/FirmwareInventory), with authentication and certificate checks added- exact resource names vary by vendor, so the first real run is checked by hand.
- Against real BMCs the same code reads the same paths (
- Why bother before the OS?
- Because firmware drift is cheap to fix before a node is in service and expensive after: a node with an older NIC firmware can pass every single-node test and only misbehave in a 1,024-GPU collective, where it is hardest to find.
6. The per-node burn-in
- One script per node, run in parallel across the delivered nodes.
- Every step either passes or keeps the node out of the pool; there is no "mostly fine".
- Logs go to a shared directory so every result is kept with the node's serial numbers.
#!/bin/bash
# burnin.sh NODE — day-zero acceptance for one 8-GPU node.
# Every step must pass; any failure keeps the node out of the pool.
set -euo pipefail
N=$1; LOG=/acceptance/$N; mkdir -p $LOG
run() { echo "== $1"; ssh $N "$2" > $LOG/$1.log 2>&1 || { echo "FAIL $1 on $N"; exit 1; }; }
run inventory "nvidia-smi --query-gpu=index,serial,vbios_version,memory.total --format=csv"
run firmware "nvidia-smi -q | grep -E 'VBIOS|Inforom' ; ibstat | grep -E 'Firmware|Rate'"
run topology "nvidia-smi topo -m" # 8 GPUs on NVSwitch, NICs on the right PCIe roots
run ecc "nvidia-smi -q -d ECC | grep -E 'Pending|Uncorr'"
run dcgm_long "dcgmi diag -r 3" # memory, PCIe, NVLink, stress (~30-60 min)
run ib_links "ibstat | grep -c 'State: Active'" # expect one per compute NIC
run nccl_local "all_reduce_perf -b 8 -e 8G -f 2 -g 8" # NVLink only, single node
run burn_24h "dcgmi diag -r 4 || gpu_burn 86400" # thermals under sustained load
grep -q "Fail" $LOG/dcgm_long.log && { echo "FAIL dcgm on $N"; exit 1; }
echo "PASS $N"Notes on the choices:
- DCGM level 3 is NVIDIA's long diagnostic (memory, PCIe bandwidth, NVLink, targeted stress); level 4 adds longer tests. They take from tens of minutes to hours, which is why nodes are tested in parallel.
- The single-node all-reduce checks NVLink and NVSwitch inside the node before the network is involved, so a bad result points to the node, not the fabric.
- The 24-hour burn exists to trigger early failures and thermal problems. It is expensive in time, but it is cheaper than losing a day of a 1,024-GPU run.
7. Reading NCCL numbers
- nccl-tests prints two bandwidths.
- algbw is data size divided by time.
- busbw rescales it so the number can be compared with the hardware's link speed, whatever the number of GPUs: for all-reduce, busbw = algbw × 2(n−1)/n, where n is the number of ranks.
- For 16 GPUs (two nodes) the factor is 1.875; for 1,024 GPUs it is almost exactly 2.
- What is "good"?
- Inside one H100 node, NVLink gives each GPU up to 900 GB/s total.
- Between nodes, each GPU has its own 400 Gb/s NIC, i.e. 50 GB/s per direction, so the inter-node bus bandwidth for large messages is bounded by the network, not NVLink.
- The exact achievable figure depends on the NCCL version, topology and settings, so Gopala does not compare against a number from the internet: he compares nodes with each other, and the trial cluster with the contract figure.
- A pair far below its peers is a defect, whatever the absolute number.
8. Watching 128 nodes at once
Node states during bring-up (scenario, week 16)
■ accepted ■ testing ■ parked ■ not yet delivered · tap a node
- One state per node, kept in a file, drives everything else: which nodes the scheduler may use, which tickets are open with the provider, and the acceptance report.
- The grid above is the story's state, drawn from the defects described below, not real data.
9. Testing in pairs, and a parser
Pairwise NCCL bus bandwidth (sample log)
Real output of nccl_check.py on a sample log written for this article, not a real cluster.
- A full-cluster test tells him whether the whole is slow, not where.
- So he first runs 2-node tests on all pairs in a pattern (node 1 with 2, 3 with 4, and so on, then shifted), and the parser below flags any pair below 90% of the best.
- A node that is in two flagged pairs is the suspect; it is drained and tested alone.
"""nccl_check.py — read nccl-tests all_reduce_perf output and flag bad nodes.
Usage: python3 nccl_check.py [log ...]. With no file it parses SAMPLE,
a log written for this article in the nccl-tests format (not from a real
cluster) so the parser and the maths can be checked.
"""
import re, sys
# Pairwise 2-node runs (16 GPUs): node pair -> busbw at 8 GB messages.
SAMPLE = """\
# pair node001,node002
8589934592 2147483648 float sum -1 45210 190.00 356.25 0
# pair node003,node004
8589934592 2147483648 float sum -1 45480 188.87 354.13 0
# pair node005,node006
8589934592 2147483648 float sum -1 98120 87.54 164.14 0
# pair node007,node008
8589934592 2147483648 float sum -1 45390 189.24 354.83 0
"""
ROW = re.compile(r"^\s*(\d+)\s+(\d+)\s+\w+\s+\w+\s+-?\d+\s+([\d.]+)\s+([\d.]+)\s+([\d.]+)\s+(\d+)")
def busbw_factor(n):
# all-reduce moves 2(n-1)/n of the data per rank (nccl-tests PERFORMANCE.md)
return 2 * (n - 1) / n
def parse(text):
out, pair = [], None
for line in text.splitlines():
if line.startswith("# pair"):
pair = line.split()[2]
m = ROW.match(line)
if m and pair:
size, t_us, alg, bus, wrong = int(m[1]), float(m[3]), float(m[4]), float(m[5]), int(m[6])
out.append((pair, size, t_us, alg, bus, wrong))
return out
def main():
text = "".join(open(f).read() for f in sys.argv[1:]) if sys.argv[1:] else SAMPLE
rows = parse(text)
n = 16
f = busbw_factor(n)
print(f"ranks={n}: busbw = algbw x {f:.4f}")
best = max(r[4] for r in rows)
for pair, size, t, alg, bus, wrong in rows:
alg_calc = size / (t * 1e-6) / 1e9
flag = "OK" if bus >= 0.9 * best and wrong == 0 else "SUSPECT"
print(f"{pair:18s} algbw {alg:6.2f} (recalc {alg_calc:6.2f}) busbw {bus:6.2f} GB/s "
f"x{f:.4f}={alg*f:6.2f} {flag}")
bad = [r[0] for r in rows if r[4] < 0.9 * best]
print("drain and re-test individually:", ", ".join(bad) if bad else "none")
if __name__ == "__main__":
main()- The parser also recomputes algbw from the message size and time and the busbw factor from the rank count, so a misread column or a mislabelled run shows up immediately.
- On a real cluster it reads one log per pair:
python3 nccl_check.py logs/pair_*.txt.
- On a real cluster it reads one log per pair:
- After the pairs pass, the full 1,024-GPU run is the final test.
- Its bus bandwidth should be close to the pair results
- if it falls well below, the problem is usually in the higher switch layers (a congested or mis-cabled spine), which the provider has to fix.
- Its bus bandwidth should be close to the pair results
10. What bring-up found
Node life cycle
A repaired node always goes back to Testing, never straight into service.
- Across 128 nodes, bring-up parks 9 nodes: 3 with GPUs failing DCGM memory tests, 2 with a fabric port trained at reduced speed (a reseated optic fixes one, a replaced cable the other), 1 with a NIC in the wrong PCIe slot after a repair, 2 with an older firmware version, and 1 that throttles under the 24-hour burn because of a blocked airflow path.
- All are fixed or replaced by the provider within the contract's replacement window.
- These counts are made up for the story; Meta's real record (section 2) shows the kind of mix to expect, not the number.
Two practices make this manageable for one person:
- One state per node: new → testing → accepted → in service, or parked with a reason. A spreadsheet works; Part 7 moves it into the scheduler's drain reasons.
- Never repair in place: a fixed node goes back to "testing" and runs the full burn-in again. Many second failures come from the repair itself.
11. Testing storage before the run depends on it
- Checkpoints are the main storage load of a training run: every node writes its share of model and optimiser state at the same moment.
- If storage cannot absorb it, all 1,024 GPUs wait.
- Gopala tests the parallel file system with the standard tools
fioand IOR, from all nodes at once, writing files of the size a checkpoint shard will have, and records three numbers: aggregate write bandwidth, time to write one full checkpoint, and read bandwidth for a restart (which matters just as much, because a restart reads everything back).
- He also checks the slow path: what happens to bandwidth when one storage server is taken out of service, because that is exactly the condition under which he will be restarting after a failure.
- The checkpoint size itself is decided in Part 9; the policy of how often to save is in Part 7.
12. Firmware: when to upgrade, and when not to
- Meta's Llama 3 paper notes that a number of its planned interruptions were firmware upgrades, and the site's bring-up article describes how much of the work at hyperscale is firmware.
Gopala's rules for a 128-node cluster:
- One golden version of every firmware, recorded with the image.
- Upgrade only for a known problem or a security fix, never because a new version exists.
- Upgrade 4 nodes, run the full burn-in on them, run a 2-day job on them, then roll out in batches between jobs.
- No upgrades from month 8 to the end of the main run.
13. The golden image
- Every accepted node runs the same software image: OS, kernel, GPU driver, CUDA, NCCL, fabric drivers, DCGM, container runtime.
- Versions are pinned and recorded, and a node whose versions differ is treated as untested.
- Gopala keeps the image definition in git alongside his training code, so a later bug can always be traced to the exact versions it ran on.
- Upgrades happen in a rolling way on a few nodes first, and never during the main run (the same freeze as Part 5).
14. Handing over to the scheduler
- Bring-up ends when every accepted node is handed to the cluster software with three things attached: its baseline numbers (bandwidths, temperatures, step time on a reference job), its firmware and image versions, and its history (which tests it failed and what was replaced).
- Part 7's health checks compare live readings with those baselines, so "node 57 is slow"
- becomes "node 57 is 6% slower than it was on day zero", which is actionable.
- Part 7's health checks compare live readings with those baselines, so "node 57 is slow"
15. Six bring-up mistakes
- Running only the full-cluster test, which says the cluster is slow but not where.
- Accepting nodes that passed a short diagnostic but never ran under sustained load.
- Putting repaired nodes straight back into service without the full burn-in.
- Letting firmware versions drift because each repair brought a different part.
- Not testing storage restart reads, only writes.
- Keeping results in chat threads instead of one file per node.
16. The acceptance report
- The report that starts billing is a short table, generated from the logs: per node, the serial numbers, firmware versions, DCGM result, burn-in result, link count and speed, and pair bandwidth; then the full-cluster all-reduce result, the storage test and the 72-hour job.
- The provider signs it too.
- It becomes the baseline: whenever a node misbehaves later, Part 7's health checks compare it with its own day-zero numbers.
17. Delegation box
- The burn-in and acceptance criteria
- Deciding which nodes to park
- Signing the acceptance report
- Running tests in parallel and collecting logs
- Summarising failures and drafting tickets for the provider
- Comparing firmware versions across nodes
- The provider's on-site team (swaps, cables, optics)
- NVIDIA's open tools: DCGM, nccl-tests
- A contractor with InfiniBand experience for two weeks
18. Budget and risk tracker
| Item | Status after Part 6 | Note |
|---|---|---|
| Spend to date | ~$0.5M (scenario) | Billing starts on accepted nodes; partial block from month 3 |
| Nodes | 128 accepted, 9 replaced during bring-up (scenario) | Baseline numbers recorded per node |
| Time | 18 of ~78 weeks | Maestro-0 post-training under way |
| Hardware risk | Reduced | Infant-mortality failures found before the main run |
| Fabric risk | Medium | Spine-layer congestion only shows under full load |
| Key-person risk | High | Bring-up knowledge written down as scripts |
19. The series
The Maestro: 18 parts in five phases
- Day zero is about distrust: every node is tested alone, every link in pairs, the whole fabric together, and only then accepted and billed.
- The parked nodes are the ones that would have stopped the run.
- Next, Part 7: the software that keeps 1,024 GPUs busy and catches failures on its own.
20. Reading the numbers
- Primary / standard
- Llama 3 405B: 16,384 H100s, 466 interruptions in 54 days, 419 unexpected; faulty GPUs 30.1% and HBM3 17.2% of unexpected; >90% effective training time (Meta). busbw = algbw × 2(n−1)/n for all-reduce (nccl-tests PERFORMANCE.md). H100 NVLink 900 GB/s; ConnectX-7 400 Gb/s (NVIDIA). DCGM diagnostic levels (NVIDIA DCGM docs).
- Secondary ⚠️
- None in this part.
- Our assumptions
- The sample NCCL log, the 90% threshold, the 24-hour burn length and every defect count in the story.