Home › Blog › The Maestro · Part 6
The Maestro · Part 6 · Phase 1 · 🏗️ Infrastructure hat

Day Zero: Bring-Up

The 128 nodes arrive in month 3. On paper they are identical; in practice some will have a loose cable, a GPU with failing memory, an old firmware version or a fabric port running at half speed. Any one of them can slow or stop a job that uses all 1,024 GPUs. Bring-up is the work of finding them before the training run does. This part is Gopala's day-zero runbook: what to check on each node, how to test the network in pairs and as a whole, how to read NCCL bandwidth numbers, a small parser that flags bad pairs, and the acceptance report that starts the bill.

FL
FrontierAGI Team
🏗️ Infrastructure🔧 Hardware tester💻 Coder
⚠️ How to read this (as of 30 Oct 2026) Commands and tools (nvidia-smi, DCGM, ibstat, nccl-tests) are real and documented by NVIDIA; the busbw formula is from nccl-tests' own documentation. burnin.sh needs a real GPU node and was not executed for this article. nccl_check.py was run, but on a sample log written for this article in the nccl-tests format, not output from a real cluster. Failure rates come from Meta's Llama 3 paper. Defect counts in the story are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 13–18

Week 13 · First 32 nodes
  • The provider delivers a quarter of the block early (Part 4). Gopala runs the per-node burn-in (section 6) and parks every failing node.
Weeks 14–16 · Network
  • Runs pairwise and full-fabric NCCL tests (sections 5–7) as more nodes arrive; finds and reports bad links.
Weeks 17–18 · Acceptance
  • The full 128 nodes pass the contract acceptance test (Part 4) after replacements; billing starts. Maestro-0 post-training begins on the first accepted nodes.

2. Why trust nothing

Why trust nothing: Llama 3 405B, 54 days

466job interruptions
419unexpected
30.1%of unexpected: faulty GPUs
17.2%of unexpected: GPU HBM3 memory

Meta, The Llama 3 Herd of Models (16,384 H100s).

3. The bring-up plan, week by week

Gopala's plan, with the gate that ends each step:

WeekStepGate to move on
13BMC inventory of the first 32 nodes over Redfish; firmware drift reportAll nodes on golden firmware or flagged
13–14Golden image installed; per-node burn-in in batches of 16Each node passes or is parked with a reason
14Storage mounts and a parallel write testStated bandwidth reached from all nodes at once
14–16Pairwise NCCL tests as racks arrive; leaf-switch testsNo pair below 90% of the best
16–17Full-fabric all-reduce on all accepted nodesWithin 10% of the contract figure
17–1872-hour run of his own training job on the full blockNo hardware stop; step time stable
18Acceptance report signed by both sidesBilling starts

4. What gets checked, layer by layer

Bring-up, from the motherboard up

📋 InventoryBMC / Redfish, nvidia-smi
🧬 FirmwareSame versions everywhere
🗺️ TopologyGPUs on NVSwitch, NICs near GPUs
🩺 GPU healthDCGM level 3
🔥 Thermals24 h burn-in
🔗 Links8 × 400 Gb/s active
📶 CollectivesNCCL in node, pairs, all
💾 Storagefio / IOR from all nodes

Each layer is tested only after the one below it passes.

LayerCheckToolTypical defect
Inventory8 GPUs, right model, memory, serial numbers recordednvidia-smi, BMC (Redfish API)A GPU missing from the bus
FirmwareSame VBIOS, NIC firmware, BIOS and BMC on every nodenvidia-smi -q, ibstatOne node with older firmware behaving differently
TopologyGPUs on NVSwitch; each NIC close to its GPU on PCIenvidia-smi topo -mNIC in the wrong slot halves bandwidth
GPU healthMemory, PCIe, NVLink and compute stressdcgmi diag -r 3 (NVIDIA DCGM)HBM errors, a weak GPU
ThermalsSustained full load for hours without throttlingDCGM level 4 or a burn toolFan or airflow fault
Fabric links8 active links at 400 Gb/sibstat, ib_write_bwPort trained at lower speed, dirty optic
Collective bandwidthAll-reduce in one node, in pairs, then all nodesnccl-tests all_reduce_perfA slow pair, a bad switch port
StorageWrite bandwidth from all nodes at oncefio, IORA misconfigured client mount

5. Before the OS: inventory through the BMC

Gopala uses it on day zero to compare every node against the expected configuration before spending an hour installing anything:

redfish_inventory.py · Python 3, no dependencies; runs against a built-in mock BMC
"""redfish_inventory.py — inventory nodes through their BMCs (DMTF Redfish)
and flag firmware drift before anything boots an OS.

Redfish is the standard REST API of server baseboard management
controllers: /redfish/v1/Systems, /Chassis, /UpdateService/FirmwareInventory.
Run with no arguments, it starts a small MOCK BMC (three fake nodes) in a
thread and inventories that, so the code path can be checked without
hardware. On a real cluster: python3 redfish_inventory.py bmc1 bmc2 ...
(add authentication and TLS verification for real BMCs).
"""
import json, sys, threading, urllib.request
from collections import Counter
from http.server import BaseHTTPRequestHandler, HTTPServer

MOCK = {  # node -> (serial, BIOS, BMC firmware, GPU VBIOS, NIC firmware, health)
    "node001": ("SN-1001", "1.4.2", "24.09", "96.00.99.00.01", "28.42.1000", "OK"),
    "node002": ("SN-1002", "1.4.2", "24.09", "96.00.99.00.01", "28.42.1000", "OK"),
    "node003": ("SN-1003", "1.3.7", "24.09", "96.00.99.00.01", "28.39.1002", "Warning"),
}

class BMC(BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        node, path = self.path.split("/", 2)[1], "/" + self.path.split("/", 2)[2]
        sn, bios, bmc, vb, nic, health = MOCK[node]
        if path == "/redfish/v1/Systems/1":
            body = {"SerialNumber": sn, "BiosVersion": bios, "Status": {"Health": health},
                    "ProcessorSummary": {"Count": 2}, "MemorySummary": {"TotalSystemMemoryGiB": 2048}}
        elif path == "/redfish/v1/UpdateService/FirmwareInventory":
            body = {"Members": [{"Id": "BMC", "Version": bmc}, {"Id": "GPU_VBIOS", "Version": vb},
                                {"Id": "NIC", "Version": nic}]}
        else:
            self.send_response(404); self.end_headers(); return
        data = json.dumps(body).encode()
        self.send_response(200); self.send_header("Content-Type", "application/json")
        self.end_headers(); self.wfile.write(data)

def get(base, path):
    with urllib.request.urlopen(base + path, timeout=5) as r:
        return json.load(r)

def inventory(base):
    s = get(base, "/redfish/v1/Systems/1")
    fw = {m["Id"]: m["Version"] for m in get(base, "/redfish/v1/UpdateService/FirmwareInventory")["Members"]}
    return {"serial": s["SerialNumber"], "health": s["Status"]["Health"], "BIOS": s["BiosVersion"], **fw}

def main():
    if sys.argv[1:]:
        targets = {h: f"https://{h}" for h in sys.argv[1:]}
    else:
        srv = HTTPServer(("127.0.0.1", 0), BMC)
        threading.Thread(target=srv.serve_forever, daemon=True).start()
        targets = {n: f"http://127.0.0.1:{srv.server_port}/{n}" for n in MOCK}
    inv = {n: inventory(b) for n, b in targets.items()}
    keys = ["BIOS", "BMC", "GPU_VBIOS", "NIC"]
    print(f"{'node':8s} {'serial':8s} {'health':8s} " + " ".join(f"{k:15s}" for k in keys))
    for n, r in inv.items():
        print(f"{n:8s} {r['serial']:8s} {r['health']:8s} " + " ".join(f"{r[k]:15s}" for k in keys))
    for k in keys:
        golden, _ = Counter(r[k] for r in inv.values()).most_common(1)[0]
        odd = [n for n, r in inv.items() if r[k] != golden]
        if odd:
            print(f"DRIFT {k}: golden {golden}, different on {', '.join(odd)}")
    sick = [n for n, r in inv.items() if r["health"] != "OK"]
    print("hold back before OS install:", ", ".join(sorted(set(sick + [n for k in keys for n, r in inv.items()
          if r[k] != Counter(x[k] for x in inv.values()).most_common(1)[0][0]]))) or "none")

if __name__ == "__main__":
    main()
$ python3 redfish_inventory.py # three mock nodes, not real hardware node serial health BIOS BMC GPU_VBIOS NIC node001 SN-1001 OK 1.4.2 24.09 96.00.99.00.01 28.42.1000 node002 SN-1002 OK 1.4.2 24.09 96.00.99.00.01 28.42.1000 node003 SN-1003 Warning 1.3.7 24.09 96.00.99.00.01 28.39.1002 DRIFT BIOS: golden 1.4.2, different on node003 DRIFT NIC: golden 28.42.1000, different on node003 hold back before OS install: node003

6. The per-node burn-in

burnin.sh · needs a real 8-GPU node · not executed for this article
#!/bin/bash
# burnin.sh NODE — day-zero acceptance for one 8-GPU node.
# Every step must pass; any failure keeps the node out of the pool.
set -euo pipefail
N=$1; LOG=/acceptance/$N; mkdir -p $LOG
run() { echo "== $1"; ssh $N "$2" > $LOG/$1.log 2>&1 || { echo "FAIL $1 on $N"; exit 1; }; }

run inventory  "nvidia-smi --query-gpu=index,serial,vbios_version,memory.total --format=csv"
run firmware   "nvidia-smi -q | grep -E 'VBIOS|Inforom' ; ibstat | grep -E 'Firmware|Rate'"
run topology   "nvidia-smi topo -m"                 # 8 GPUs on NVSwitch, NICs on the right PCIe roots
run ecc        "nvidia-smi -q -d ECC | grep -E 'Pending|Uncorr'"
run dcgm_long  "dcgmi diag -r 3"                    # memory, PCIe, NVLink, stress (~30-60 min)
run ib_links   "ibstat | grep -c 'State: Active'"   # expect one per compute NIC
run nccl_local "all_reduce_perf -b 8 -e 8G -f 2 -g 8"   # NVLink only, single node
run burn_24h   "dcgmi diag -r 4 || gpu_burn 86400"      # thermals under sustained load
grep -q "Fail" $LOG/dcgm_long.log && { echo "FAIL dcgm on $N"; exit 1; }
echo "PASS $N"

Notes on the choices:

7. Reading NCCL numbers

8. Watching 128 nodes at once

Node states during bring-up (scenario, week 16)

■ accepted ■ testing ■ parked ■ not yet delivered · tap a node

9. Testing in pairs, and a parser

Pairwise NCCL bus bandwidth (sample log)

node001 + node002356 GB/s
node003 + node004354 GB/s
node005 + node006164 GB/s ✗
node007 + node008355 GB/s

Real output of nccl_check.py on a sample log written for this article, not a real cluster.

nccl_check.py · Python 3, no dependencies
"""nccl_check.py — read nccl-tests all_reduce_perf output and flag bad nodes.

Usage: python3 nccl_check.py [log ...]. With no file it parses SAMPLE,
a log written for this article in the nccl-tests format (not from a real
cluster) so the parser and the maths can be checked.
"""
import re, sys

# Pairwise 2-node runs (16 GPUs): node pair -> busbw at 8 GB messages.
SAMPLE = """\
# pair node001,node002
  8589934592    2147483648     float     sum      -1  45210  190.00  356.25      0
# pair node003,node004
  8589934592    2147483648     float     sum      -1  45480  188.87  354.13      0
# pair node005,node006
  8589934592    2147483648     float     sum      -1  98120   87.54  164.14      0
# pair node007,node008
  8589934592    2147483648     float     sum      -1  45390  189.24  354.83      0
"""
ROW = re.compile(r"^\s*(\d+)\s+(\d+)\s+\w+\s+\w+\s+-?\d+\s+([\d.]+)\s+([\d.]+)\s+([\d.]+)\s+(\d+)")

def busbw_factor(n):
    # all-reduce moves 2(n-1)/n of the data per rank (nccl-tests PERFORMANCE.md)
    return 2 * (n - 1) / n

def parse(text):
    out, pair = [], None
    for line in text.splitlines():
        if line.startswith("# pair"):
            pair = line.split()[2]
        m = ROW.match(line)
        if m and pair:
            size, t_us, alg, bus, wrong = int(m[1]), float(m[3]), float(m[4]), float(m[5]), int(m[6])
            out.append((pair, size, t_us, alg, bus, wrong))
    return out

def main():
    text = "".join(open(f).read() for f in sys.argv[1:]) if sys.argv[1:] else SAMPLE
    rows = parse(text)
    n = 16
    f = busbw_factor(n)
    print(f"ranks={n}: busbw = algbw x {f:.4f}")
    best = max(r[4] for r in rows)
    for pair, size, t, alg, bus, wrong in rows:
        alg_calc = size / (t * 1e-6) / 1e9
        flag = "OK" if bus >= 0.9 * best and wrong == 0 else "SUSPECT"
        print(f"{pair:18s} algbw {alg:6.2f} (recalc {alg_calc:6.2f}) busbw {bus:6.2f} GB/s "
              f"x{f:.4f}={alg*f:6.2f}  {flag}")
    bad = [r[0] for r in rows if r[4] < 0.9 * best]
    print("drain and re-test individually:", ", ".join(bad) if bad else "none")

if __name__ == "__main__":
    main()
$ python3 nccl_check.py # on the built-in sample log ranks=16: busbw = algbw x 1.8750 node001,node002 algbw 190.00 (recalc 190.00) busbw 356.25 GB/s x1.8750=356.25 OK node003,node004 algbw 188.87 (recalc 188.87) busbw 354.13 GB/s x1.8750=354.13 OK node005,node006 algbw 87.54 (recalc 87.55) busbw 164.14 GB/s x1.8750=164.14 SUSPECT node007,node008 algbw 189.24 (recalc 189.25) busbw 354.83 GB/s x1.8750=354.83 OK drain and re-test individually: node005,node006

10. What bring-up found

Node life cycle

📦NewDelivered, inventoried
🧪TestingBurn-in, links, pairs
✅AcceptedBaseline numbers recorded
🏃In serviceHanded to the scheduler
🅿️ParkedFailed a test; fixed, then back to Testing

A repaired node always goes back to Testing, never straight into service.

Scenario
  • Across 128 nodes, bring-up parks 9 nodes: 3 with GPUs failing DCGM memory tests, 2 with a fabric port trained at reduced speed (a reseated optic fixes one, a replaced cable the other), 1 with a NIC in the wrong PCIe slot after a repair, 2 with an older firmware version, and 1 that throttles under the 24-hour burn because of a blocked airflow path.
    • All are fixed or replaced by the provider within the contract's replacement window.
    • These counts are made up for the story; Meta's real record (section 2) shows the kind of mix to expect, not the number.

Two practices make this manageable for one person:

11. Testing storage before the run depends on it

12. Firmware: when to upgrade, and when not to

Gopala's rules for a 128-node cluster:

13. The golden image

14. Handing over to the scheduler

15. Six bring-up mistakes

  1. Running only the full-cluster test, which says the cluster is slow but not where.
  2. Accepting nodes that passed a short diagnostic but never ran under sustained load.
  3. Putting repaired nodes straight back into service without the full burn-in.
  4. Letting firmware versions drift because each repair brought a different part.
  5. Not testing storage restart reads, only writes.
  6. Keeping results in chat threads instead of one file per node.

16. The acceptance report

17. Delegation box

🙋 Gopala himself
  • The burn-in and acceptance criteria
  • Deciding which nodes to park
  • Signing the acceptance report
🤖 AI agents
  • Running tests in parallel and collecting logs
  • Summarising failures and drafting tickets for the provider
  • Comparing firmware versions across nodes
🤝 Bought, rented or borrowed
  • The provider's on-site team (swaps, cables, optics)
  • NVIDIA's open tools: DCGM, nccl-tests
  • A contractor with InfiniBand experience for two weeks

18. Budget and risk tracker

ItemStatus after Part 6Note
Spend to date~$0.5M (scenario)Billing starts on accepted nodes; partial block from month 3
Nodes128 accepted, 9 replaced during bring-up (scenario)Baseline numbers recorded per node
Time18 of ~78 weeksMaestro-0 post-training under way
Hardware riskReducedInfant-mortality failures found before the main run
Fabric riskMediumSpine-layer congestion only shows under full load
Key-person riskHighBring-up knowledge written down as scripts

19. The series

The Maestro: 18 parts in five phases

Verdict
  • Day zero is about distrust: every node is tested alone, every link in pairs, the whole fabric together, and only then accepted and billed.
    • The parked nodes are the ones that would have stopped the run.
    • Next, Part 7: the software that keeps 1,024 GPUs busy and catches failures on its own.

20. Reading the numbers

Primary / standard
Llama 3 405B: 16,384 H100s, 466 interruptions in 54 days, 419 unexpected; faulty GPUs 30.1% and HBM3 17.2% of unexpected; >90% effective training time (Meta). busbw = algbw × 2(n−1)/n for all-reduce (nccl-tests PERFORMANCE.md). H100 NVLink 900 GB/s; ConnectX-7 400 Gb/s (NVIDIA). DCGM diagnostic levels (NVIDIA DCGM docs).
Secondary ⚠️
None in this part.
Our assumptions
The sample NCCL log, the 90% threshold, the 24-hour burn length and every defect count in the story.