Day Zero: The Engineers Who Turn 10,000 Boxes of GPUs into a Working AI Factory
Before an AI factory trains a single token, thousands of servers arrive on pallets, and someone has to make every one of them work, agree with every other one and survive days of punishing tests. This is the first part of a series that follows the software stack of an AI factory layer by layer, through the people who own each layer. It starts at the bottom: hardware bring-up and firmware, and the deployment engineers who turn racks of parts into a cluster the rest of the company can trust.
1. The series: one stack, twelve roles
Producing Intelligence Part 1 described the AI factory's software stack in one sweep. This series slows down and follows each layer through the people who run it: what software lives there, what they do all day, what breaks, and who they hand work to. Meta, whose Llama 3 run we followed in Producing Intelligence, is the main example, with other labs and clouds for contrast. Click a layer to see who owns it.
The AI factory stack, by role
2. The role: deployment and bring-up engineer
The job goes by many names: hardware deployment engineer, bring-up engineer, fleet engineer, data-centre hardware engineer. Real postings make it concrete. xAI's Lead Hardware Deployment Engineer role in Memphis covers rack-level integration and bring-up of GB300-class systems, building a deployment team and holding vendors to service levels; it asks for 5+ years of data-centre deployment and rack-scale GPU bring-up, prefers experience with liquid-cooled GB200/GB300 NVL72 racks, and warns of extended hours and weekends during critical phases. Nscale's Senior and Principal Deployment Engineer roles list node and rack deployment, firmware validation, InfiniBand/RoCE/Ethernet fabric bring-up and NCCL/GPUDirect performance validation; the principal level adds defining standards and acceptance criteria for multi-site superclusters.
What the role needs
Skill emphasis distilled from the postings above (our judgement, 1โ5).
The day timeline is a composite illustration of a bring-up phase, not a quote from any company.
3. From loading dock to first job
Bring-up is a pipeline. Every stage catches a different class of fault, and a pass at one stage says nothing about the next. A published acceptance runbook lays it out as baseline, preflight, single-node function, single-node stress, multi-node paths, performance qualification and disposition โ ๏ธ secondary. Click a stage.
The bring-up pipeline
4. Firmware: the invisible layer
Below the operating system sits a stack of firmware that most engineers never see, and that the bring-up team must get exactly right. Every server has a BMC (baseboard management controller), a small separate computer that powers the machine on and off, reads sensors and accepts updates even when the main system is down; it is increasingly built on the open-source OpenBMC, and NVIDIA's own BMC firmware is an OpenBMC-based distribution. Above it sit BIOS/UEFI, and separate firmware for every GPU, NIC, switch, NVLink switch tray, power shelf and, in liquid-cooled racks, the cooling controllers.
At fleet scale, firmware updates become a coordination problem. The Open Compute Project's GPU Firmware Update Specification, written by AMD, Google, Meta, Microsoft and NVIDIA, standardises how GPU firmware is updated through the Redfish management API, with requirements for secure updates, rollback prevention and non-disruptive updates.
Version drift: why "the same" servers are not the same
Illustrative. Each square is a node in a 64-node job. Change how much of the fleet has been updated.
5. Burn-in and acceptance testing
Hardware fails most often early in its life and late in its life. Burn-in is the practice of running new hardware hard before users depend on it, so the early failures happen on the bring-up team's watch. Published guides describe layered testing โ ๏ธ secondary:
- Per component: NVIDIA DCGM diagnostics (up to its longest "level 4" run) on each GPU and its memory. A DCGM pass describes only the test that ran; it does not certify the node.
- Per node: HPL (the Linpack benchmark) to load compute and power, plus NVLink bandwidth checks; fio for local storage.
- Fabric: nccl-tests (all-reduce and friends) at increasing scale, from a pair of nodes to the full cluster.
- System: a full-cluster training job, typically 12โ48 hours.
Durations vary: one guide calls 3โ7 days of continuous load the working norm and anything under 24 hours a smoke test; another gives 72โ168 hours. Example exit criteria: HPL and nccl-tests bus bandwidth within 5โ10% of the vendor's figures at full scale, and zero uncorrectable errors and zero fabric link flaps over the final 48 hours. GPU clouds such as Crusoe publish their burn-in process for customers.
Burn-in simulator: how many bad nodes get through?
Illustrative model of early-life failures. Weak nodes fail at random during their first weeks; burn-in catches those that fail while it runs.
6. Case study: bringing up GB200 NVL72
NVIDIA's GB200 NVL72 put 72 GPUs into one liquid-cooled rack joined by NVLink, which turned the rack, not the server, into the unit of bring-up. The early months were rough. In late 2024 The Information reported overheating that led NVIDIA to ask suppliers for rack design changes, and the Financial Times reported software bugs, connectivity problems between chips and leaking liquid-cooling systems. Suppliers responded by adding test protocols before racks shipped, and partners including Foxconn, Inventec, Dell and Wistron reported fixes, with shipments ramping from the end of the first quarter of 2025.
7. Why bring-up matters months later
Every weak node that slips through bring-up becomes someone else's incident. In the Llama 3 405B run (Producing Intelligence Part 5), faulty GPUs caused 148 interruptions (30.1%) and HBM3 memory another 72 (17.2%) over 54 days; automation handled almost all of them, with significant manual intervention needed only three times. Meta's earlier OPT-175B run, with less automation, saw 35 manual restarts and more than 100 hosts cycled out over two months. Good bring-up does not end failures, but it removes the weakest nodes before they reach a 16,000-GPU job, and the "lemon node" detection in Producing Intelligence Part 9 catches what is left.
8. Handoffs and careers
Who bring-up works with
- Typical backgrounds
- Data-centre technicians, HPC system administrators, hardware validation and test engineers, network engineers.
- Core skills
- Linux, scripting (Python, Bash), BMC tools (Redfish, IPMI), high-speed networking (InfiniBand, RoCE), NVIDIA tools (DCGM, nccl-tests), an instinct for power and thermal behaviour.
- Growth path
- Technician โ deployment engineer โ lead/principal (owning acceptance standards across sites) or a move into cluster platform and SRE roles.
- Further reading
- The On-Call Reality ยท Keeping the Factory Running
๐๏ธ Case file: Meta's hardware
- Grand Teton
- Meta's GPU server platform, first shown at OCP in 2022: a single monolithic design with integrated power, control, compute and fabric interfaces; later expanded to AMD MI300X.
- Catalina
- Meta's rack for NVIDIA Blackwell systems, unveiled at OCP in 2024.
- Llama 3 fleet
- Two 24,576-GPU clusters on Grand Teton (Producing Intelligence Part 9).
- Not published
- Meta's internal bring-up and burn-in procedures.
๐ Next in the series
Part 2: OS, drivers and runtime โ the Linux and systems engineers who keep kernels, NVIDIA drivers, CUDA and container images consistent across tens of thousands of machines.
๐ข Reading the numbers
- Role descriptions come from xAI and Nscale job postings.
- Pipeline stages, test types, durations and exit criteria come from operator and integrator guides (GPUSmith, Introl, Crusoe) โ ๏ธ secondary; practices vary by company.
- Firmware standards come from the OCP GPU Firmware Update Specification and an NVIDIA paper on its OpenBMC-based BMC firmware.
- GB200 NVL72 issues come from reporting by The Information and the Financial Times, as summarised by DCD and others.
- Llama 3 and OPT-175B failure figures come from Meta's publications.
- Skills radar, day timeline, version-drift demo and burn-in simulator are illustrations.
๐ References
- Bring-up and acceptance
- GPUSmith: GPU cluster acceptance testing runbook
- Introl: Validation frameworks for GPU clusters before production
- Crusoe: How we burn-in test every node
- Firmware
- Open Compute Project: GPU Firmware Update Specification v1.1
- NVIDIA: platform configuration for its OpenBMC-based BMC firmware (arXiv 2602.24237)
- DCD: Meta expands Grand Teton, unveils Catalina
- GB200 NVL72
- DCD: NVIDIA redesigns 72-GPU racks after overheating reports
- DCD: Server makers solve Blackwell issues, ramp GB200 shipments
- Roles
- xAI: Lead, Hardware Deployment Engineer (posting)
- Nscale: Deployment Engineer, GPU infrastructure (posting)
- Failures
- DCD: Meta report details GPU and HBM3 interruptions in Llama 3 training
- Gigazine: How Meta trains large language models (OPT-175B restarts)
- Related on this site
- Producing Intelligence Part 1: The Software Stack ยท Part 9: Operations ยท Inside the AI Factory