Home โ€บ Blog โ€บ Day Zero: Bring-Up
Who Runs the AI Factory ยท Part 1 ยท Bring-up

Day Zero: The Engineers Who Turn 10,000 Boxes of GPUs into a Working AI Factory

Before an AI factory trains a single token, thousands of servers arrive on pallets, and someone has to make every one of them work, agree with every other one and survive days of punishing tests. This is the first part of a series that follows the software stack of an AI factory layer by layer, through the people who own each layer. It starts at the bottom: hardware bring-up and firmware, and the deployment engineers who turn racks of parts into a cluster the rest of the company can trust.

FL
FrontierAGI Team
โš ๏ธ How this was researched Frontier labs rarely publish their bring-up procedures. The process and test thresholds here come from published runbooks and guides by GPU cloud operators and integrators (Crusoe, GPUSmith, Introl) and are marked โš ๏ธ secondary; firmware standards come from the Open Compute Project; role descriptions come from real job postings (xAI, Nscale). Meta-specific facts are limited to its published hardware and failure data. The day-in-the-life timeline and simulators are illustrations. See Reading the numbers.

1. The series: one stack, twelve roles

Producing Intelligence Part 1 described the AI factory's software stack in one sweep. This series slows down and follows each layer through the people who run it: what software lives there, what they do all day, what breaks, and who they hand work to. Meta, whose Llama 3 run we followed in Producing Intelligence, is the main example, with other labs and clouds for contrast. Click a layer to see who owns it.

The AI factory stack, by role

2. The role: deployment and bring-up engineer

The job goes by many names: hardware deployment engineer, bring-up engineer, fleet engineer, data-centre hardware engineer. Real postings make it concrete. xAI's Lead Hardware Deployment Engineer role in Memphis covers rack-level integration and bring-up of GB300-class systems, building a deployment team and holding vendors to service levels; it asks for 5+ years of data-centre deployment and rack-scale GPU bring-up, prefers experience with liquid-cooled GB200/GB300 NVL72 racks, and warns of extended hours and weekends during critical phases. Nscale's Senior and Principal Deployment Engineer roles list node and rack deployment, firmware validation, InfiniBand/RoCE/Ethernet fabric bring-up and NCCL/GPUDirect performance validation; the principal level adds defining standards and acceptance criteria for multi-site superclusters.

What the role needs

Skill emphasis distilled from the postings above (our judgement, 1โ€“5).

The day timeline is a composite illustration of a bring-up phase, not a quote from any company.

3. From loading dock to first job

Bring-up is a pipeline. Every stage catches a different class of fault, and a pass at one stage says nothing about the next. A published acceptance runbook lays it out as baseline, preflight, single-node function, single-node stress, multi-node paths, performance qualification and disposition โš ๏ธ secondary. Click a stage.

The bring-up pipeline

4. Firmware: the invisible layer

Below the operating system sits a stack of firmware that most engineers never see, and that the bring-up team must get exactly right. Every server has a BMC (baseboard management controller), a small separate computer that powers the machine on and off, reads sensors and accepts updates even when the main system is down; it is increasingly built on the open-source OpenBMC, and NVIDIA's own BMC firmware is an OpenBMC-based distribution. Above it sit BIOS/UEFI, and separate firmware for every GPU, NIC, switch, NVLink switch tray, power shelf and, in liquid-cooled racks, the cooling controllers.

At fleet scale, firmware updates become a coordination problem. The Open Compute Project's GPU Firmware Update Specification, written by AMD, Google, Meta, Microsoft and NVIDIA, standardises how GPU firmware is updated through the Redfish management API, with requirements for secure updates, rollback prevention and non-disruptive updates.

Version drift: why "the same" servers are not the same

Illustrative. Each square is a node in a 64-node job. Change how much of the fleet has been updated.

5. Burn-in and acceptance testing

Hardware fails most often early in its life and late in its life. Burn-in is the practice of running new hardware hard before users depend on it, so the early failures happen on the bring-up team's watch. Published guides describe layered testing โš ๏ธ secondary:

Durations vary: one guide calls 3โ€“7 days of continuous load the working norm and anything under 24 hours a smoke test; another gives 72โ€“168 hours. Example exit criteria: HPL and nccl-tests bus bandwidth within 5โ€“10% of the vendor's figures at full scale, and zero uncorrectable errors and zero fabric link flaps over the final 48 hours. GPU clouds such as Crusoe publish their burn-in process for customers.

Burn-in simulator: how many bad nodes get through?

Illustrative model of early-life failures. Weak nodes fail at random during their first weeks; burn-in catches those that fail while it runs.

6. Case study: bringing up GB200 NVL72

NVIDIA's GB200 NVL72 put 72 GPUs into one liquid-cooled rack joined by NVLink, which turned the rack, not the server, into the unit of bring-up. The early months were rough. In late 2024 The Information reported overheating that led NVIDIA to ask suppliers for rack design changes, and the Financial Times reported software bugs, connectivity problems between chips and leaking liquid-cooling systems. Suppliers responded by adding test protocols before racks shipped, and partners including Foxconn, Inventec, Dell and Wistron reported fixes, with shipments ramping from the end of the first quarter of 2025.

๐Ÿ’ก Lesson: the blast radius grew In an 8-GPU server, one bad part takes down 8 GPUs. In an NVL72 rack, a fault in one compute tray, a pump or an NVLink switch tray can affect every job using that 72-GPU NVLink domain. Rack-scale systems moved bring-up work upstream to the factory and made liquid-cooling checks a standard part of acceptance.

7. Why bring-up matters months later

Every weak node that slips through bring-up becomes someone else's incident. In the Llama 3 405B run (Producing Intelligence Part 5), faulty GPUs caused 148 interruptions (30.1%) and HBM3 memory another 72 (17.2%) over 54 days; automation handled almost all of them, with significant manual intervention needed only three times. Meta's earlier OPT-175B run, with less automation, saw 35 manual restarts and more than 100 hosts cycled out over two months. Good bring-up does not end failures, but it removes the weakest nodes before they reach a 16,000-GPU job, and the "lemon node" detection in Producing Intelligence Part 9 catches what is left.

8. Handoffs and careers

Who bring-up works with

๐Ÿงญ Getting into the role
Typical backgrounds
Data-centre technicians, HPC system administrators, hardware validation and test engineers, network engineers.
Core skills
Linux, scripting (Python, Bash), BMC tools (Redfish, IPMI), high-speed networking (InfiniBand, RoCE), NVIDIA tools (DCGM, nccl-tests), an instinct for power and thermal behaviour.
Growth path
Technician โ†’ deployment engineer โ†’ lead/principal (owning acceptance standards across sites) or a move into cluster platform and SRE roles.
Further reading
The On-Call Reality ยท Keeping the Factory Running

๐Ÿ—‚๏ธ Case file: Meta's hardware

Grand Teton
Meta's GPU server platform, first shown at OCP in 2022: a single monolithic design with integrated power, control, compute and fabric interfaces; later expanded to AMD MI300X.
Catalina
Meta's rack for NVIDIA Blackwell systems, unveiled at OCP in 2024.
Llama 3 fleet
Two 24,576-GPU clusters on Grand Teton (Producing Intelligence Part 9).
Not published
Meta's internal bring-up and burn-in procedures.
Verdict Bring-up is the least glamorous and one of the most consequential jobs in an AI factory. It is part logistics, part firmware archaeology and part statistics: deciding how long to torture new hardware, what "good enough" means and when a node is fit to join a job that will run for months. As racks grow to 72 GPUs and beyond and move to liquid cooling, the role is getting harder and more important.

๐Ÿ‘€ Next in the series

Part 2: OS, drivers and runtime โ€” the Linux and systems engineers who keep kernels, NVIDIA drivers, CUDA and container images consistent across tens of thousands of machines.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References