Home โ€บ Blog โ€บ Keeping the Factory Running
Producing Intelligence ยท Part 9 ยท Operations

Keeping the Factory Running: Health Checks, Lemon Nodes and Rolling Maintenance Behind Llama 3

Building an AI factory is day one. Every day after that is operations: tens of thousands of GPUs that must stay healthy, busy and up to date while training runs for months and inference never stops. Hardware fails every few hours, a single slow server can hold back thousands of others, and firmware must be updated without taking the fleet offline. This final part of the series looks at how Meta runs the clusters that trained Llama 3, from automated health checks and "lemon node" hunting to rolling maintenance and the hunt for chips that silently compute wrong answers.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are two Meta engineering posts ("Building Meta's GenAI Infrastructure", March 2024; "Maintaining large scale AI capacity at Meta", June 2024), Meta's paper "Revisiting Reliability in Large-Scale Machine Learning Research Clusters" (arXiv 2410.21680), the Llama 3 paper and Meta's "Silent Data Corruptions at Scale" paper, mostly as quoted in search results. Some details come only from secondary summaries of Meta talks and are marked โš ๏ธ secondary. The failure calculator is anchored to Llama 3's published interruption count; the other simulators are illustrations. See Reading the numbers.
2 ร— 24,576GPUs in Meta's two Llama 3-era clusters
~3 hoursbetween unexpected interruptions in the Llama 3 405B run
150M+A100 GPU-hours studied in Meta's reliability paper
Hundredsof CPUs with silent errors found in Meta's fleet

1. Day two

The earlier parts of this series followed one model through one factory: the software stack, the data, the plan, the parallelism, the run, post-training, evaluation and serving. All of it assumed that the machines were there and working. Part 5 showed how fragile that assumption is: in a 54-day snapshot of the Llama 3 405B run there were 466 job interruptions, 419 of them unexpected, roughly one every three hours. Operations is the work that turns that unreliable hardware into a dependable service.

It covers four jobs that never finish: detect broken or degraded hardware, isolate it before it hurts jobs, repair or replace it and bring it back, and update software and firmware across the fleet without stopping the work.

2. The fleet

In March 2024 Meta described two clusters of 24,576 GPUs each, built on its Grand Teton servers, OpenRack and PyTorch, and said Llama 3 was being trained on this design. The two clusters were deliberately different. One uses an RoCE Ethernet fabric (Arista 7800 switches with Meta's Wedge400 and Minipack2 rack switches); the other uses NVIDIA Quantum-2 InfiniBand. Both connect 400 Gbps endpoints. Meta's largest earlier RoCE cluster had about 4,000 GPUs, so building both at the same scale let it compare the two; Meta says it tuned them to deliver equivalent performance for generative AI workloads.

Two clusters, two fabrics

Click a cluster.

3. Failure gets worse with scale

Individual components are reliable; fleets are not. If each GPU (with its share of memory, network and host) fails on average once every several years, a job that needs all of 16,384 of them at once will still be interrupted every few hours, because any single failure stops the synchronised job. Meta's reliability paper on its research clusters RSC-1 and RSC-2 (about 24,000 A100 GPUs, 11 months, 4 million jobs and over 150 million GPU-hours) built projections of mean time to failure by job size and validated them on real jobs of up to 4,000 GPUs.

Job size vs time between failures

Failure rate per GPU derived from Llama 3: 419 unexpected interruptions in 54 days on 16,384 GPUs. Assumes failures are independent and spread evenly, which is a simplification.

๐Ÿ’ก Why the curve is so steep Time between failures falls in proportion to job size: double the GPUs and you halve the time between interruptions. A job that would see one failure a year on 8 GPUs sees several a day at frontier scale. This is why checkpointing, fast restarts and automated detection (Part 5) are not optional at 16K GPUs.

4. Health checks

The first line of defence is automated testing that runs constantly in the background. Meta's research-cluster paper describes health checks integrated with the Slurm scheduler, which run regularly on every node (every five minutes, per a review of the paper โš ๏ธ secondary) and catch overlapping problems: GPU errors, memory errors, network link problems, file-system issues. A node that fails a check is drained, meaning no new jobs are placed on it, and sent to repair. A summary of a Meta talk on the Llama 3 clusters lists the most common hardware failures as GPUs dropping off the PCIe bus, GPU driver hangs, and uncorrectable DRAM or SRAM errors โš ๏ธ secondary.

The life of a node

Click a state to see what happens there.

5. Lemon nodes

Health checks catch nodes that are clearly broken. Harder are nodes that pass every check yet keep causing jobs to fail. Meta's paper calls these lemons and describes a detection system that removes them from Slurm. It scores each node on signals such as how many distinct jobs excluded it, the number of unique GPU (XID) errors, repair-ticket count, how often it was taken out of service, and its single- and multi-node job failure counts. The thresholds were tuned by hand on a 28-day snapshot of the RSC-1 cluster to balance accuracy against false positives, and Meta reports the system noticeably reduced failure rates of large jobs.

Is this node a lemon?

Illustrative scoring using the signal types from Meta's paper. Thresholds are invented for the demo.

6. The slow host

A failed node is obvious. A node that is only a little slow is worse, because nothing alerts on it. Meta's maintenance post makes the point directly: in jobs where all hosts must stay synchronised, a host that is just a little slow, has minor hardware issues or has network problems becomes a huge problem. Every training step waits for the slowest GPU.

One straggler, a thousand waiting GPUs

Illustrative. Each bar is a group of GPUs; the step ends when the slowest finishes.

7. Maintenance without downtime

Software and firmware never stop changing: GPU drivers, NIC firmware, BIOS, kernels, switch software. Meta's June 2024 post defines maintenance as any operation that updates or verifies software and firmware components in the clusters, including the networking path. Because many jobs are time-critical, recurring or online, Meta cannot take large amounts of capacity offline for it. Its answer is staged, rolling maintenance: interruptive updates are rolled out a slice of capacity at a time, and jobs are made to run correctly on a fleet where hosts are temporarily at different versions. A summary of the post says the AI-facing stack (such as CUDA libraries) is kept consistent while lower layers such as the OS, firmware and network are staged โš ๏ธ secondary. It took careful testing, dedicated monitoring and close work with vendors.

A rolling maintenance wave

Illustrative: 48 racks updated in slices. Press play and change the slice size.

8. Measuring goodput

The metric that matters is not how busy GPUs look but how much useful training they do. Meta's reliability paper introduces the Effective Training Time Ratio (ETTR): the share of a job's wall-clock time spent making real progress, after losses to failures, restarts, re-doing work since the last checkpoint and waiting in queues. The Llama 3 paper reported more than 90% effective training time for the 405B run. A summary of a Meta talk puts training efficiency for Llama 3 70B at about 95% over a rolling 24-hour window โš ๏ธ secondary.

Where the GPU-hours go

Illustrative waterfall for a large training job. Adjust the losses.

9. Silent data corruption

The most insidious fault does not crash anything. A defective chip occasionally computes a wrong answer and reports no error. Meta's 2021 paper "Silent Data Corruptions at Scale" described running a large library of silent-error tests across hundreds of thousands of machines and finding hundreds of CPUs with such defects, monitored over more than 18 months. Press coverage of the follow-up work reports that in-production testing catches about 70% of these corruptions within 15 days, with the rest needing opportunistic testing that each machine undergoes about every six months.

For AI training the danger is clear: a wrong value in a gradient or a weight can quietly spread through a weeks-long run, showing up later as a loss spike or a subtle quality drop. The same idea, constantly testing hardware in production rather than trusting burn-in tests, applies to GPUs as well as CPUs; Meta's published SDC figures above are for CPUs.

๐Ÿ—‚๏ธ Llama 3 case file: operations

Fleet
Two 24,576-GPU clusters (RoCE and InfiniBand), Grand Teton servers, 400 Gbps endpoints
Failure rate
466 interruptions in 54 days for the 405B run, 419 unexpected
Detection
Health checks integrated with the scheduler; lemon-node detection removes repeat offenders
Maintenance
Staged rolling updates; jobs tolerate mixed host versions
Goodput
>90% effective training time (405B); ~95% (70B) โš ๏ธ secondary
Hidden faults
Slow hosts and silent data corruption treated as first-class problems
Verdict At AI-factory scale, hardware failure is a constant, not an exception. Meta's approach is industrial: measure everything, automate detection and removal, roll changes through the fleet in slices, and judge success by goodput rather than uptime. The lesson for anyone planning a large cluster is that operational tooling is as much part of the factory as the GPUs; without it, a 16,000-GPU job would spend much of its life restarting.

๐Ÿ‘€ What to watch

๐Ÿ The series in one view

Part 1 ยท Software stack. CUDA, NCCL, PyTorch, schedulers and storage.
Part 2 ยท Data. 15 trillion tokens collected, cleaned and mixed.
Part 3 ยท Planning. Scaling laws and a 3.8ร—10ยฒโต FLOP budget.
Part 4 ยท Parallelism. 4D split across 16,384 GPUs.
Part 5 ยท The run. 54 days, 466 interruptions, annealing.
Part 6 ยท Post-training. Rounds of reward model, rejection sampling, SFT and DPO.
Part 7 ยท Evaluation. Benchmarks, contamination, human judges, red teams.
Part 8 ยท Serving. Two nodes, FP8 and the KV cache.
Part 9 ยท Operations. Keeping the factory running.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References