Keeping the Factory Running: Health Checks, Lemon Nodes and Rolling Maintenance Behind Llama 3
Building an AI factory is day one. Every day after that is operations: tens of thousands of GPUs that must stay healthy, busy and up to date while training runs for months and inference never stops. Hardware fails every few hours, a single slow server can hold back thousands of others, and firmware must be updated without taking the fleet offline. This final part of the series looks at how Meta runs the clusters that trained Llama 3, from automated health checks and "lemon node" hunting to rolling maintenance and the hunt for chips that silently compute wrong answers.
1. Day two
The earlier parts of this series followed one model through one factory: the software stack, the data, the plan, the parallelism, the run, post-training, evaluation and serving. All of it assumed that the machines were there and working. Part 5 showed how fragile that assumption is: in a 54-day snapshot of the Llama 3 405B run there were 466 job interruptions, 419 of them unexpected, roughly one every three hours. Operations is the work that turns that unreliable hardware into a dependable service.
It covers four jobs that never finish: detect broken or degraded hardware, isolate it before it hurts jobs, repair or replace it and bring it back, and update software and firmware across the fleet without stopping the work.
2. The fleet
In March 2024 Meta described two clusters of 24,576 GPUs each, built on its Grand Teton servers, OpenRack and PyTorch, and said Llama 3 was being trained on this design. The two clusters were deliberately different. One uses an RoCE Ethernet fabric (Arista 7800 switches with Meta's Wedge400 and Minipack2 rack switches); the other uses NVIDIA Quantum-2 InfiniBand. Both connect 400 Gbps endpoints. Meta's largest earlier RoCE cluster had about 4,000 GPUs, so building both at the same scale let it compare the two; Meta says it tuned them to deliver equivalent performance for generative AI workloads.
Two clusters, two fabrics
Click a cluster.
3. Failure gets worse with scale
Individual components are reliable; fleets are not. If each GPU (with its share of memory, network and host) fails on average once every several years, a job that needs all of 16,384 of them at once will still be interrupted every few hours, because any single failure stops the synchronised job. Meta's reliability paper on its research clusters RSC-1 and RSC-2 (about 24,000 A100 GPUs, 11 months, 4 million jobs and over 150 million GPU-hours) built projections of mean time to failure by job size and validated them on real jobs of up to 4,000 GPUs.
Job size vs time between failures
Failure rate per GPU derived from Llama 3: 419 unexpected interruptions in 54 days on 16,384 GPUs. Assumes failures are independent and spread evenly, which is a simplification.
4. Health checks
The first line of defence is automated testing that runs constantly in the background. Meta's research-cluster paper describes health checks integrated with the Slurm scheduler, which run regularly on every node (every five minutes, per a review of the paper โ ๏ธ secondary) and catch overlapping problems: GPU errors, memory errors, network link problems, file-system issues. A node that fails a check is drained, meaning no new jobs are placed on it, and sent to repair. A summary of a Meta talk on the Llama 3 clusters lists the most common hardware failures as GPUs dropping off the PCIe bus, GPU driver hangs, and uncorrectable DRAM or SRAM errors โ ๏ธ secondary.
The life of a node
Click a state to see what happens there.
5. Lemon nodes
Health checks catch nodes that are clearly broken. Harder are nodes that pass every check yet keep causing jobs to fail. Meta's paper calls these lemons and describes a detection system that removes them from Slurm. It scores each node on signals such as how many distinct jobs excluded it, the number of unique GPU (XID) errors, repair-ticket count, how often it was taken out of service, and its single- and multi-node job failure counts. The thresholds were tuned by hand on a 28-day snapshot of the RSC-1 cluster to balance accuracy against false positives, and Meta reports the system noticeably reduced failure rates of large jobs.
Is this node a lemon?
Illustrative scoring using the signal types from Meta's paper. Thresholds are invented for the demo.
6. The slow host
A failed node is obvious. A node that is only a little slow is worse, because nothing alerts on it. Meta's maintenance post makes the point directly: in jobs where all hosts must stay synchronised, a host that is just a little slow, has minor hardware issues or has network problems becomes a huge problem. Every training step waits for the slowest GPU.
One straggler, a thousand waiting GPUs
Illustrative. Each bar is a group of GPUs; the step ends when the slowest finishes.
7. Maintenance without downtime
Software and firmware never stop changing: GPU drivers, NIC firmware, BIOS, kernels, switch software. Meta's June 2024 post defines maintenance as any operation that updates or verifies software and firmware components in the clusters, including the networking path. Because many jobs are time-critical, recurring or online, Meta cannot take large amounts of capacity offline for it. Its answer is staged, rolling maintenance: interruptive updates are rolled out a slice of capacity at a time, and jobs are made to run correctly on a fleet where hosts are temporarily at different versions. A summary of the post says the AI-facing stack (such as CUDA libraries) is kept consistent while lower layers such as the OS, firmware and network are staged โ ๏ธ secondary. It took careful testing, dedicated monitoring and close work with vendors.
A rolling maintenance wave
Illustrative: 48 racks updated in slices. Press play and change the slice size.
8. Measuring goodput
The metric that matters is not how busy GPUs look but how much useful training they do. Meta's reliability paper introduces the Effective Training Time Ratio (ETTR): the share of a job's wall-clock time spent making real progress, after losses to failures, restarts, re-doing work since the last checkpoint and waiting in queues. The Llama 3 paper reported more than 90% effective training time for the 405B run. A summary of a Meta talk puts training efficiency for Llama 3 70B at about 95% over a rolling 24-hour window โ ๏ธ secondary.
Where the GPU-hours go
Illustrative waterfall for a large training job. Adjust the losses.
9. Silent data corruption
The most insidious fault does not crash anything. A defective chip occasionally computes a wrong answer and reports no error. Meta's 2021 paper "Silent Data Corruptions at Scale" described running a large library of silent-error tests across hundreds of thousands of machines and finding hundreds of CPUs with such defects, monitored over more than 18 months. Press coverage of the follow-up work reports that in-production testing catches about 70% of these corruptions within 15 days, with the rest needing opportunistic testing that each machine undergoes about every six months.
For AI training the danger is clear: a wrong value in a gradient or a weight can quietly spread through a weeks-long run, showing up later as a loss spike or a subtle quality drop. The same idea, constantly testing hardware in production rather than trusting burn-in tests, applies to GPUs as well as CPUs; Meta's published SDC figures above are for CPUs.
๐๏ธ Llama 3 case file: operations
- Fleet
- Two 24,576-GPU clusters (RoCE and InfiniBand), Grand Teton servers, 400 Gbps endpoints
- Failure rate
- 466 interruptions in 54 days for the 405B run, 419 unexpected
- Detection
- Health checks integrated with the scheduler; lemon-node detection removes repeat offenders
- Maintenance
- Staged rolling updates; jobs tolerate mixed host versions
- Goodput
- >90% effective training time (405B); ~95% (70B) โ ๏ธ secondary
- Hidden faults
- Slow hosts and silent data corruption treated as first-class problems
๐ What to watch
- Fault-tolerant training that keeps going when a node dies, instead of restarting the whole job.
- GPU-specific silent-corruption detection at 100,000-GPU scale.
- Power and cooling becoming operational constraints alongside hardware faults.
- The Producing Intelligence hub, coming next, bringing all nine parts together.
๐ The series in one view
๐ข Reading the numbers
- Cluster design comes from Meta's March 2024 engineering post; maintenance definitions and approach from its June 2024 post.
- Research-cluster scale, MTTF projections, ETTR and lemon detection signals come from arXiv 2410.21680 as quoted in search results.
- Interruption counts and >90% effective training time come from the Llama 3 paper.
- Health-check interval, common failure types, 70B efficiency and the version-staging detail come from secondary summaries โ ๏ธ.
- SDC findings come from Meta's 2021 paper and press coverage of later work.
- The failure calculator uses a per-GPU rate derived from Llama 3's 419 unexpected interruptions; specific MTTF figures by job size from Meta's paper are not reproduced because we could not verify them. Lemon scoring, straggler, maintenance and waterfall demos are illustrations.
๐ References
- Meta
- Meta Engineering: Building Meta's GenAI Infrastructure (Mar 2024)
- Meta Engineering: Maintaining large-scale AI capacity at Meta (Jun 2024)
- Kokolis et al., Meta: "Revisiting Reliability in Large-Scale Machine Learning Research Clusters" (2024)
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024), section 3.3.4
- Dixit et al., Meta: "Silent Data Corruptions at Scale" (2021)
- Secondary
- ZenML: Building and operating 24K GPU clusters for Llama training (talk summary)
- Glenn Klockwood: review of "Revisiting Reliabilityโฆ"
- Gigazine: How Meta maintains infrastructure to run AI at scale
- Data Center Knowledge: How Meta mitigates hidden CPU defects at scale
- DCD: Meta reveals details of two new 24K GPU AI clusters
- Related on this site
- Part 8: From Checkpoint to Endpoint ยท Part 5: Running the Run ยท Debugging a Distributed Job