Running the Run: 54 Days, 466 Interruptions and the Final Polish of Llama 3 405B
Plans, data and parallelism layouts are the easy part. Then the job starts, and for months 16,384 GPUs must take hundreds of thousands of synchronised steps while hardware fails every few hours, the afternoon heat slows the chips, and the data centre's power draw swings by megawatts. Meta's paper is unusually candid about what happened. This part follows the run itself: the recipe in motion, the numerical tricks that kept it stable, every cause of failure in a 54-day window, the long-context extension, and the final annealing and averaging that produced the Llama 3 405B base model.
๐ The plan meets reality
With the software (Part 1), data (Part 2), plan (Part 3) and parallelism layout (Part 4) in place, Meta started the 405B run. Over a 54-day snapshot of pre-training it recorded 466 job interruptions: 47 planned (firmware upgrades, configuration changes) and 419 unexpected, roughly one every three hours. Yet the team kept effective training time above 90%, needed manual intervention only three times, and reported a recipe that was "very stable", with few loss spikes and no intervention needed to correct divergence.
โ๏ธ The recipe in motion
According to the paper, the 405B model used AdamW with a peak learning rate of 8ร10โปโต, a linear warm-up over 8,000 steps, and cosine decay to 8ร10โปโท over 1.2 million steps. The batch started small for stability and grew for efficiency: 4M tokens of 4,096-token sequences, doubling to 8M tokens of 8,192-token sequences after 252M tokens, and to 16M after 2.87T tokens. Meanwhile the data mix shifted (more non-English and maths, recent web data added late), followed by the long-context stage and annealing.
๐๏ธ Scrub through the run
Learning rate (line), batch size and phase across ~15.6T tokens. Published milestones are exact; curve shapes between them are illustrative.
๐ข Precision and numerical stability
Llama 3 trained in BF16, a 16-bit format that keeps a wide range but only about three significant digits. That is fine for multiplying matrices but dangerous when adding up many small values: once a running total is large, small additions round away to nothing. Meta fixed numerical issues that affected stability by accumulating gradients in FP32 across micro-batches during the pipeline's backward pass and reduce-scattering gradients in FP32 across data-parallel workers in FSDP. Meta's ISCA 2025 paper describes the same setup. (DeepSeek-V3, by contrast, trained largely in FP8, with careful higher-precision accumulation.)
๐งช Why accumulation precision matters
Add the same small gradient value many times, keeping the running total in BF16 vs FP32. A toy demonstration of rounding, not Meta's code.
๐งพ The failure ledger
Table 5 of the paper classifies the 419 unexpected interruptions. About 78% were attributed to confirmed or suspected hardware problems, and GPU-related issues alone made up 58.7%.
๐ง Root causes of unexpected interruptions (54 days)
Count and share as reported in Table 5. Click a cause.
Llama 3 paper Table 5 via search results. "Other" groups the remaining smaller categories not individually listed in our sources. Shares are as the paper reports them.
๐ Detect, isolate, recover
At one failure every three hours, recovery has to be automatic. Meta reduced job start-up and checkpointing time, and built tools for fast diagnosis: PyTorch's NCCL flight recorder captured the state of collective operations so hangs and slowdowns could be traced, especially in NCCLX, Meta's communication library; and tooling identified stragglers, slow GPUs or links that drag every synchronised step down. Part 1's failure simulator shows how checkpoint frequency and restart time set the effective training time.
โป๏ธ The incident loop
Press play to step through what happens after a GPU fails.
๐ก๏ธ Physics shows up: heat and power
Two environmental effects made it into the paper. First, throughput varied by about 1โ2% over the course of each day, because higher mid-day temperatures affected the GPUs' dynamic voltage and frequency scaling (see Heat and Water). Second, when tens of thousands of GPUs change their power draw together, for example all waiting for a checkpoint or a collective to finish, or the whole job starting or stopping, data-centre power consumption can swing by tens of megawatts almost instantly, stretching the limits of the power grid (see The Power Bottleneck).
โก What the grid sees (illustrative)
Site power over a few minutes of training. Choose an event.
Illustrative shape. Assumes ~16,384 GPUs at up to 700 W plus servers, networking and cooling (~20โ25 MW at full load). Meta reported swings "on the order of tens of megawatts" across the data centre.
๐ The long-context stage
Most pre-training used 8,192-token sequences. Near the end, Meta extended the context window to 128K tokens gradually, in six stages, using about 800 billion tokens. It moved to each next stage only when two conditions held: performance on short-context tasks had fully recovered, and the model could solve "needle in a haystack" tasks, retrieving a specific fact hidden anywhere in a long input. This stage used context parallelism (Part 4) and ran at lower efficiency, about 38% MFU.
๐ช From 8K to 128K in six stages
Start and end lengths and the stage count are from the paper; intermediate lengths are not specified in our sources, so steps are drawn evenly on a log scale.
โจ Annealing and checkpoint averaging
The final polish came in the last 40 million tokens. Meta linearly annealed the learning rate to zero while keeping a 128K context, and adjusted the data mix to upsample very high-quality sources (Part 2). Then, rather than taking the last checkpoint, it computed the average of model checkpoints taken during annealing (Polyak averaging, 1991) to produce the final pre-trained model. Averaging smooths out the noise of individual training steps, typically giving a slightly better and more stable model.
๐ Why average checkpoints?
One weight's value across checkpoints during annealing (illustrative). The average is less noisy than any single checkpoint.
๐ The result: a base model
What comes out of pre-training is a base model: it has absorbed 15.6 trillion tokens of knowledge and can continue any text, handle 128K-token contexts and solve many problems, but it is not yet a helpful, safe assistant that follows instructions. Turning it into Llama 3.1 405B Instruct takes post-training, the subject of Part 6.
๐ฐ๏ธ Timeline of the run (as described)
๐ญ What to watch
- Fault tolerance at 100,000+ GPUs: training that continues through failures without stopping the whole job.
- FP8 and lower-precision training becoming standard, with careful accumulation.
- Power smoothing: batteries and software that keep synchronised GPUs from jolting the grid.
- Next in this series: Part 6 turns the base model into an assistant: supervised fine-tuning, reward models, preference optimisation and reinforcement learning.
๐ข Reading the numbers
- Recipe, batch schedule, stability claim, Table 5 rows, 78%/58.7%, diurnal 1โ2%, tens-of-MW power swings, FP32 accumulation, long-context stages and annealing come from the Llama 3 paper as quoted in search results.
- Table 5 percentages are reproduced as reported; the "Other" bar is the remainder we computed.
- Charts (timeline curve shapes, power trace, rounding demo, averaging) are illustrative.
- Checkpoint frequency, loss curves and intermediate context lengths are not published in our sources.
๐ References
- Llama 3
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024), sections 3.3.4 and 3.4, Table 5
- "Scaling Llama 3 Training with Efficient Parallelism Strategies" (ISCA 2025)
- Tom's Hardware: Faulty H100s and HBM3 caused half of Llama 3's training failures
- Simon Willison: A quote from The Llama 3 Herd of Models (power swings)
- MLOps newsletter: Scaling and reliability challenges of Llama 3
- Why it's hard to train a model like Llama 3 405B
- Papers Explained: Llama 3.1 (long-context stages)
- Methods and related work
- Polyak & Juditsky: "Acceleration of Stochastic Approximation by Averaging" (1992)
- Loshchilov & Hutter: "Decoupled Weight Decay Regularization" (AdamW)
- Micikevicius et al.: "Mixed Precision Training" (2018)
- DeepSeek-AI: "DeepSeek-V3 Technical Report" (FP8 training)
- "Training LLMs with Fault Tolerant HSDP on 100,000 GPUs" (2026)
- "From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs" (2026)
- "TrainMover: An Interruption-Resilient Runtime for ML Training"
- Related on this site
- Part 4: Splitting a Giant ยท Debugging a Distributed Job ยท How LLM Pretraining Works ยท Heat and Water