Home โ€บ Blog โ€บ Running the Run
Producing Intelligence ยท Part 5 ยท The run

Running the Run: 54 Days, 466 Interruptions and the Final Polish of Llama 3 405B

Plans, data and parallelism layouts are the easy part. Then the job starts, and for months 16,384 GPUs must take hundreds of thousands of synchronised steps while hardware fails every few hours, the afternoon heat slows the chips, and the data centre's power draw swings by megawatts. Meta's paper is unusually candid about what happened. This part follows the run itself: the recipe in motion, the numerical tricks that kept it stable, every cause of failure in a 54-day window, the long-context extension, and the final annealing and averaging that produced the Llama 3 405B base model.

FL
FrontierAGI Team
โš ๏ธ How this was researched Facts come from Meta's paper "The Llama 3 Herd of Models" (sections 3.3.4 reliability, 3.4 training recipe; Table 5), quoted in search results and in coverage by Tom's Hardware, Simon Willison and others, plus Meta's ISCA 2025 paper. Meta did not publish its checkpoint interval, exact intermediate context lengths or loss curves, so the training-timeline chart, power-swing chart, rounding demo and averaging diagram are illustrations built around the published numbers. See Reading the numbers.

๐Ÿ The plan meets reality

With the software (Part 1), data (Part 2), plan (Part 3) and parallelism layout (Part 4) in place, Meta started the 405B run. Over a 54-day snapshot of pre-training it recorded 466 job interruptions: 47 planned (firmware upgrades, configuration changes) and 419 unexpected, roughly one every three hours. Yet the team kept effective training time above 90%, needed manual intervention only three times, and reported a recipe that was "very stable", with few loss spikes and no intervention needed to correct divergence.

466interruptions in 54 days (419 unexpected)
~3 haverage time between unexpected interruptions
>90%effective training time
3incidents that needed manual intervention

โš™๏ธ The recipe in motion

According to the paper, the 405B model used AdamW with a peak learning rate of 8ร—10โปโต, a linear warm-up over 8,000 steps, and cosine decay to 8ร—10โปโท over 1.2 million steps. The batch started small for stability and grew for efficiency: 4M tokens of 4,096-token sequences, doubling to 8M tokens of 8,192-token sequences after 252M tokens, and to 16M after 2.87T tokens. Meanwhile the data mix shifted (more non-English and maths, recent web data added late), followed by the long-context stage and annealing.

๐ŸŽž๏ธ Scrub through the run

Learning rate (line), batch size and phase across ~15.6T tokens. Published milestones are exact; curve shapes between them are illustrative.

๐Ÿ”ข Precision and numerical stability

Llama 3 trained in BF16, a 16-bit format that keeps a wide range but only about three significant digits. That is fine for multiplying matrices but dangerous when adding up many small values: once a running total is large, small additions round away to nothing. Meta fixed numerical issues that affected stability by accumulating gradients in FP32 across micro-batches during the pipeline's backward pass and reduce-scattering gradients in FP32 across data-parallel workers in FSDP. Meta's ISCA 2025 paper describes the same setup. (DeepSeek-V3, by contrast, trained largely in FP8, with careful higher-precision accumulation.)

๐Ÿงช Why accumulation precision matters

Add the same small gradient value many times, keeping the running total in BF16 vs FP32. A toy demonstration of rounding, not Meta's code.

๐Ÿงพ The failure ledger

Table 5 of the paper classifies the 419 unexpected interruptions. About 78% were attributed to confirmed or suspected hardware problems, and GPU-related issues alone made up 58.7%.

๐Ÿ”ง Root causes of unexpected interruptions (54 days)

Count and share as reported in Table 5. Click a cause.

Llama 3 paper Table 5 via search results. "Other" groups the remaining smaller categories not individually listed in our sources. Shares are as the paper reports them.

๐Ÿ” Detect, isolate, recover

At one failure every three hours, recovery has to be automatic. Meta reduced job start-up and checkpointing time, and built tools for fast diagnosis: PyTorch's NCCL flight recorder captured the state of collective operations so hangs and slowdowns could be traced, especially in NCCLX, Meta's communication library; and tooling identified stragglers, slow GPUs or links that drag every synchronised step down. Part 1's failure simulator shows how checkpoint frequency and restart time set the effective training time.

โ™ป๏ธ The incident loop

Press play to step through what happens after a GPU fails.

๐ŸŒก๏ธ Physics shows up: heat and power

Two environmental effects made it into the paper. First, throughput varied by about 1โ€“2% over the course of each day, because higher mid-day temperatures affected the GPUs' dynamic voltage and frequency scaling (see Heat and Water). Second, when tens of thousands of GPUs change their power draw together, for example all waiting for a checkpoint or a collective to finish, or the whole job starting or stopping, data-centre power consumption can swing by tens of megawatts almost instantly, stretching the limits of the power grid (see The Power Bottleneck).

โšก What the grid sees (illustrative)

Site power over a few minutes of training. Choose an event.

Illustrative shape. Assumes ~16,384 GPUs at up to 700 W plus servers, networking and cooling (~20โ€“25 MW at full load). Meta reported swings "on the order of tens of megawatts" across the data centre.

๐Ÿ“ The long-context stage

Most pre-training used 8,192-token sequences. Near the end, Meta extended the context window to 128K tokens gradually, in six stages, using about 800 billion tokens. It moved to each next stage only when two conditions held: performance on short-context tasks had fully recovered, and the model could solve "needle in a haystack" tasks, retrieving a specific fact hidden anywhere in a long input. This stage used context parallelism (Part 4) and ran at lower efficiency, about 38% MFU.

๐Ÿชœ From 8K to 128K in six stages

Start and end lengths and the stage count are from the paper; intermediate lengths are not specified in our sources, so steps are drawn evenly on a log scale.

โœจ Annealing and checkpoint averaging

The final polish came in the last 40 million tokens. Meta linearly annealed the learning rate to zero while keeping a 128K context, and adjusted the data mix to upsample very high-quality sources (Part 2). Then, rather than taking the last checkpoint, it computed the average of model checkpoints taken during annealing (Polyak averaging, 1991) to produce the final pre-trained model. Averaging smooths out the noise of individual training steps, typically giving a slightly better and more stable model.

๐Ÿ“‰ Why average checkpoints?

One weight's value across checkpoints during annealing (illustrative). The average is less noisy than any single checkpoint.

๐ŸŽ The result: a base model

What comes out of pre-training is a base model: it has absorbed 15.6 trillion tokens of knowledge and can continue any text, handle 128K-token contexts and solve many problems, but it is not yet a helpful, safe assistant that follows instructions. Turning it into Llama 3.1 405B Instruct takes post-training, the subject of Part 6.

๐Ÿฆ™ Llama 3 case file: the run AdamW, peak LR 8ร—10โปโต, 8,000 warm-up steps, cosine to 8ร—10โปโท over 1.2M steps ยท batch 4M โ†’ 8M (after 252M tokens) โ†’ 16M (after 2.87T) ยท BF16 with FP32 gradient accumulation and reduce-scatter ยท 466 interruptions in 54 days (419 unexpected; 148 faulty GPUs, 72 HBM3) ยท >90% effective time ยท 1โ€“2% diurnal throughput swing ยท power swings of tens of MW ยท 8K โ†’ 128K context in six stages (~800B tokens) ยท final 40M tokens annealed to LR 0 ยท Polyak-averaged checkpoints = the base model.

๐Ÿ•ฐ๏ธ Timeline of the run (as described)

Step 0 ยท Warm-up over 8,000 steps to a peak learning rate of 8ร—10โปโต; 4M-token batches of 4,096-token sequences.
After 252M tokens ยท Batch doubles to 8M tokens; sequence length 8,192.
After 2.87T tokens ยท Batch doubles again to 16M tokens.
A 54-day window ยท 466 interruptions, 419 unexpected; >90% effective training time.
Throughout ยท Data mix adjusted: more non-English data, upsampled maths, recent web data added late.
Near the end ยท Long-context pre-training: six stages from 8K to 128K, ~800B tokens.
Final 40M tokens ยท Learning rate annealed linearly to 0 on high-quality data; checkpoints averaged into the base model.
July 2024 ยท Llama 3.1 405B released, base and instruct.

๐Ÿ”ญ What to watch

๐ŸŽฏ The bottom line Running a frontier pre-training job is operations at extreme scale. Llama 3 405B's recipe was stable, but the hardware was not: 419 unexpected interruptions in 54 days, mostly failing GPUs and memory, handled almost entirely by automation. Small numerical choices (FP32 accumulation) kept training stable; physics intruded through heat and megawatt power swings; and the run ended with a carefully staged long-context extension, a final annealing on the best data and an average of checkpoints. The output is a base model, ready for post-training.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References