Home › Blog › Planning the Run
Producing Intelligence · Part 3 · Planning

Planning the Run: How Meta Sized Llama 3 405B with Scaling Laws and a 3.8×10²⁵ FLOP Budget

A frontier training run is a bet worth tens of millions of dollars, and most of its key decisions are made before the first GPU starts. How big should the model be? How many tokens should it read? How many GPUs, for how long, and what will it score when it is done? Meta answered those questions for Llama 3 with small-scale experiments and scaling laws, and its paper shows the arithmetic. This part walks through it: the compute equation, IsoFLOP curves, predicting benchmark scores in advance, why small models are deliberately "overtrained", and the final bill in GPU-hours and carbon.

FL
FrontierAGI Team
⚠️ How this was researched Figures come from Meta's paper "The Llama 3 Herd of Models" (scaling-law section 3.2.1 and training recipe 3.4.1, as quoted in search results) and the official Llama 3.1 model card (GPU-hours and emissions). Architecture numbers come from spec pages consistent with Meta's release. The dollar cost is our estimate from GPU-hours and typical rental prices; Meta has not published one. The IsoFLOP explorer uses the published Chinchilla fit (Hoffmann et al., 2022), not Meta's own fitted constants; the two-step prediction and cost-of-ownership charts are illustrative. See Reading the numbers.

🎯 One number decides everything

After assembling the software (Part 1) and the data (Part 2), Meta had to commit compute. Llama 3's flagship was pre-trained with about 3.8×10²⁵ floating-point operations, almost 50 times the largest Llama 2. Everything else follows from how that budget is split between model size and training data.

3.8×10²⁵FLOPs to pre-train Llama 3 405B (~50× the largest Llama 2)
405Bparameters (scaling laws suggested ~402B)
15.6Ttraining tokens (scaling laws suggested ~16.55T)
30.84MH100 GPU-hours for 405B (Llama 3.1 model card)

🧮 The training compute equation

For a dense transformer, a widely used approximation is training FLOPs ≈ 6 × parameters × tokens: about 2 operations per parameter per token for the forward pass and 4 for the backward pass. For Llama 3 405B: 6 × 405×10⁹ × 15.6×10¹² ≈ 3.79×10²⁵, matching Meta's figure.

Turning FLOPs into GPU-hours requires the chip's peak speed and how much of it the job actually achieves, known as model FLOPs utilisation (MFU). An H100 peaks at roughly 989 teraflops in dense BF16. Meta's model card reports 30.84 million H100-hours for 405B; dividing the training FLOPs by that implies an average of about 35% of peak, consistent with the 38–43% MFU reported for the main training stages once failures, restarts and the slower long-context phase are included.

🎛️ Plan your own run

🧮 Frontier run calculator

Defaults reproduce Llama 3 405B. Cost and energy use illustrative assumptions you can change.

GPU

FLOPs = 6ND. GPU-hours = FLOPs ÷ (peak dense BF16 × MFU) ÷ 3,600. Peak: H100 ~989 TF, B200 ~2,250 TF (vendor dense BF16). Energy: GPU-hours × GPU power × 1.3 (servers, networking, cooling). CO₂ at 0.35 t/MWh (illustrative grid average). Real runs add experiments, ablations and post-training, often more than the main run itself.

📈 Scaling laws: from Kaplan to Chinchilla to Llama 3

A scaling law describes how a model's loss falls as you add parameters, data and compute, usually as a smooth power law. In 2020, OpenAI's Kaplan et al. concluded that, for a given budget, models should grow faster than their data, which is how GPT-3 ended up with 175 billion parameters trained on about 300 billion tokens. In 2022 DeepMind's Chinchilla paper reversed that: with more careful experiments, the compute-optimal recipe was roughly 20 tokens per parameter, and its 70B model trained on 1.4T tokens beat much larger models trained on less data.

Meta fitted its own law for Llama 3. It trained many models across compute budgets from 6×10¹⁸ to 10²² FLOPs and, at each budget, found the model size with the lowest loss (the minimum of a fitted parabola on an IsoFLOP curve, a curve of loss against model size at fixed compute). Extrapolating those optima to 3.8×10²⁵ FLOPs suggested 402 billion parameters trained on 16.55 trillion tokens, about 41 tokens per parameter. Meta chose 405B on 15.6T, reasoning that IsoFLOP curves become flatter near the minimum as budgets grow, so the result is robust to small deviations.

🔍 IsoFLOP explorer

📉 Loss vs model size at a fixed compute budget

Each curve fixes compute; along it, bigger models see fewer tokens. The dot marks the best size. Uses the published Chinchilla fit, not Meta's constants, so numbers differ from Llama 3's.

L(N, D) = 1.69 + 406.4 / N^0.34 + 410.7 / D^0.28, with D = C ÷ 6N (Hoffmann et al. 2022, approach 3). Grey curves show neighbouring budgets. Notice the curves flatten near the minimum at larger budgets, Meta's reason for rounding to 405B.

🔮 Predicting benchmark scores before training

Loss is not what users care about; benchmark accuracy is. Meta's paper describes a two-step method to forecast it. First, it relates the model's negative log-likelihood (NLL) on the correct answers of a benchmark to training FLOPs, a near-linear relationship fitted only on models up to 10²² FLOPs. Second, it maps that likelihood to accuracy with a sigmoid curve, fitted using the scaling-law models plus earlier Llama 2 models. Chaining the two extrapolated across four orders of magnitude of compute and only slightly underestimated the flagship's final score on the ARC Challenge reasoning benchmark.

📊 The two-step forecast (illustrative)

Illustrative shapes of the method described in the Llama 3 paper, not Meta's data.

🏋️ Why train small models far past "optimal"

Compute-optimal only minimises training cost. Once a model is deployed, every query costs roughly 2 × parameters in FLOPs per token (see The Great Flip), so a smaller model that is trained longer can be far cheaper over its lifetime. Llama 3 8B was trained on about 15 trillion tokens, around 1,875 tokens per parameter, nearly 100 times the Chinchilla ratio. The 405B flagship sits at about 38.

💰 Lifetime compute: train cost vs serving cost

Two illustrative configurations. Model quality is not equal; this shows how costs shift with usage.

Training = 6ND; serving = 2N per token. A: 70B on 1.4T tokens (Chinchilla-style). B: 8B on 15T tokens (Llama 3 8B-style).

🏗️ Choosing the architecture

Meta kept Llama 3 a dense transformer, rather than a mixture-of-experts (MoE) model like DeepSeek-V3, citing training stability at scale. (Its Llama 4 family later moved to MoE.)

🗂️ Llama 3.1 405B at a glance

PropertyValueWhy it matters
Parameters405 billion (dense)Every parameter is used for every token
Layers126Depth; also sets how the model can be split across pipeline stages (Part 4)
Model width16,384Size of each token's internal representation
Attention heads128, with 8 key-value heads (grouped-query attention)Sharing key-value heads shrinks the KV cache, cutting inference memory
Vocabulary~128K tokensBetter compression and multilingual coverage (Part 2)
Context window128K tokens (Llama 3.1)Reached in a late long-context training stage

Spec pages (ApX, Nvidia NeMo docs) consistent with Meta's release.

📝 The training recipe

The plan also fixes the optimisation schedule. According to the paper, Llama 3 405B used the AdamW optimiser with a peak learning rate of 8×10⁻⁵, a warm-up, then cosine decay to 8×10⁻⁷ over about 1.2 million steps. The batch size started small for stability and grew, reaching 16 million tokens per step after 2.87 trillion tokens. Then come the stages from Part 2: main pre-training, long-context extension and annealing. Part 5 covers what happens when the plan meets reality.

🧾 The bill

⚡ GPU-hours and emissions, Llama 3.1 family

Llama 3.1 model card: 39.3M H100-80GB GPU-hours in total at 700 W per GPU; location-based emissions 11,390 t CO₂e; market-based 0 because Meta matches its electricity use with renewable energy purchases.

Meta does not publish dollar costs. At rental prices of roughly $2–4 per H100-hour, 30.84 million GPU-hours for 405B would be about $60–120 million of compute, our rough estimate. Meta owns its clusters, so its own marginal cost differs, and the main run is only part of the total: experiments, failed runs, smaller models and post-training add substantially (see Who Pays for the AI Build-Out?).

🦙 Llama 3 case file: the plan Budget 3.8×10²⁵ FLOPs → scaling-law optimum 402B / 16.55T → chosen 405B / 15.6T (~38 tokens per parameter) · dense transformer, 126 layers, GQA · AdamW, peak LR 8×10⁻⁵, batch up to 16M tokens, ~1.2M steps · up to 16,384 H100s · 30.84M GPU-hours · 8,930 t CO₂e (location-based).

🕰️ Timeline

Jan 2020 · Kaplan et al. (OpenAI) publish neural scaling laws.
May 2020 · GPT-3: 175B parameters, ~300B tokens.
Mar 2022 · Chinchilla (DeepMind): ~20 tokens per parameter is compute-optimal.
Feb 2023 · Llama 1 trains small models well beyond Chinchilla-optimal for cheaper inference.
Jul 2023 · Llama 2.
Dec 2023 · "Beyond Chinchilla-Optimal" formalises adding inference demand to scaling laws.
Apr 2024 · Llama 3 8B and 70B released, 8B trained on ~15T tokens.
Jul 2024 · Llama 3.1 405B and the paper with its scaling-law methodology.
Apr 2025 · Llama 4 moves Meta to mixture-of-experts.

🔭 What to watch

🎯 The bottom line Planning a frontier run is applied forecasting. Meta started from a compute budget of 3.8×10²⁵ FLOPs, used small experiments to fit scaling laws that pointed to roughly 400 billion parameters and 16 trillion tokens, forecast benchmark scores before training, and translated it all into a GPU schedule of 30.84 million H100-hours. Smaller siblings were deliberately trained far past the "optimal" point because serving costs dominate over a model's life. The same arithmetic now governs every frontier lab's biggest decisions.

🔢 Reading the numbers

📚 References and papers