Planning the Run: How Meta Sized Llama 3 405B with Scaling Laws and a 3.8×10²⁵ FLOP Budget
A frontier training run is a bet worth tens of millions of dollars, and most of its key decisions are made before the first GPU starts. How big should the model be? How many tokens should it read? How many GPUs, for how long, and what will it score when it is done? Meta answered those questions for Llama 3 with small-scale experiments and scaling laws, and its paper shows the arithmetic. This part walks through it: the compute equation, IsoFLOP curves, predicting benchmark scores in advance, why small models are deliberately "overtrained", and the final bill in GPU-hours and carbon.
🎯 One number decides everything
After assembling the software (Part 1) and the data (Part 2), Meta had to commit compute. Llama 3's flagship was pre-trained with about 3.8×10²⁵ floating-point operations, almost 50 times the largest Llama 2. Everything else follows from how that budget is split between model size and training data.
🧮 The training compute equation
For a dense transformer, a widely used approximation is training FLOPs ≈ 6 × parameters × tokens: about 2 operations per parameter per token for the forward pass and 4 for the backward pass. For Llama 3 405B: 6 × 405×10⁹ × 15.6×10¹² ≈ 3.79×10²⁵, matching Meta's figure.
Turning FLOPs into GPU-hours requires the chip's peak speed and how much of it the job actually achieves, known as model FLOPs utilisation (MFU). An H100 peaks at roughly 989 teraflops in dense BF16. Meta's model card reports 30.84 million H100-hours for 405B; dividing the training FLOPs by that implies an average of about 35% of peak, consistent with the 38–43% MFU reported for the main training stages once failures, restarts and the slower long-context phase are included.
🎛️ Plan your own run
🧮 Frontier run calculator
Defaults reproduce Llama 3 405B. Cost and energy use illustrative assumptions you can change.
FLOPs = 6ND. GPU-hours = FLOPs ÷ (peak dense BF16 × MFU) ÷ 3,600. Peak: H100 ~989 TF, B200 ~2,250 TF (vendor dense BF16). Energy: GPU-hours × GPU power × 1.3 (servers, networking, cooling). CO₂ at 0.35 t/MWh (illustrative grid average). Real runs add experiments, ablations and post-training, often more than the main run itself.
📈 Scaling laws: from Kaplan to Chinchilla to Llama 3
A scaling law describes how a model's loss falls as you add parameters, data and compute, usually as a smooth power law. In 2020, OpenAI's Kaplan et al. concluded that, for a given budget, models should grow faster than their data, which is how GPT-3 ended up with 175 billion parameters trained on about 300 billion tokens. In 2022 DeepMind's Chinchilla paper reversed that: with more careful experiments, the compute-optimal recipe was roughly 20 tokens per parameter, and its 70B model trained on 1.4T tokens beat much larger models trained on less data.
Meta fitted its own law for Llama 3. It trained many models across compute budgets from 6×10¹⁸ to 10²² FLOPs and, at each budget, found the model size with the lowest loss (the minimum of a fitted parabola on an IsoFLOP curve, a curve of loss against model size at fixed compute). Extrapolating those optima to 3.8×10²⁵ FLOPs suggested 402 billion parameters trained on 16.55 trillion tokens, about 41 tokens per parameter. Meta chose 405B on 15.6T, reasoning that IsoFLOP curves become flatter near the minimum as budgets grow, so the result is robust to small deviations.
🔍 IsoFLOP explorer
📉 Loss vs model size at a fixed compute budget
Each curve fixes compute; along it, bigger models see fewer tokens. The dot marks the best size. Uses the published Chinchilla fit, not Meta's constants, so numbers differ from Llama 3's.
L(N, D) = 1.69 + 406.4 / N^0.34 + 410.7 / D^0.28, with D = C ÷ 6N (Hoffmann et al. 2022, approach 3). Grey curves show neighbouring budgets. Notice the curves flatten near the minimum at larger budgets, Meta's reason for rounding to 405B.
🔮 Predicting benchmark scores before training
Loss is not what users care about; benchmark accuracy is. Meta's paper describes a two-step method to forecast it. First, it relates the model's negative log-likelihood (NLL) on the correct answers of a benchmark to training FLOPs, a near-linear relationship fitted only on models up to 10²² FLOPs. Second, it maps that likelihood to accuracy with a sigmoid curve, fitted using the scaling-law models plus earlier Llama 2 models. Chaining the two extrapolated across four orders of magnitude of compute and only slightly underestimated the flagship's final score on the ARC Challenge reasoning benchmark.
📊 The two-step forecast (illustrative)
Illustrative shapes of the method described in the Llama 3 paper, not Meta's data.
🏋️ Why train small models far past "optimal"
Compute-optimal only minimises training cost. Once a model is deployed, every query costs roughly 2 × parameters in FLOPs per token (see The Great Flip), so a smaller model that is trained longer can be far cheaper over its lifetime. Llama 3 8B was trained on about 15 trillion tokens, around 1,875 tokens per parameter, nearly 100 times the Chinchilla ratio. The 405B flagship sits at about 38.
💰 Lifetime compute: train cost vs serving cost
Two illustrative configurations. Model quality is not equal; this shows how costs shift with usage.
Training = 6ND; serving = 2N per token. A: 70B on 1.4T tokens (Chinchilla-style). B: 8B on 15T tokens (Llama 3 8B-style).
🏗️ Choosing the architecture
Meta kept Llama 3 a dense transformer, rather than a mixture-of-experts (MoE) model like DeepSeek-V3, citing training stability at scale. (Its Llama 4 family later moved to MoE.)
🗂️ Llama 3.1 405B at a glance
| Property | Value | Why it matters |
|---|---|---|
| Parameters | 405 billion (dense) | Every parameter is used for every token |
| Layers | 126 | Depth; also sets how the model can be split across pipeline stages (Part 4) |
| Model width | 16,384 | Size of each token's internal representation |
| Attention heads | 128, with 8 key-value heads (grouped-query attention) | Sharing key-value heads shrinks the KV cache, cutting inference memory |
| Vocabulary | ~128K tokens | Better compression and multilingual coverage (Part 2) |
| Context window | 128K tokens (Llama 3.1) | Reached in a late long-context training stage |
Spec pages (ApX, Nvidia NeMo docs) consistent with Meta's release.
📝 The training recipe
The plan also fixes the optimisation schedule. According to the paper, Llama 3 405B used the AdamW optimiser with a peak learning rate of 8×10⁻⁵, a warm-up, then cosine decay to 8×10⁻⁷ over about 1.2 million steps. The batch size started small for stability and grew, reaching 16 million tokens per step after 2.87 trillion tokens. Then come the stages from Part 2: main pre-training, long-context extension and annealing. Part 5 covers what happens when the plan meets reality.
🧾 The bill
⚡ GPU-hours and emissions, Llama 3.1 family
Llama 3.1 model card: 39.3M H100-80GB GPU-hours in total at 700 W per GPU; location-based emissions 11,390 t CO₂e; market-based 0 because Meta matches its electricity use with renewable energy purchases.
Meta does not publish dollar costs. At rental prices of roughly $2–4 per H100-hour, 30.84 million GPU-hours for 405B would be about $60–120 million of compute, our rough estimate. Meta owns its clusters, so its own marginal cost differs, and the main run is only part of the total: experiments, failed runs, smaller models and post-training add substantially (see Who Pays for the AI Build-Out?).
🕰️ Timeline
🔭 What to watch
- Scaling laws for MoE, data quality and synthetic data, which change the optimal ratios.
- Inference-aware planning: ever more tokens per parameter as serving dominates compute.
- Reinforcement-learning compute becoming a planned share of the budget alongside pre-training.
- Next in this series: Part 4 shows how a 405B model is split across 16,384 GPUs.
🔢 Reading the numbers
- 3.8×10²⁵ FLOPs, 402B / 16.55T, the 6×10¹⁸–10²² experiment range and the two-step method come from the Llama 3 paper via search results.
- GPU-hours and emissions are from the official Llama 3.1 model card.
- The ~35% average utilisation is our calculation from those figures; 38–43% MFU per stage is reported in summaries of the paper.
- The $60–120M cost is our estimate.
- The IsoFLOP explorer uses Chinchilla's published constants, so its optimum differs from Meta's.
- Calculators and charts are simplified illustrations.
📚 References and papers
- Llama 3
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024), sections 3.2 and 3.4
- Llama 3.1 model card (GPU-hours, emissions)
- Llama 3 model card
- ApX: Llama 3.1 405B specifications
- Nvidia Megatron Bridge: Llama 3 models
- Scaling laws
- Kaplan et al.: "Scaling Laws for Neural Language Models" (2020)
- Hoffmann et al.: "Training Compute-Optimal Large Language Models" (Chinchilla, 2022)
- Sardana et al.: "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws" (2023)
- Brown et al.: "Language Models are Few-Shot Learners" (GPT-3, 2020)
- Touvron et al.: "LLaMA: Open and Efficient Foundation Language Models" (2023)
- LessWrong: On AI scaling (discussion of overtraining)
- Related on this site
- Scaling Laws, GPUs and Distributed Training · Architectures & Hyperparameters · Part 2: The Data Refinery · The Great Flip