Splitting a Giant: How Llama 3 405B Was Divided Across 16,384 GPUs
Llama 3 405B does not fit on any single GPU, or even a single server. Training it means cutting the model, the data and even individual sequences into pieces, spreading them over thousands of H100s, and keeping all those pieces in step without drowning the network. Meta did it with "4D parallelism": tensor, context, pipeline and data parallelism layered on top of each other and matched to the shape of its network. This part explains each dimension, Meta's exact configurations, the memory arithmetic, the pipeline "bubble" problem, and why 38โ43% of peak is a good result.
๐งณ The model does not fit
Part 3 fixed the plan: 405 billion parameters, 15.6 trillion tokens, up to 16,384 H100 GPUs. Now the problem is physical. In 16-bit precision the weights alone take about 810 GB. Training also needs gradients and optimiser state: with standard mixed-precision Adam, roughly 16 bytes per parameter, about 6.5 TB, before counting the activations saved during each forward pass. Each H100 has 80 GB of memory.
So the model has to be split, and not in one way but four at once. For the basic concepts, see Scaling Laws, GPUs and Distributed Training; this article focuses on how Meta actually did it.
โ๏ธ The four (and a half) ways to split
๐ฌ Parallelism visualiser
Pick a strategy to see what each of four GPUs holds and what has to cross the network.
๐บ๏ธ Llama 3's layout: TP 8 ร CP ร PP 16 ร DP
Meta grouped GPUs in the order [TP, CP, PP, DP] and implemented data parallelism with FSDP, which shards parameters, gradients and optimiser state across data-parallel ranks. Table 4 of the paper lists the configuration for each stage of pre-training. In every row, multiplying the four dimensions gives the GPU count, and the global batch stays at about 16 million tokens.
๐ Configurations by training stage
Click a row to check the arithmetic.
Llama 3 paper Table 4: 8,192-GPU row quoted in search results; 16K-GPU throughput (400 and 380 TFLOPs/GPU) also reported in Meta's ISCA 2025 paper; long-context row (CP 16, DP 8) from secondary reproductions of the table.
Doubling from 8,192 to 16,384 GPUs doubled data parallelism (64 โ 128), but to keep the global batch at 16M tokens each data-parallel group had to process half as many sequences, so pipelines had less work to overlap and MFU fell from 43% to 41%. Long-context training kept the GPU count but redistributed it: context parallelism of 16 splits each 131,072-token sequence across 16 GPU groups, leaving only 8 data-parallel replicas.
๐ Matching the split to the network
The order is not arbitrary. Each dimension communicates with a different intensity, so Meta placed the chattiest one on the fastest links and the most tolerant one on the slowest (see the network map in Part 1).
๐ From NVLink to the building
Innermost to outermost, for the 8,192-GPU configuration.
๐งฎ Memory budget per GPU
๐๏ธ Will it fit in 80 GB?
Simplified model. Defaults approximate Llama 3 405B's main stage; Meta's actual memory breakdown is not published.
Weights 2 bytes/param, gradients 2, optimiser 12 (FP32 master copy + two Adam moments), each divided by TP ร PP and, where sharded, by DP. Activations โ layers per stage ร (sequence รท CP) ร hidden ร bytes-per-token-per-layer รท TP ร PP micro-batches in flight (34 bytes with none recomputed, ~8 with selective recomputation, ~2 with full). Hidden size and layers scale with model size. Ignores buffers, fragmentation and embeddings.
๐ซง Pipeline bubbles and schedules
Pipeline parallelism gives each group of GPUs a slice of layers (Llama 3: 126 layers across 16 stages). The catch is the bubble: at the start of each batch, later stages wait for the first micro-batch to reach them, and at the end, early stages sit idle. Splitting the batch into more micro-batches shrinks the bubble, and interleaved schedules give each GPU several smaller slices so the pipeline fills faster. Meta modified the interleaved schedule so the batch no longer had to be a multiple of the number of pipeline stages, accounted for the first stage's extra memory (embeddings and warm-up micro-batches), and co-designed layer counts with the parallelism layout to balance compute and memory across stages.
๐๏ธ Bubble simulator
GPipe-style schedule: forward passes (blue) then backward passes (purple). Grey cells are idle time.
Bubble fraction โ (p โ 1) รท (v ร m + p โ 1) for p stages, m micro-batches and v virtual stages (Narayanan et al., 2021). The chart draws the non-interleaved case; the interleaving slider only changes the computed fraction.
๐ Long context: context parallelism
Attention cost grows with the square of sequence length, and activations grow linearly, so a 131,072-token sequence cannot be handled by one GPU group. Context parallelism splits each sequence along its length across GPUs. Meta used an all-gather approach: each GPU computes queries for its own chunk but gathers the keys and values it needs from the others, and the design supports document masks, so separate documents packed into one sequence do not attend to each other. Related techniques such as ring attention pass key-value blocks around a ring instead. Even so, throughput fell to about 380 TFLOPs per GPU (โ38% MFU) at 131K tokens.
๐ What limits efficiency
๐ Throughput by configuration
BF16 MFU and TFLOPs per GPU from the Llama 3 paper and Meta's ISCA 2025 paper; H100 dense BF16 peak โ 989 TFLOPs.
- Communication that cannot be hidden behind computation, especially across the oversubscribed top network layer.
- Pipeline bubbles, larger when each data-parallel group has fewer micro-batches.
- Imbalance: uneven layers per stage, embeddings on the first stage, and uneven document lengths with document masks (a problem the later WLB-LLM paper targets).
- Stragglers: one slow GPU or link slows all 16,384, because every step waits for the slowest.
Against those forces, 38โ43% of theoretical peak on 16,384 GPUs is a strong result; the remaining gap is why labs keep investing in kernels, communication libraries and network design.
๐ฐ๏ธ Timeline
๐ญ What to watch
- Bigger NVLink domains (72+ GPUs per rack, see The Interconnect Wars) letting tensor and expert parallelism span more GPUs.
- Expert parallelism for mixture-of-experts models, now standard at frontier labs.
- Automatic parallelism planners that search layouts instead of hand-tuning them.
- Next in this series: Part 5 runs the job: precision, checkpoints, failures and loss spikes over 54 days.
๐ข Reading the numbers
- Table 4 values: 8,192-GPU row and 16K-GPU throughput are directly quoted; the long-context row's CP 16 and DP 8 come from secondary reproductions, consistent with 8ร16ร16ร8 = 16,384 GPUs.
- The 6.5 TB and 16 bytes/parameter are standard mixed-precision Adam estimates, not Meta's published figures.
- The memory calculator uses textbook approximations (Korthikanti et al. for activations) and ignores many real overheads.
- Which FSDP sharding mode Meta used is not confirmed in our sources.
- The bubble formula is the standard one for interleaved schedules; Meta's modified schedule differs in detail.
๐ References and papers
- Llama 3
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024), section 3.3.2 and Table 4
- "Scaling Llama 3 Training with Efficient Parallelism Strategies" (ISCA 2025)
- Llama 3 Table 4: scaling configurations and MFU (reproduction)
- "WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training" (2025)
- "Context Parallelism for Scalable Million-Token Inference" (Meta, 2024)
- Parallelism foundations
- Shoeybi et al.: "Megatron-LM" (2019)
- Narayanan et al.: "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM" (2021)
- Korthikanti et al.: "Reducing Activation Recomputation in Large Transformer Models" (2022)
- Rajbhandari et al.: "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (2020)
- Huang et al.: "GPipe" (2019)
- Harlap et al.: "PipeDream: Fast and Efficient Pipeline Parallel DNN Training" (2018)
- Zhao et al.: "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel" (2023)
- Liu et al.: "Ring Attention with Blockwise Transformers for Near-Infinite Context" (2023)
- DeepSeek-AI: "DeepSeek-V3 Technical Report" (2024)
- Related on this site
- Part 1: The Software Stack ยท Part 3: Planning the Run ยท Scaling Laws, GPUs and Distributed Training ยท Debugging a Distributed Job