Architecture and Parallelism
Part 3 decided on a mixture-of-experts model with about 12B active parameters, trained on 10T tokens. Now Gopala has to design it for real: how many layers and experts, what attention, what tokenizer, and how to cut the model into pieces so that 1,024 GPUs can train it together without running out of memory or waiting on the network. This part counts the parameters with a script, plans memory per GPU with another, explains the four kinds of parallelism in plain terms, and ends with the small ablation runs that de-risk the design before the main run.
moe_params.py and memory_plan.py were run (real output below); the architecture is our design for the story, and the activation-memory estimate is a rough rule of thumb, not a measurement. launch_megatron.sh was not executed for this article. Ablation results in the story are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 24โ38
- Writes
moe_params.py; fixes the shape: 40 layers, 64 experts, 8 active per token, one shared expert. - Chooses grouped-query attention and a 128k-token tokenizer.
- Writes
memory_plan.py; picks TP 2 ร EP 8 ร DP 512. - Runs ablations at 1โ3B active parameters on 64โ256 GPUs to test router settings, the data mix (Part 8) and stability.
- Freezes the architecture and the launch configuration; measures throughput on the full block; books the main run for month 9.
2. Why mixture-of-experts
- Compute follows active parameters; knowledge capacity follows total parameters more closely.
- A dense 12B model and an MoE with 12B active parameters cost about the same to train per token.
- The MoE holds several times more weights, so it can store more.
- The pattern is proven in public models.
- Mixtral 8x7B: about 46.7B total parameters, about 12.9B used per token (Mistral AI).
- DeepSeek-V3: 671B total, 37B active, trained on 14.8T tokens for about 2.788M H800 GPU-hours (DeepSeek).
- The costs are real too.
- All weights must sit in GPU memory even though few are used per token.
- Tokens must travel between GPUs to reach their experts (all-to-all traffic).
- Routers can collapse onto a few experts unless balanced.
Dense vs mixture-of-experts at the same compute per token
Real output of moe_params.py for our design.
3. Counting the parameters
"""moe_params.py โ count total and active parameters of Maestro-1's MoE.
A mixture-of-experts layer holds many feed-forward "experts" but each
token uses only a few, so compute follows ACTIVE parameters and memory
follows TOTAL parameters. Configuration is our design for this story.
"""
CFG = dict(layers=40, d=4096, heads=32, kv_heads=8, vocab=128_000,
experts=64, top_k=8, shared=1, d_ff_expert=2048, dense_first=2, d_ff_dense=14336)
def count(c):
d, hd = c["d"], c["d"] // c["heads"]
attn = d * d * 2 + d * hd * c["kv_heads"] * 2 # Q,O full; K,V grouped (GQA)
expert = 3 * d * c["d_ff_expert"] # SwiGLU: gate, up, down
dense_ffn = 3 * d * c["d_ff_dense"]
router = d * c["experts"]
moe_layers = c["layers"] - c["dense_first"]
emb = c["vocab"] * d * 2 # input + output (untied)
total = (c["layers"] * attn + c["dense_first"] * dense_ffn
+ moe_layers * ((c["experts"] + c["shared"]) * expert + router) + emb)
active = (c["layers"] * attn + c["dense_first"] * dense_ffn
+ moe_layers * ((c["top_k"] + c["shared"]) * expert + router) + emb)
return attn, expert, total, active, emb
def main():
attn, expert, total, active, emb = count(CFG)
print(f"per layer: attention {attn/1e6:.1f}M, one expert {expert/1e6:.1f}M")
print(f"total parameters {total/1e9:6.1f}B (memory, checkpoints)")
print(f"active per token {active/1e9:6.1f}B (compute: 6 x active x tokens)")
print(f" of which embeddings {emb/1e9:.1f}B; active without embeddings {(active-emb)/1e9:.1f}B")
print(f"sparsity: each token uses {active/total:.0%} of the weights")
for k in (2, 4, 6, 8):
c = dict(CFG, top_k=k)
print(f" top_k={k}: active {count(c)[3]/1e9:5.1f}B")
if __name__ == "__main__":
main()- Reading it.
- One expert is a 25M-parameter SwiGLU feed-forward block; each MoE layer holds 64 of them plus one shared expert that every token uses.
- With 8 experts active per token, the model uses 11.7B parameters per token out of 65.2B: the "12B active" that Part 3's budget assumed.
- The top-k knob trades cost for capacity per token.
- Going from 8 to 2 active experts would halve compute per token but leave each token far less model to use.
- He keeps 8 because the budget was built on ~12B active.
4. Design choices, and why
| Choice | Decision | Reason |
|---|---|---|
| Attention | Grouped-query attention, 32 query heads, 8 key/value groups | Smaller KV cache at serving time (Part 15) with little quality loss; used by Llama 3 |
| Context length | 8,192 in pre-training, extended later | Long-context training is expensive; extend in a short later phase, as Llama 3 did |
| Tokenizer | 128k vocabulary, trained on the blend, code-aware | Fewer tokens per line of code means cheaper training and serving |
| First layers | 2 dense layers before the MoE layers | A common choice for stability in early layers |
| Router balance | Auxiliary load-balancing loss (small coefficient) | Stops experts collapsing; DeepSeek-V3 shows auxiliary-loss-free alternatives exist |
| Precision | BF16 with FP32 master weights | Proven at scale; FP8 training (as in DeepSeek-V3) is a later option |
| Positional encoding | RoPE | Standard, extends to longer contexts |
5. Four ways to split the work
The four kinds of parallelism
Real systems combine them. Llama 3 405B (dense) used tensor, pipeline, context and data parallelism together.
- Each split trades memory for communication.
- DP is the simplest but every copy needs the full model, unless optimizer state is sharded (ZeRO / distributed optimizer).
- TP cuts memory per GPU but sends data every layer; NVLink inside a node can carry it, the network cannot.
- PP cuts memory but leaves "bubbles" where stages wait.
- EP keeps each GPU's expert set small but sends every token across the network twice per MoE layer.
- The site's ML Layer article on parallelism libraries goes deeper on each, and Producing Intelligence Part 4 shows how Meta combined them for Llama 3.
6. Will it fit? A memory plan
- Mixed-precision Adam needs about 16 bytes per parameter.
- 2 bytes BF16 weights, 2 bytes gradients, 4 bytes FP32 master weights, 8 bytes for Adam's two moments.
- For 65.2B parameters that is over a terabyte, against 80 GB per H100.
- Sharding and splitting bring it down.
- The distributed optimizer spreads the 12 optimizer bytes across data-parallel ranks.
- TP, PP and EP split the weights themselves.
"""memory_plan.py โ which parallel layout fits Maestro-1 on 1,024 H100s?
Memory per GPU = weights + gradients + optimizer state (sharded) +
activations. Mixed-precision Adam keeps ~16 bytes per parameter in total
(bf16 weights and grads, fp32 master weights, two fp32 moments);
ZeRO-1-style sharding spreads the 12 optimizer bytes across data-parallel
ranks. Activation memory is a rough per-token estimate (our assumption).
"""
TOTAL, NON_EXPERT = 65.2e9, 4.1e9 # from moe_params.py: all params; params not split by EP (attention, dense, shared expert, embeddings)
GPUS, HBM = 1024, 80e9
SEQ, MICRO = 8192, 1
ACT_BYTES_PER_TOKEN_LAYER = 34 * 4096 * 2 / 8 # rough, with selective recompute (assumption)
LAYERS = 40
def per_gpu(tp, pp, ep):
dp = GPUS // (tp * pp)
expert_params = TOTAL - NON_EXPERT
params = NON_EXPERT / (tp * pp) + expert_params / (tp * pp * ep)
w_g = params * 4 # bf16 weights + bf16 grads
opt = params * 12 / dp # fp32 master + 2 moments, sharded over DP
act = SEQ * MICRO * ACT_BYTES_PER_TOKEN_LAYER * (LAYERS / pp) / tp * pp # 1F1B keeps ~pp micro-batches
return dp, (w_g + opt + act) / 1e9
def main():
print(f"{'TP':>3s} {'PP':>3s} {'EP':>3s} {'DP':>4s} {'GB/GPU':>7s} fits 80 GB with 10% headroom?")
for tp, pp, ep in [(1, 1, 1), (1, 1, 8), (2, 1, 8), (1, 2, 8), (4, 1, 8), (1, 4, 16), (2, 2, 8), (8, 1, 8)]:
dp, gb = per_gpu(tp, pp, ep)
if dp % ep:
continue
print(f"{tp:3d} {pp:3d} {ep:3d} {dp:4d} {gb:7.1f} {'yes' if gb < 0.9 * HBM / 1e9 else 'no'}")
print("rule: keep TP inside one 8-GPU node (NVLink); EP across nodes costs all-to-all traffic")
if __name__ == "__main__":
main()Memory per GPU by layout (80 GB available)
Real output of memory_plan.py (activation memory is a rough estimate).
- Why TP 2 ร EP 8 and not the smallest number.
- Fitting is necessary, not sufficient: every extra split adds communication.
- TP 2 stays inside a node on NVLink; EP 8 keeps the all-to-all within groups of 8 GPUs where possible.
- The headroom (about 29 GB used of 80) leaves room for longer sequences, bigger micro-batches or less recomputation, which buys speed.
7. Try a layout
Parallel layout planner for 1,024 GPUs (our model)
8. How fast will it run?
- Model FLOPs utilisation (MFU) is the honest speed number.
- The share of the GPUs' peak arithmetic that goes into the model's useful maths.
- Meta reported 38โ43% MFU for Llama 3 405B (BF16) on up to 16,384 H100s.
- MoE models usually reach lower MFU than dense ones.
- Smaller matrices per expert and all-to-all communication leave GPUs waiting.
- Part 3's budget assumed 30%, a deliberately conservative figure.
- He measures before he believes.
- On the full block, a 2-hour run with the frozen configuration measures tokens per second and MFU.
- In the story the measured throughput is about 32% MFU, slightly better than the plan's 30%, after two changes found in profiling: overlapping the expert all-to-all with computation, and raising the micro-batch to use the memory headroom.
- These are story values, not measurements.
9. Ablations: small runs that de-risk the big one
The ablation ladder
Each rung tests what the next rung would be too expensive to get wrong.
- What he ablates, and what he deliberately does not.
- Yes: data mixture, learning rate and batch size, router balance coefficient, number of active experts.
- No: exotic architecture ideas. A start-up's advantage is its data and its loop, not a new layer type.
- Scaling rules from Part 2 apply again.
- Fit loss against size on the ladder and predict the 12B-active model; compare with the first days of the main run.
- Part 2 showed small ladders extrapolate imperfectly, so the prediction is a check, not a promise.
10. The communication budget
- Every training step moves data three ways.
- Inside a node over NVLink: tensor-parallel partial results, every layer (fast, up to 900 GB/s per GPU on H100).
- Across nodes over InfiniBand: expert all-to-all twice per MoE layer, and the data-parallel gradient reduction once per step (50 GB/s per GPU per direction at 400 Gb/s).
- To storage: checkpoints every 500 steps (Part 7).
- The design rule that follows.
- Put the chattiest split (TP) on the fastest link (NVLink).
- Overlap the slow splits (EP all-to-all, DP reduce) with computation so GPUs are not idle while data moves.
- Profile the first hour of every run: a single slow link (Part 6) shows up as a fat all-to-all bar in the trace.
Where each kind of traffic travels
Fast links carry frequent traffic; slow links carry traffic that can overlap with computation.
11. The tokenizer, the decision that cannot be undone
- The tokenizer is chosen once and lives forever.
- Every shard of Part 8's data is tokenised with it, and every serving request in Phase 4 pays for its efficiency.
- His choices.
- A 128k-entry vocabulary, the size Llama 3 moved to from Llama 2's 32k, trained on a sample of his own blend so code and engineering text are compressed well.
- Byte fallback, so no input is ever unrepresentable.
- Digits split individually, a common choice that helps arithmetic.
- How he checks it.
- Tokens per line on held-out code and docs, compared with an open tokenizer of similar size; fewer tokens means cheaper training and serving for the same text.
12. What could still go wrong
| Risk | Early sign | Response |
|---|---|---|
| Router collapse | A few experts take most tokens; others idle | Raise the balance coefficient; restart from the last balanced checkpoint |
| Loss spikes | Sudden jumps in loss (Part 10) | Rewind and skip batches; lower learning rate if repeated |
| Throughput drop | Step time rises on one rank | Straggler detection (Part 7) and node swap |
| Memory fragmentation | Out-of-memory after hours | Headroom kept on purpose; allocator settings tested in ablations |
13. The launch configuration
- Everything above ends in one file kept in git.
- Model shape, parallel layout, batch size, learning-rate schedule (Part 10), checkpointing, data blend (Part 8).
- The sketch below uses Megatron-LM-style flags; it was written for this article and not run.
#!/bin/bash # launch_megatron.sh โ Maestro-1 layout on 128 nodes (sketch; flag names follow # Megatron-LM's documented arguments, verify against your version). #SBATCH --partition=main --nodes=128 --gpus-per-node=8 --exclusive export NCCL_IB_HCA=mlx5 CUDA_DEVICE_MAX_CONNECTIONS=1 srun python pretrain_gpt.py \ --num-layers 40 --hidden-size 4096 --num-attention-heads 32 \ --group-query-attention --num-query-groups 8 --swiglu \ --num-experts 64 --moe-router-topk 8 --moe-shared-expert-intermediate-size 2048 \ --moe-ffn-hidden-size 2048 --moe-router-load-balancing-type aux_loss --moe-aux-loss-coeff 1e-3 \ --tensor-model-parallel-size 2 --pipeline-model-parallel-size 1 \ --expert-model-parallel-size 8 --sequence-parallel --use-distributed-optimizer \ --seq-length 8192 --global-batch-size 2048 --micro-batch-size 1 \ --bf16 --recompute-granularity selective \ --lr 3e-4 --min-lr 3e-5 --lr-warmup-iters 2000 --train-iters 600000 \ --save /checkpoints/maestro-1 --save-interval 500 --async-save \ --data-path $(cat /data/mixture/maestro1_blend.txt)
- Notes.
- A global batch of 2,048 sequences ร 8,192 tokens is about 16.8M tokens per step: 600,000 steps โ 10T tokens.
- Saving every 500 steps with asynchronous saves follows Part 7's checkpoint policy.
14. Designing for serving, not only training
- Serving cost decides the business (Phase 4), and architecture fixes most of it.
- Compute per generated token follows active parameters: 11.7B, not 65.2B.
- Memory per user follows the key-value (KV) cache, which attention design controls.
- The KV-cache arithmetic (our calculation).
- Per token: 2 (keys and values) ร 40 layers ร 8 KV heads ร 128 dimensions ร 2 bytes โ 160 KB.
- An 8,192-token conversation needs about 1.3 GB; with full multi-head attention (32 KV heads) it would need four times that.
- Grouped-query attention therefore lets each GPU hold about four times more simultaneous conversations.
- The catch: all 65.2B weights must be in memory at serving time.
- About 130 GB in BF16, so serving needs at least two H100s per replica, or 8-bit weights. Part 15 deals with it.
Grouped-query attention and serving memory (8,192-token context)
Our arithmetic from the architecture in this part.
15. The road not taken: dense 8B
| Dense 8B (Part 3, option B) | MoE 12B active (chosen) | |
|---|---|---|
| Training compute (Part 3) | 7.2e23 FLOPs, $5.34M with R&D | 7.2e23 FLOPs for the final run, $7.25M with R&D |
| Weights in memory | 8B | 65.2B |
| Compute per token at serving | 8B | 11.7B |
| Engineering risk | Low: well-trodden | Medium: routers, all-to-all, balance |
| Expected quality for the cost | Lower | Higher, as MoE results such as Mixtral and DeepSeek-V3 suggest |
- Why he accepts the risk.
- The quality per serving dollar matters more to clients than the simplicity of training.
- The ablation ladder retires most of the engineering risk before the main run.
16. The design review
- Before freezing, he walks through the design with two outside reviewers.
- One who has trained MoE models; one who serves models in production.
- The questions they asked.
- What happens to routing when the data mix shifts in the annealing phase?
- How does the layout change if 8 nodes are lost for a week?
- What is the serving cost per million tokens at the chosen size (Part 15)?
- Which ablation result would make you change the design, and did you run it?
- The answers went into the run plan.
- A fallback layout for 120 nodes (DP 480), tested for an hour on the block.
- A router-balance alarm in the run dashboards (Part 10).
17. Seven architecture mistakes
- Inventing a new architecture when the advantage is the data.
- Choosing a layout that fits memory but floods the network.
- Running tensor parallel across nodes.
- Ignoring router balance until experts collapse.
- Trusting a peak-FLOPs estimate instead of measured MFU.
- Skipping ablations to "save time", then losing weeks in the main run.
- Changing the tokenizer after the data refinery has tokenised everything.
18. Delegation box
- The architecture and the parallel layout
- Which ablations to run and what they decided
- Freezing the launch configuration
- Profiling summaries and kernel-trace triage
- Running the ablation grid and plotting results
- Checking config files against the plan
- Open-source training frameworks (Megatron-LM style)
- Published recipes from open model papers
- Two days of a contractor who has profiled MoE training
19. Budget and risk tracker
| Item | Status after Part 9 | Note |
|---|---|---|
| Spend to date | ~$3.4M (scenario) | Ablations on 64โ256 GPUs plus Maestro-0 serving |
| Model | Frozen: 65.2B total, 11.7B active | TP 2 ร EP 8 ร DP 512 |
| Time | Week 38 (month ~9) | Main run booked |
| Memory risk | Low | ~29 GB of 80 GB used (our estimate) |
| Throughput risk | Medium | MoE all-to-all; measured ~32% MFU (scenario) |
| Stability risk | Medium | Tested on the ablation ladder; watched in Part 10 |
20. The series
The Maestro: 18 parts in five phases
- Maestro-1 is now a concrete design: 65.2B parameters, 11.7B used per token, split across 1,024 GPUs with a layout that fits in memory with room to spare and keeps the heaviest communication on NVLink.
- Next, Part 10: 31 days of running it.
21. Reading the numbers
- Primary / standard
- Mixtral 8x7B ~46.7B total, ~12.9B active (Mistral AI). DeepSeek-V3 671B total, 37B active, 14.8T tokens, ~2.788M H800 GPU-hours, FP8 and auxiliary-loss-free balancing (DeepSeek). Llama 3 405B: 38โ43% MFU in BF16, 4D parallelism, grouped-query attention, 128k vocabulary (Meta). Mixed-precision Adam ~16 bytes/parameter and ZeRO sharding (Rajbhandari et al., 2020).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- The Maestro-1 architecture, activation-memory estimate, chosen layout, measured MFU and ablation outcomes.