Home โ€บ Blog โ€บ The Maestro ยท Part 9
The Maestro ยท Part 9 ยท Phase 2 ยท ๐Ÿง  ML engineer hat

Architecture and Parallelism

Part 3 decided on a mixture-of-experts model with about 12B active parameters, trained on 10T tokens. Now Gopala has to design it for real: how many layers and experts, what attention, what tokenizer, and how to cut the model into pieces so that 1,024 GPUs can train it together without running out of memory or waiting on the network. This part counts the parameters with a script, plans memory per GPU with another, explains the four kinds of parallelism in plain terms, and ends with the small ablation runs that de-risk the design before the main run.

FL
FrontierAGI Team
๐Ÿง  ML engineer๐Ÿ—๏ธ Systems architect๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) Published figures for Mixtral, DeepSeek-V3 and Llama 3 are from their papers. moe_params.py and memory_plan.py were run (real output below); the architecture is our design for the story, and the activation-memory estimate is a rough rule of thumb, not a measurement. launch_megatron.sh was not executed for this article. Ablation results in the story are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 24โ€“38

Weeks 24โ€“28 ยท Design
  • Writes moe_params.py; fixes the shape: 40 layers, 64 experts, 8 active per token, one shared expert.
  • Chooses grouped-query attention and a 128k-token tokenizer.
Weeks 28โ€“34 ยท Layout and ablations
  • Writes memory_plan.py; picks TP 2 ร— EP 8 ร— DP 512.
  • Runs ablations at 1โ€“3B active parameters on 64โ€“256 GPUs to test router settings, the data mix (Part 8) and stability.
Weeks 34โ€“38 ยท Freeze
  • Freezes the architecture and the launch configuration; measures throughput on the full block; books the main run for month 9.

2. Why mixture-of-experts

Dense vs mixture-of-experts at the same compute per token

11.7Bactive parameters per token (both)
65.2Btotal parameters in Maestro-1 (MoE)
18%of the weights each token uses
~5.6ร—more weights than a dense model of equal compute

Real output of moe_params.py for our design.

3. Counting the parameters

moe_params.py ยท Python 3, no dependencies
"""moe_params.py โ€” count total and active parameters of Maestro-1's MoE.

A mixture-of-experts layer holds many feed-forward "experts" but each
token uses only a few, so compute follows ACTIVE parameters and memory
follows TOTAL parameters. Configuration is our design for this story.
"""
CFG = dict(layers=40, d=4096, heads=32, kv_heads=8, vocab=128_000,
           experts=64, top_k=8, shared=1, d_ff_expert=2048, dense_first=2, d_ff_dense=14336)

def count(c):
    d, hd = c["d"], c["d"] // c["heads"]
    attn = d * d * 2 + d * hd * c["kv_heads"] * 2           # Q,O full; K,V grouped (GQA)
    expert = 3 * d * c["d_ff_expert"]                        # SwiGLU: gate, up, down
    dense_ffn = 3 * d * c["d_ff_dense"]
    router = d * c["experts"]
    moe_layers = c["layers"] - c["dense_first"]
    emb = c["vocab"] * d * 2                                 # input + output (untied)
    total = (c["layers"] * attn + c["dense_first"] * dense_ffn
             + moe_layers * ((c["experts"] + c["shared"]) * expert + router) + emb)
    active = (c["layers"] * attn + c["dense_first"] * dense_ffn
              + moe_layers * ((c["top_k"] + c["shared"]) * expert + router) + emb)
    return attn, expert, total, active, emb

def main():
    attn, expert, total, active, emb = count(CFG)
    print(f"per layer: attention {attn/1e6:.1f}M, one expert {expert/1e6:.1f}M")
    print(f"total parameters  {total/1e9:6.1f}B   (memory, checkpoints)")
    print(f"active per token  {active/1e9:6.1f}B   (compute: 6 x active x tokens)")
    print(f"  of which embeddings {emb/1e9:.1f}B; active without embeddings {(active-emb)/1e9:.1f}B")
    print(f"sparsity: each token uses {active/total:.0%} of the weights")
    for k in (2, 4, 6, 8):
        c = dict(CFG, top_k=k)
        print(f"  top_k={k}: active {count(c)[3]/1e9:5.1f}B")

if __name__ == "__main__":
    main()
$ python3 moe_params.py per layer: attention 41.9M, one expert 25.2M total parameters 65.2B (memory, checkpoints) active per token 11.7B (compute: 6 x active x tokens) of which embeddings 1.0B; active without embeddings 10.6B sparsity: each token uses 18% of the weights top_k=2: active 6.0B top_k=4: active 7.9B top_k=6: active 9.8B top_k=8: active 11.7B

4. Design choices, and why

ChoiceDecisionReason
AttentionGrouped-query attention, 32 query heads, 8 key/value groupsSmaller KV cache at serving time (Part 15) with little quality loss; used by Llama 3
Context length8,192 in pre-training, extended laterLong-context training is expensive; extend in a short later phase, as Llama 3 did
Tokenizer128k vocabulary, trained on the blend, code-awareFewer tokens per line of code means cheaper training and serving
First layers2 dense layers before the MoE layersA common choice for stability in early layers
Router balanceAuxiliary load-balancing loss (small coefficient)Stops experts collapsing; DeepSeek-V3 shows auxiliary-loss-free alternatives exist
PrecisionBF16 with FP32 master weightsProven at scale; FP8 training (as in DeepSeek-V3) is a later option
Positional encodingRoPEStandard, extends to longer contexts

5. Four ways to split the work

The four kinds of parallelism

๐Ÿ“‹ Data parallel (DP)Every group of GPUs holds a copy of the model and sees different data; gradients are averaged
โœ‚๏ธ Tensor parallel (TP)Each layer's matrices split across GPUs; needs very fast links, so stays inside one node
๐Ÿชœ Pipeline parallel (PP)Layers split into stages on different GPUs; micro-batches flow through like an assembly line
๐Ÿง‘โ€๐Ÿคโ€๐Ÿง‘ Expert parallel (EP)Experts spread across GPUs; tokens travel to their experts (all-to-all)

Real systems combine them. Llama 3 405B (dense) used tensor, pipeline, context and data parallelism together.

6. Will it fit? A memory plan

memory_plan.py ยท Python 3, no dependencies
"""memory_plan.py โ€” which parallel layout fits Maestro-1 on 1,024 H100s?

Memory per GPU = weights + gradients + optimizer state (sharded) +
activations. Mixed-precision Adam keeps ~16 bytes per parameter in total
(bf16 weights and grads, fp32 master weights, two fp32 moments);
ZeRO-1-style sharding spreads the 12 optimizer bytes across data-parallel
ranks. Activation memory is a rough per-token estimate (our assumption).
"""
TOTAL, NON_EXPERT = 65.2e9, 4.1e9   # from moe_params.py: all params; params not split by EP (attention, dense, shared expert, embeddings)
GPUS, HBM = 1024, 80e9
SEQ, MICRO = 8192, 1
ACT_BYTES_PER_TOKEN_LAYER = 34 * 4096 * 2 / 8    # rough, with selective recompute (assumption)
LAYERS = 40

def per_gpu(tp, pp, ep):
    dp = GPUS // (tp * pp)
    expert_params = TOTAL - NON_EXPERT
    params = NON_EXPERT / (tp * pp) + expert_params / (tp * pp * ep)
    w_g = params * 4                                   # bf16 weights + bf16 grads
    opt = params * 12 / dp                             # fp32 master + 2 moments, sharded over DP
    act = SEQ * MICRO * ACT_BYTES_PER_TOKEN_LAYER * (LAYERS / pp) / tp * pp  # 1F1B keeps ~pp micro-batches
    return dp, (w_g + opt + act) / 1e9

def main():
    print(f"{'TP':>3s} {'PP':>3s} {'EP':>3s} {'DP':>4s} {'GB/GPU':>7s}  fits 80 GB with 10% headroom?")
    for tp, pp, ep in [(1, 1, 1), (1, 1, 8), (2, 1, 8), (1, 2, 8), (4, 1, 8), (1, 4, 16), (2, 2, 8), (8, 1, 8)]:
        dp, gb = per_gpu(tp, pp, ep)
        if dp % ep:
            continue
        print(f"{tp:3d} {pp:3d} {ep:3d} {dp:4d} {gb:7.1f}  {'yes' if gb < 0.9 * HBM / 1e9 else 'no'}")
    print("rule: keep TP inside one 8-GPU node (NVLink); EP across nodes costs all-to-all traffic")

if __name__ == "__main__":
    main()
$ python3 memory_plan.py TP PP EP DP GB/GPU fits 80 GB with 10% headroom? 1 1 1 1024 273.0 no 1 1 8 1024 58.5 yes 2 1 8 512 29.3 yes 1 2 8 512 35.0 yes 4 1 8 256 14.7 yes 1 4 16 256 19.4 yes 2 2 8 256 17.6 yes 8 1 8 128 7.4 yes rule: keep TP inside one 8-GPU node (NVLink); EP across nodes costs all-to-all traffic

Memory per GPU by layout (80 GB available)

No split (TP1 PP1 EP1)273 GB โœ—
EP 8 only58.5 GB
TP 2 ร— EP 8 (chosen)29.3 GB
PP 2 ร— EP 835.0 GB
TP 8 ร— EP 87.4 GB

Real output of memory_plan.py (activation memory is a rough estimate).

7. Try a layout

Parallel layout planner for 1,024 GPUs (our model)

8. How fast will it run?

Scenario
  • In the story the measured throughput is about 32% MFU, slightly better than the plan's 30%, after two changes found in profiling: overlapping the expert all-to-all with computation, and raising the micro-batch to use the memory headroom.
  • These are story values, not measurements.

9. Ablations: small runs that de-risk the big one

The ablation ladder

๐Ÿ”ฌ~100M activeRouter and loss settings; hours on 8 GPUs
๐Ÿงช~1B activeData mixture choices from Part 8; 64 GPUs
โš—๏ธ~3B activeStability at higher LR, batch size; 256 GPUs
๐Ÿš€12B activeThroughput and the first 1% of the main run

Each rung tests what the next rung would be too expensive to get wrong.

10. The communication budget

Where each kind of traffic travels

โšกNVLink, inside a nodeTensor parallel, every layer
๐Ÿ•ธ๏ธInfiniBand, across nodesExpert all-to-all and gradient reduce
๐Ÿ’พStorage networkCheckpoints and data loading

Fast links carry frequent traffic; slow links carry traffic that can overlap with computation.

11. The tokenizer, the decision that cannot be undone

12. What could still go wrong

RiskEarly signResponse
Router collapseA few experts take most tokens; others idleRaise the balance coefficient; restart from the last balanced checkpoint
Loss spikesSudden jumps in loss (Part 10)Rewind and skip batches; lower learning rate if repeated
Throughput dropStep time rises on one rankStraggler detection (Part 7) and node swap
Memory fragmentationOut-of-memory after hoursHeadroom kept on purpose; allocator settings tested in ablations

13. The launch configuration

launch_megatron.sh ยท Megatron-LM launch sketch ยท not executed for this article
#!/bin/bash
# launch_megatron.sh โ€” Maestro-1 layout on 128 nodes (sketch; flag names follow
# Megatron-LM's documented arguments, verify against your version).
#SBATCH --partition=main --nodes=128 --gpus-per-node=8 --exclusive
export NCCL_IB_HCA=mlx5 CUDA_DEVICE_MAX_CONNECTIONS=1
srun python pretrain_gpt.py \
  --num-layers 40 --hidden-size 4096 --num-attention-heads 32 \
  --group-query-attention --num-query-groups 8 --swiglu \
  --num-experts 64 --moe-router-topk 8 --moe-shared-expert-intermediate-size 2048 \
  --moe-ffn-hidden-size 2048 --moe-router-load-balancing-type aux_loss --moe-aux-loss-coeff 1e-3 \
  --tensor-model-parallel-size 2 --pipeline-model-parallel-size 1 \
  --expert-model-parallel-size 8 --sequence-parallel --use-distributed-optimizer \
  --seq-length 8192 --global-batch-size 2048 --micro-batch-size 1 \
  --bf16 --recompute-granularity selective \
  --lr 3e-4 --min-lr 3e-5 --lr-warmup-iters 2000 --train-iters 600000 \
  --save /checkpoints/maestro-1 --save-interval 500 --async-save \
  --data-path $(cat /data/mixture/maestro1_blend.txt)

14. Designing for serving, not only training

Grouped-query attention and serving memory (8,192-token context)

Full multi-head attention, 32 KV heads~5.4 GB per conversation
Grouped-query, 8 KV heads (chosen)~1.3 GB per conversation

Our arithmetic from the architecture in this part.

15. The road not taken: dense 8B

Dense 8B (Part 3, option B)MoE 12B active (chosen)
Training compute (Part 3)7.2e23 FLOPs, $5.34M with R&D7.2e23 FLOPs for the final run, $7.25M with R&D
Weights in memory8B65.2B
Compute per token at serving8B11.7B
Engineering riskLow: well-troddenMedium: routers, all-to-all, balance
Expected quality for the costLowerHigher, as MoE results such as Mixtral and DeepSeek-V3 suggest

16. The design review

17. Seven architecture mistakes

18. Delegation box

๐Ÿ™‹ Gopala himself
  • The architecture and the parallel layout
  • Which ablations to run and what they decided
  • Freezing the launch configuration
๐Ÿค– AI agents
  • Profiling summaries and kernel-trace triage
  • Running the ablation grid and plotting results
  • Checking config files against the plan
๐Ÿค Bought, rented or borrowed
  • Open-source training frameworks (Megatron-LM style)
  • Published recipes from open model papers
  • Two days of a contractor who has profiled MoE training

19. Budget and risk tracker

ItemStatus after Part 9Note
Spend to date~$3.4M (scenario)Ablations on 64โ€“256 GPUs plus Maestro-0 serving
ModelFrozen: 65.2B total, 11.7B activeTP 2 ร— EP 8 ร— DP 512
TimeWeek 38 (month ~9)Main run booked
Memory riskLow~29 GB of 80 GB used (our estimate)
Throughput riskMediumMoE all-to-all; measured ~32% MFU (scenario)
Stability riskMediumTested on the ablation ladder; watched in Part 10

20. The series

The Maestro: 18 parts in five phases

Verdict
  • Maestro-1 is now a concrete design: 65.2B parameters, 11.7B used per token, split across 1,024 GPUs with a layout that fits in memory with room to spare and keeps the heaviest communication on NVLink.
    • Next, Part 10: 31 days of running it.

21. Reading the numbers

Primary / standard
Mixtral 8x7B ~46.7B total, ~12.9B active (Mistral AI). DeepSeek-V3 671B total, 37B active, 14.8T tokens, ~2.788M H800 GPU-hours, FP8 and auxiliary-loss-free balancing (DeepSeek). Llama 3 405B: 38โ€“43% MFU in BF16, 4D parallelism, grouped-query attention, 128k vocabulary (Meta). Mixed-precision Adam ~16 bytes/parameter and ZeRO sharding (Rajbhandari et al., 2020).
Secondary โš ๏ธ
None in this part.
Our assumptions
The Maestro-1 architecture, activation-memory estimate, chosen layout, measured MFU and ablation outcomes.