Home โ€บ Blog โ€บ Frontier Model Case Studies
Frontier Architecture & Reasoning ยท Part 4 ยท Snapshot: September 2026 ๐Ÿ”ฌ

Frontier Model Case Studies: DeepSeek-V3 โ†’ R1, Llama 4 and Qwen3 / Qwen3-Next

Three open model families, three different bets. DeepSeek showed that a cheap-to-run sparse model plus pure reinforcement learning could produce frontier reasoning. Meta built Llama 4 around native multimodality and alternating sparse layers. Alibaba's Qwen put thinking and non-thinking into one model, then rebuilt attention itself for long context. This visual deep dive takes each apart โ€” with interactive diagrams of their real architectures and a calculator built from their published configurations.

Why these three: they are among the best-documented frontier-scale model families โ€” each has a technical report or model card plus downloadable weights and configuration files, so their design choices can be checked rather than guessed. Every number below comes from those sources (linked in References); interactive figures compute from the published configurations. Newer releases (DeepSeek-V4, Qwen3.5/3.8, Meta's Muse models) are covered in Part 1 and Part 3.

00 At a Glance: Big Models, Small Active Slices

All six models below are mixtures-of-experts: the full bar is every parameter stored, the bright part is what actually runs for each token. That gap โ€” from 3.8% to 15.6% โ€” is the single biggest idea these families share.

โ–ถ Total vs active parameters Tap a bar for details.
Tap a model.

01 Case Study 1: DeepSeek-V3 โ†’ DeepSeek-R1

Theme: architecture efficiency + reinforcement-learning reasoning + distillation.

DeepSeek-V3: the efficient base model (December 2024)

671Btotal parameters
37Bactive per token (5.5%)
14.8Ttraining tokens
61layers (first 3 dense)
256 + 1routed + shared experts per MoE layer
128Kcontext

V3 combined four efficiency ideas: DeepSeekMoE (many small experts โ€” each token uses 8 of 256 routed experts plus one always-on shared expert), multi-head latent attention (MLA) that caches one compressed vector per token instead of full keys and values, an auxiliary-loss-free load-balancing bias, and a multi-token prediction objective. It was trained largely in FP8, with the DualPipe schedule overlapping expert communication with computation.

โ–ถ Route a token through one DeepSeek-V3 MoE layer 256 routed experts in 8 groups (one per node); each token picks 8 experts from at most 4 groups.
Simulated scores, real structure: 256 routed experts in 8 groups, at most 4 groups per token (DeepSeek-V3 limits each token to 4 nodes to bound all-to-all traffic), top-8 experts among them, plus the shared expert every token uses.
The cost headline: DeepSeek reported that V3's full training took 2.788M H800 GPU-hours โ€” about $5.6M at an assumed $2 per GPU-hour. The report notes this covers the official training run only, not prior research, ablations or the cost of the cluster itself, so it is not the total cost of building the model.

From V3 to R1: reasoning through reinforcement learning (January 2025)

DeepSeek then asked whether reasoning could be learned from reward alone. R1-Zero applied reinforcement learning (GRPO, with rule-based rewards for answer accuracy and format) directly to the V3 base model, with no supervised fine-tuning first. It learned to reason at length, check its work and reflect โ€” but its outputs mixed languages and were hard to read. R1 fixed this with a four-stage pipeline. Step through it:

โ–ถ The R1 training pipeline Press play or click a stage.
โ–ถ AIME 2024 pass@1, as reported Maths competition accuracy from the DeepSeek-R1 paper.
R1-Zero rose from 15.6% to 71.0% during RL alone (86.7% with majority voting over 64 samples). Distilled models were trained with supervised fine-tuning only, on about 800K samples generated by R1 โ€” no RL โ€” yet the 32B distilled model scores 72.6%.

02 Case Study 2: Meta Llama 4 Scout / Maverick (April 2025)

Theme: sparse activation + native multimodality + deployment efficiency.

Llama 4 was Meta's first mixture-of-experts family and its first natively multimodal one. Both released models activate 17B parameters per token โ€” each token goes to one routed expert plus a shared expert โ€” but they spend their parameter budget differently. Toggle between them:

โ–ถ 48 layers, two designs Each block is one transformer layer.
MoE layerdense layerno positional encoding (every 4th layer)
Llama 4 ScoutLlama 4 Maverick
Total / active109B / 17B400B / 17B
Experts16 routed + shared128 routed + shared
Layer layoutMoE in every layerAlternating dense and MoE layers
Context (reported)10M tokens1M tokens
Pre-training data~40T tokens~22T tokens
DeploymentFits one H100 GPU with INT4 quantizationOne H100 host (8 GPUs)

Early fusion is the multimodal idea: image patches become tokens in the same sequence as text from the start of pre-training, instead of being attached to a finished language model. The backbone learns text and vision jointly.

โ–ถ Early fusion Text and image tokens enter one shared backbone.
Also documented: Llama 4's long context relies on "iRoPE" โ€” interleaved attention layers without positional embeddings, plus temperature scaling of attention at inference. Both models were distilled from Llama 4 Behemoth (288B active, about 2T total), which Meta previewed but did not release. Meta also acknowledged that the version ranked on the LMArena leaderboard at launch was an experimental chat-optimized variant, not the released model โ€” a reminder to read benchmark claims carefully.

03 Case Study 3: Qwen3 / Qwen3-Next (2025)

Theme: controllable reasoning + sparse experts + hybrid attention for long context.

Qwen3: one model, two modes (April 2025)

Qwen3 shipped dense models from 0.6B to 32B and two MoE models โ€” Qwen3-235B-A22B (94 layers, 128 experts, 8 active) and Qwen3-30B-A3B (48 layers, same expert layout) โ€” trained on about 36T tokens in 119 languages, all under Apache 2.0. Its signature feature was hybrid thinking: one model that can reason step by step or answer directly, switched with /think and /no_think tags or an API flag, with a thinking budget to cap reasoning tokens.

โ–ถ Hybrid thinking Same question, same weights, two modes. Token counts are illustrative.
What happened next: in July 2025 Qwen moved away from the single hybrid model, releasing separate Instruct-2507 and Thinking-2507 versions, citing better quality from training each mode separately. Controllable reasoning survived โ€” but as a choice of model or setting, not always one set of weights.

Qwen3-Next-80B-A3B: rebuilding attention (September 2025)

Qwen3-Next asked a different question: can attention itself get cheaper for long inputs? It replaces three of every four attention layers with Gated DeltaNet, a linear-attention layer that keeps a fixed-size memory state instead of a growing KV cache, and uses gated attention in the remaining layers. It pairs this with extreme sparsity: 512 experts, 10 routed + 1 shared active โ€” 3B of 80B parameters per token.

โ–ถ The 3 : 1 hybrid layout and what it does to memory 48 layers = 12 ร— (3 Gated DeltaNet + 1 gated attention). Slide the context length.
Attention-state memory for one sequence in BF16, from the published configurations: KV bytes per token ร— context, plus Qwen3-Next's fixed DeltaNet state (36 layers ร— 32 heads ร— 128 ร— 128 values โ‰ˆ 38 MB). Qwen3-235B-A22B: 94 layers ร— 4 KV heads ร— 128 dims; DeepSeek-V3: 61 layers ร— (512 + 64) MLA latent; Qwen3-Next: 12 attention layers ร— 2 KV heads ร— 256 dims.
80B / 3Btotal / active (3.8%)
512experts (10 routed + 1 shared)
262,144native context (~1M with YaRN)
15Ttraining tokens
<10%of Qwen3-32B's training cost, per Qwen
>10ร—throughput over Qwen3-32B beyond 32K context, per Qwen

04 Cross-Case Comparison (Key Public Details)

ModelDesignTotalActiveModalitiesReasoning methodContextLicence / docs
DeepSeek-V3MoE (256+1, top-8) + MLA671B37BText onlyGeneral base + chat model128KOpen weights ยท technical report
DeepSeek-R1Same as V3671B37BText onlyLarge-scale RL (GRPO) + cold start128KMIT ยท Nature paper
Llama 4 ScoutMoE every layer (16 experts)109B17BText + image (early fusion)Instruction-tuned10M (reported)Llama 4 licence ยท model card
Llama 4 MaverickAlternating dense / MoE (128 experts)400B17BText + image (early fusion)Instruction-tuned1M (reported)Llama 4 licence ยท model card
Qwen3-235B-A22BMoE (128, top-8)235B22BText onlyHybrid thinking32K native, 128K with YaRNApache 2.0 ยท technical report
Qwen3-30B-A3BMoE (128, top-8)30.5B3.3BText onlyHybrid thinking32K native, 128K with YaRNApache 2.0 ยท technical report
Qwen3-Next-80B-A3BHybrid attention + MoE (512, 10+1)80B3BText onlySeparate Instruct and Thinking models262K native, ~1M with YaRNApache 2.0 ยท model card, blog

Multimodal versions of DeepSeek and Qwen models exist as separate releases (for example Qwen3-VL); the language models in this table are text-only.

05 What the Published Configs Imply

Two numbers decide much of a model's serving cost: compute per token (about 2 ร— active parameters in FLOPs) and KV-cache memory per token (set by the attention design). This script computes both from the published configurations:

"""Five open model families, one calculator: what their published configs imply for inference.

Parameter counts and attention shapes come from each model's technical report and config.json.
Per-token compute ~ 2 x active parameters (forward pass FLOPs). KV cache per token depends on the
attention design: GQA stores keys and values per KV head; DeepSeek's MLA stores one compressed latent;
Qwen3-Next keeps a KV cache only in its 12 full-attention layers (Gated DeltaNet layers hold a
fixed-size state instead). Standard library only.
"""
BF16 = 2  # bytes per value

MODELS = {
    # name: (total params B, active params B, KV-cache bytes per token, extra fixed state bytes per sequence)
    "DeepSeek-V3 / R1":    (671, 37, 61 * (512 + 64) * BF16, 0),                 # MLA: 61 layers x (512 latent + 64 RoPE key)
    "Llama 4 Scout":       (109, 17, 2 * 48 * 8 * 128 * BF16, 0),                # GQA: 48 layers x 8 KV heads x 128 dims, K and V
    "Llama 4 Maverick":    (400, 17, 2 * 48 * 8 * 128 * BF16, 0),
    "Qwen3-235B-A22B":     (235, 22, 2 * 94 * 4 * 128 * BF16, 0),                # GQA: 94 layers x 4 KV heads x 128 dims
    "Qwen3-30B-A3B":       (30.5, 3.3, 2 * 48 * 4 * 128 * BF16, 0),
    "Qwen3-Next-80B-A3B":  (80, 3, 2 * 12 * 2 * 256 * BF16, 36 * 32 * 128 * 128 * BF16),
    # Qwen3-Next: 12 gated-attention layers x 2 KV heads x 256 dims; 36 Gated DeltaNet layers x 32 heads x 128x128 state
}

print(f"{'model':<20}{'total':>7}{'active':>8}{'active %':>10}{'GFLOPs/token':>14}{'weights FP8':>13}{'KV/token':>11}")
for name, (tot, act, kv, _) in MODELS.items():
    print(f"{name:<20}{tot:>6g}B{act:>7g}B{100 * act / tot:>9.1f}%{2 * act:>14.0f}{tot:>11.0f} GB{kv / 1024:>8.0f} KB")

print("\nMemory for one sequence's attention state (BF16), by context length:")
ctxs = [32_768, 131_072, 1_048_576]
print(f"{'model':<20}" + "".join(f"{c:>14,}" for c in ctxs))
for name, (_, _, kv, fixed) in MODELS.items():
    print(f"{name:<20}" + "".join(f"{(kv * c + fixed) / 1e9:>11.2f} GB" for c in ctxs))

base = MODELS["Qwen3-235B-A22B"][2]
print(f"\nPer token, DeepSeek's MLA cache is {MODELS['DeepSeek-V3 / R1'][2] / base:.0%} of Qwen3-235B's GQA cache, "
      f"and Qwen3-Next's is {MODELS['Qwen3-Next-80B-A3B'][2] / base:.0%}.")
print("Weights at FP8 = 1 byte per parameter; BF16 doubles it. Context lengths beyond each model's supported maximum are shown only for comparison.")

Output:

model                 total  active  active %  GFLOPs/token  weights FP8   KV/token
DeepSeek-V3 / R1       671B     37B      5.5%            74        671 GB      69 KB
Llama 4 Scout          109B     17B     15.6%            34        109 GB     192 KB
Llama 4 Maverick       400B     17B      4.2%            34        400 GB     192 KB
Qwen3-235B-A22B        235B     22B      9.4%            44        235 GB     188 KB
Qwen3-30B-A3B         30.5B    3.3B     10.8%             7         30 GB      96 KB
Qwen3-Next-80B-A3B      80B      3B      3.8%             6         80 GB      24 KB

Memory for one sequence's attention state (BF16), by context length:
model                       32,768       131,072     1,048,576
DeepSeek-V3 / R1           2.30 GB       9.21 GB      73.69 GB
Llama 4 Scout              6.44 GB      25.77 GB     206.16 GB
Llama 4 Maverick           6.44 GB      25.77 GB     206.16 GB
Qwen3-235B-A22B            6.31 GB      25.23 GB     201.86 GB
Qwen3-30B-A3B              3.22 GB      12.88 GB     103.08 GB
Qwen3-Next-80B-A3B         0.84 GB       3.26 GB      25.81 GB

Per token, DeepSeek's MLA cache is 37% of Qwen3-235B's GQA cache, and Qwen3-Next's is 13%.
Weights at FP8 = 1 byte per parameter; BF16 doubles it. Context lengths beyond each model's supported maximum are shown only for comparison.

Three lessons fall out. Compute per token tracks active parameters, so Maverick costs no more per token than Scout despite being nearly four times larger. Memory for weights tracks total parameters, so Maverick needs far more GPUs to hold. And attention design dominates long-context memory: MLA and hybrid linear attention cut per-sequence memory by large factors, which is why the next generation (DeepSeek-V4's compressed sparse attention, covered in Part 1) keeps pushing here. One caveat for Llama 4: three of every four layers use chunked attention over 8,192-token windows, so implementations that exploit this store less than the full-attention figure shown.

06 What Each Case Teaches

DeepSeek-V3 โ†’ R1
  • A MoE model can reach high performance with low compute per token.
  • Large-scale RL with verifiable rewards can teach reasoning โ€” but a small cold-start dataset makes it readable and stable.
  • Distillation transfers reasoning to small open models cheaply.
Llama 4
  • Sparse activation lets one family span very different sizes at the same per-token cost.
  • Early fusion builds multimodality into pre-training.
  • Layout choices (every-layer vs alternating MoE) trade memory for capacity.
  • Benchmark claims need checking against the released model.
Qwen3 / Qwen3-Next
  • Thinking vs non-thinking gives users control over depth, latency and cost.
  • Hybrid linear + full attention can cut long-context cost dramatically.
  • Very many small experts with few active keeps capacity high and compute low.

07 Architecture โ†’ Training โ†’ Inference Consequences

โ–ถ Pick an architecture choice See what it demands in training and what it changes in serving.

08 From Research Idea to Product Impact

1Research ideaMoE, RL reasoning, linear attention, early fusion
2Model architecturesparse experts, hybrid layers
3Training & post-trainingtrillions of tokens, RL, distillation
4Inference behaviourcheap tokens, controllable reasoning
5Deployment economicscost per token, hardware fit
6Product capabilityreasoning, multimodal, long documents

09 Common Design Patterns Across Cases

โœ“Mixture-of-experts to scale total size while keeping per-token compute low (all three families).
โœ“Shared experts alongside routed ones for common knowledge (DeepSeek, Llama 4, Qwen3-Next).
โœ“Hardware-aware design: FP8 training, node-limited routing, single-GPU deployment targets.
โœ“Very large, diverse training data: 14.8T to ~40T tokens.
โœ“Reinforcement learning and test-time reasoning, with controls over when to use it.
โœ“Long-context support through attention changes (MLA, iRoPE and chunking, hybrid linear attention).
โœ“Distillation and multiple sizes for different cost and capability needs.

10 What Remains Proprietary or Unknown

Even "open" models leave a lot unpublished โ€” and closed frontier models disclose far less still:

1Exact training-data composition and filtering
2Complete training recipes and hyperparameter searches
3Full compute budgets, including failed runs and research
4RL environments, reward models and evaluation pipelines in detail
5Internal infrastructure and optimization tricks
6Internal safety evaluations
7Future model plans
Key takeaways
The three families made different trade-offs but converged on the same toolkit. DeepSeek proved that an efficient sparse model plus reinforcement learning on verifiable rewards could reach frontier reasoning, and that distillation could hand it to small models. Llama 4 showed how one sparse recipe spans very different sizes at the same per-token cost, with multimodality fused in from pre-training. Qwen made reasoning depth a user-facing control and then attacked the cost of long context at the attention layer itself. Sparse activation, RL-based reasoning and cheaper long context are now standard frontier strategy โ€” and technical reports and model cards, not marketing, are where you can check which trade-offs a model really made.

โš ๏ธ Notes

Corrections to the source infographic: (1) modalities โ€” it marks DeepSeek-V3, DeepSeek-R1, Qwen3-235B-A22B, Qwen3-30B-A3B and Qwen3-Next-80B-A3B as multimodal; all five are text-only language models (multimodal variants such as Qwen3-VL are separate releases), and only Llama 4 is natively multimodal; (2) Llama 4 Scout is described as alternating dense and MoE layers โ€” its published configuration uses MoE in every layer; alternating layers are Maverick's design; (3) Qwen3-Next is listed with hybrid thinking โ€” it was released as separate Instruct and Thinking models (Qwen had already split Qwen3 into Instruct-2507 and Thinking-2507 in July 2025); (4) context โ€” Qwen3's MoE models have 32K native context, extendable to about 128K with YaRN, rather than an unspecified long context; Llama 4's reported windows differ (Scout 10M, Maverick 1M). The DeepSeek-V3 cost figure is the reported cost of the final training run only. Thinking-mode token counts are illustrative; all other interactive figures use published values.

๐Ÿ”— References