Frontier Model Case Studies: DeepSeek-V3 โ R1, Llama 4 and Qwen3 / Qwen3-Next
Three open model families, three different bets. DeepSeek showed that a cheap-to-run sparse model plus pure reinforcement learning could produce frontier reasoning. Meta built Llama 4 around native multimodality and alternating sparse layers. Alibaba's Qwen put thinking and non-thinking into one model, then rebuilt attention itself for long context. This visual deep dive takes each apart โ with interactive diagrams of their real architectures and a calculator built from their published configurations.
00 At a Glance: Big Models, Small Active Slices
All six models below are mixtures-of-experts: the full bar is every parameter stored, the bright part is what actually runs for each token. That gap โ from 3.8% to 15.6% โ is the single biggest idea these families share.
01 Case Study 1: DeepSeek-V3 โ DeepSeek-R1
Theme: architecture efficiency + reinforcement-learning reasoning + distillation.
DeepSeek-V3: the efficient base model (December 2024)
V3 combined four efficiency ideas: DeepSeekMoE (many small experts โ each token uses 8 of 256 routed experts plus one always-on shared expert), multi-head latent attention (MLA) that caches one compressed vector per token instead of full keys and values, an auxiliary-loss-free load-balancing bias, and a multi-token prediction objective. It was trained largely in FP8, with the DualPipe schedule overlapping expert communication with computation.
From V3 to R1: reasoning through reinforcement learning (January 2025)
DeepSeek then asked whether reasoning could be learned from reward alone. R1-Zero applied reinforcement learning (GRPO, with rule-based rewards for answer accuracy and format) directly to the V3 base model, with no supervised fine-tuning first. It learned to reason at length, check its work and reflect โ but its outputs mixed languages and were hard to read. R1 fixed this with a four-stage pipeline. Step through it:
02 Case Study 2: Meta Llama 4 Scout / Maverick (April 2025)
Theme: sparse activation + native multimodality + deployment efficiency.
Llama 4 was Meta's first mixture-of-experts family and its first natively multimodal one. Both released models activate 17B parameters per token โ each token goes to one routed expert plus a shared expert โ but they spend their parameter budget differently. Toggle between them:
| Llama 4 Scout | Llama 4 Maverick | |
|---|---|---|
| Total / active | 109B / 17B | 400B / 17B |
| Experts | 16 routed + shared | 128 routed + shared |
| Layer layout | MoE in every layer | Alternating dense and MoE layers |
| Context (reported) | 10M tokens | 1M tokens |
| Pre-training data | ~40T tokens | ~22T tokens |
| Deployment | Fits one H100 GPU with INT4 quantization | One H100 host (8 GPUs) |
Early fusion is the multimodal idea: image patches become tokens in the same sequence as text from the start of pre-training, instead of being attached to a finished language model. The backbone learns text and vision jointly.
03 Case Study 3: Qwen3 / Qwen3-Next (2025)
Theme: controllable reasoning + sparse experts + hybrid attention for long context.
Qwen3: one model, two modes (April 2025)
Qwen3 shipped dense models from 0.6B to 32B and two MoE models โ Qwen3-235B-A22B (94 layers, 128 experts, 8 active) and Qwen3-30B-A3B (48 layers, same expert layout) โ trained on about 36T tokens in 119 languages, all under Apache 2.0. Its signature feature was hybrid thinking: one model that can reason step by step or answer directly, switched with /think and /no_think tags or an API flag, with a thinking budget to cap reasoning tokens.
Qwen3-Next-80B-A3B: rebuilding attention (September 2025)
Qwen3-Next asked a different question: can attention itself get cheaper for long inputs? It replaces three of every four attention layers with Gated DeltaNet, a linear-attention layer that keeps a fixed-size memory state instead of a growing KV cache, and uses gated attention in the remaining layers. It pairs this with extreme sparsity: 512 experts, 10 routed + 1 shared active โ 3B of 80B parameters per token.
04 Cross-Case Comparison (Key Public Details)
| Model | Design | Total | Active | Modalities | Reasoning method | Context | Licence / docs |
|---|---|---|---|---|---|---|---|
| DeepSeek-V3 | MoE (256+1, top-8) + MLA | 671B | 37B | Text only | General base + chat model | 128K | Open weights ยท technical report |
| DeepSeek-R1 | Same as V3 | 671B | 37B | Text only | Large-scale RL (GRPO) + cold start | 128K | MIT ยท Nature paper |
| Llama 4 Scout | MoE every layer (16 experts) | 109B | 17B | Text + image (early fusion) | Instruction-tuned | 10M (reported) | Llama 4 licence ยท model card |
| Llama 4 Maverick | Alternating dense / MoE (128 experts) | 400B | 17B | Text + image (early fusion) | Instruction-tuned | 1M (reported) | Llama 4 licence ยท model card |
| Qwen3-235B-A22B | MoE (128, top-8) | 235B | 22B | Text only | Hybrid thinking | 32K native, 128K with YaRN | Apache 2.0 ยท technical report |
| Qwen3-30B-A3B | MoE (128, top-8) | 30.5B | 3.3B | Text only | Hybrid thinking | 32K native, 128K with YaRN | Apache 2.0 ยท technical report |
| Qwen3-Next-80B-A3B | Hybrid attention + MoE (512, 10+1) | 80B | 3B | Text only | Separate Instruct and Thinking models | 262K native, ~1M with YaRN | Apache 2.0 ยท model card, blog |
Multimodal versions of DeepSeek and Qwen models exist as separate releases (for example Qwen3-VL); the language models in this table are text-only.
05 What the Published Configs Imply
Two numbers decide much of a model's serving cost: compute per token (about 2 ร active parameters in FLOPs) and KV-cache memory per token (set by the attention design). This script computes both from the published configurations:
"""Five open model families, one calculator: what their published configs imply for inference. Parameter counts and attention shapes come from each model's technical report and config.json. Per-token compute ~ 2 x active parameters (forward pass FLOPs). KV cache per token depends on the attention design: GQA stores keys and values per KV head; DeepSeek's MLA stores one compressed latent; Qwen3-Next keeps a KV cache only in its 12 full-attention layers (Gated DeltaNet layers hold a fixed-size state instead). Standard library only. """ BF16 = 2 # bytes per value MODELS = { # name: (total params B, active params B, KV-cache bytes per token, extra fixed state bytes per sequence) "DeepSeek-V3 / R1": (671, 37, 61 * (512 + 64) * BF16, 0), # MLA: 61 layers x (512 latent + 64 RoPE key) "Llama 4 Scout": (109, 17, 2 * 48 * 8 * 128 * BF16, 0), # GQA: 48 layers x 8 KV heads x 128 dims, K and V "Llama 4 Maverick": (400, 17, 2 * 48 * 8 * 128 * BF16, 0), "Qwen3-235B-A22B": (235, 22, 2 * 94 * 4 * 128 * BF16, 0), # GQA: 94 layers x 4 KV heads x 128 dims "Qwen3-30B-A3B": (30.5, 3.3, 2 * 48 * 4 * 128 * BF16, 0), "Qwen3-Next-80B-A3B": (80, 3, 2 * 12 * 2 * 256 * BF16, 36 * 32 * 128 * 128 * BF16), # Qwen3-Next: 12 gated-attention layers x 2 KV heads x 256 dims; 36 Gated DeltaNet layers x 32 heads x 128x128 state } print(f"{'model':<20}{'total':>7}{'active':>8}{'active %':>10}{'GFLOPs/token':>14}{'weights FP8':>13}{'KV/token':>11}") for name, (tot, act, kv, _) in MODELS.items(): print(f"{name:<20}{tot:>6g}B{act:>7g}B{100 * act / tot:>9.1f}%{2 * act:>14.0f}{tot:>11.0f} GB{kv / 1024:>8.0f} KB") print("\nMemory for one sequence's attention state (BF16), by context length:") ctxs = [32_768, 131_072, 1_048_576] print(f"{'model':<20}" + "".join(f"{c:>14,}" for c in ctxs)) for name, (_, _, kv, fixed) in MODELS.items(): print(f"{name:<20}" + "".join(f"{(kv * c + fixed) / 1e9:>11.2f} GB" for c in ctxs)) base = MODELS["Qwen3-235B-A22B"][2] print(f"\nPer token, DeepSeek's MLA cache is {MODELS['DeepSeek-V3 / R1'][2] / base:.0%} of Qwen3-235B's GQA cache, " f"and Qwen3-Next's is {MODELS['Qwen3-Next-80B-A3B'][2] / base:.0%}.") print("Weights at FP8 = 1 byte per parameter; BF16 doubles it. Context lengths beyond each model's supported maximum are shown only for comparison.")
Output:
model total active active % GFLOPs/token weights FP8 KV/token DeepSeek-V3 / R1 671B 37B 5.5% 74 671 GB 69 KB Llama 4 Scout 109B 17B 15.6% 34 109 GB 192 KB Llama 4 Maverick 400B 17B 4.2% 34 400 GB 192 KB Qwen3-235B-A22B 235B 22B 9.4% 44 235 GB 188 KB Qwen3-30B-A3B 30.5B 3.3B 10.8% 7 30 GB 96 KB Qwen3-Next-80B-A3B 80B 3B 3.8% 6 80 GB 24 KB Memory for one sequence's attention state (BF16), by context length: model 32,768 131,072 1,048,576 DeepSeek-V3 / R1 2.30 GB 9.21 GB 73.69 GB Llama 4 Scout 6.44 GB 25.77 GB 206.16 GB Llama 4 Maverick 6.44 GB 25.77 GB 206.16 GB Qwen3-235B-A22B 6.31 GB 25.23 GB 201.86 GB Qwen3-30B-A3B 3.22 GB 12.88 GB 103.08 GB Qwen3-Next-80B-A3B 0.84 GB 3.26 GB 25.81 GB Per token, DeepSeek's MLA cache is 37% of Qwen3-235B's GQA cache, and Qwen3-Next's is 13%. Weights at FP8 = 1 byte per parameter; BF16 doubles it. Context lengths beyond each model's supported maximum are shown only for comparison.
Three lessons fall out. Compute per token tracks active parameters, so Maverick costs no more per token than Scout despite being nearly four times larger. Memory for weights tracks total parameters, so Maverick needs far more GPUs to hold. And attention design dominates long-context memory: MLA and hybrid linear attention cut per-sequence memory by large factors, which is why the next generation (DeepSeek-V4's compressed sparse attention, covered in Part 1) keeps pushing here. One caveat for Llama 4: three of every four layers use chunked attention over 8,192-token windows, so implementations that exploit this store less than the full-attention figure shown.
06 What Each Case Teaches
- A MoE model can reach high performance with low compute per token.
- Large-scale RL with verifiable rewards can teach reasoning โ but a small cold-start dataset makes it readable and stable.
- Distillation transfers reasoning to small open models cheaply.
- Sparse activation lets one family span very different sizes at the same per-token cost.
- Early fusion builds multimodality into pre-training.
- Layout choices (every-layer vs alternating MoE) trade memory for capacity.
- Benchmark claims need checking against the released model.
- Thinking vs non-thinking gives users control over depth, latency and cost.
- Hybrid linear + full attention can cut long-context cost dramatically.
- Very many small experts with few active keeps capacity high and compute low.
07 Architecture โ Training โ Inference Consequences
08 From Research Idea to Product Impact
09 Common Design Patterns Across Cases
10 What Remains Proprietary or Unknown
Even "open" models leave a lot unpublished โ and closed frontier models disclose far less still: