Home โ€บ Blog โ€บ Frontier Model Production Engineering
Frontier Architecture & Reasoning ยท Part 2 ยท Snapshot: September 2026 ๐Ÿญ

Frontier Model Production Engineering: Training, Serving and Operating Models at Scale

A research checkpoint is a file. A production model is a service that trains across tens of thousands of GPUs without losing weeks to failures, and then answers millions of users fast and cheaply. This visual deep dive walks the whole path โ€” parallelism, memory, MoE, reliability, serving, prefill and decode, KV cache, speculative decoding, kernels, quantization, metrics and deployment โ€” with interactive diagrams that compute the real numbers.

How to read this: every diagram is interactive and every number it shows is computed from the formula stated beside it. Hardware figures use NVIDIA's published H100 SXM specifications; claims about systems and papers link to their sources in References. The runnable simulator in section 06 uses only the Python standard library and its output is pasted verbatim.

01 The Production Challenge

Training produces a checkpoint: hundreds of billions of numbers plus research code written to run experiments, not to serve customers. Production engineering turns that into a service with guarantees โ€” fast first tokens, steady streaming, high GPU utilization, survival of hardware failures, and a cost per million tokens a business can live with. It happens on both sides of the checkpoint: engineering the training run so it finishes, and engineering the serving system so it scales.

From a research checkpointโ€ฆ
๐Ÿง  Trained weights (e.g. 100B+ parameters)
๐Ÿงช Research code and training framework
๐Ÿ–ฅ๏ธ Demanding hardware needs
๐Ÿข Not optimized for latency, cost or many tenants
๐Ÿšซ No production guardrails
โ€ฆto a production service
โœ… Optimized and quantized
โœ… High throughput, low latency
โœ… Scales to millions of users
โœ… Fault-tolerant and self-healing
โœ… Cost-efficient, monitored and controlled
โœ… Meets real-world SLAs

02 The Distributed Training Stack

No single GPU can hold a frontier model, its gradients and its optimizer state, let alone process trillions of tokens in reasonable time. Training splits the work along several axes at once โ€” "3D", "4D" or "5D" parallelism. Pick a strategy to see what each GPU holds:

โ–ถ What gets split? Four GPUs, one small model with 4 layers. Colours show which piece each GPU holds.
StrategyWhat is splitCommunicationOrigin / example
Data parallelThe batch; each GPU has a full model replicaAll-reduce gradients every stepPyTorch DDP
Tensor parallelEach weight matrix, column- or row-wiseAll-reduce inside every layer โ€” needs fast NVLink, so kept within a serverMegatron-LM (Shoeybi et al., 2019)
Pipeline parallelLayers into stages; micro-batches flow throughPoint-to-point activations between stagesGPipe (Huang et al., 2019), 1F1B schedules
Expert parallelMoE experts across GPUsAll-to-all token dispatch and combineGShard, DeepSeek-V3
Context / sequence parallelThe sequence (tokens) itselfExchange keys/values (ring attention) or all-gatherMegatron sequence parallelism, Llama 3 context parallelism
Gradient accumulationA large batch into micro-batches over timeNone until the final updateStandard in every trainer

Pipeline parallelism has a catch: while the first micro-batch travels through the stages, later stages sit idle, and they idle again as the pipeline drains. More micro-batches shrink that "bubble":

โ–ถ The pipeline bubble Forward passes (purple) then backward passes (amber), GPipe-style. Grey = idle.
Idle fraction = (p โˆ’ 1) รท (m + p โˆ’ 1), the standard GPipe bubble formula, computed live and matching the drawn schedule. Interleaved and zero-bubble schedules shrink it further; DeepSeek-V3's DualPipe overlaps computation with communication for the same reason.
Documented example โ€” Llama 3 405B: Meta trained it on up to 16,384 H100 GPUs using 4D parallelism: tensor, pipeline, context and data (FSDP) parallelism, with tensor parallelism kept inside each 8-GPU server where NVLink is fastest.

03 Memory Engineering During Training

With mixed-precision Adam, each parameter costs about 16 bytes before any activations: 2 for the BF16 weight, 2 for its gradient, and 12 of optimizer state (an FP32 master copy plus two FP32 Adam moments). That is why a 70B model needs over a terabyte just for model state. ZeRO (the idea behind DeepSpeed ZeRO and PyTorch FSDP) shards these pieces across data-parallel GPUs instead of replicating them:

โ–ถ ZeRO sharding calculator Model-state memory per GPU. Activations are extra.
Per GPU: stage 0 = 16ฮจ; stage 1 (shard optimizer) = 4ฮจ + 12ฮจ/N; stage 2 (+ gradients) = 2ฮจ + 14ฮจ/N; stage 3 (+ parameters, like FSDP full sharding) = 16ฮจ/N, where ฮจ = parameter count โ€” the formulas from Rajbhandari et al. (2020). The red line is an 80 GB H100.
๐ŸŽฏ Mixed precisionCompute in BF16 (or FP8) while keeping an FP32 master copy for stable updates (Micikevicius et al., 2018). DeepSeek-V3 was trained largely in FP8.
โ™ป๏ธ Activation checkpointingKeep only some layers' activations and recompute the rest during the backward pass: trades roughly one extra forward pass for far less memory โ€” as little as O(โˆšn) for n layers (Chen et al., 2016).
๐Ÿงฉ Optimizer & parameter shardingZeRO stages 1โ€“3 / FSDP, above: shard state and gather it just in time.
๐Ÿ’พ Offload to CPU / NVMeZeRO-Offload and ZeRO-Infinity move optimizer states or parameters to host memory or SSDs when GPU memory runs out, at the cost of bandwidth.
๐Ÿ“ Sequence parallelismSplit activations along the sequence so long-context training fits.
๐Ÿ“ฅ Training checkpointsSave state every N steps โ€” asynchronously to distributed storage, so GPUs keep working while the file is written.

04 MoE Production Engineering

Mixture-of-Experts models (see Part 1) are cheap per token but hard to run: experts live on different GPUs, so every MoE layer sends each token to its experts' GPUs and back โ€” an all-to-all exchange twice per layer. And tokens don't spread evenly: if the router over-picks one expert, its GPU becomes the straggler everyone waits for. Systems cap each expert at a capacity:

โ–ถ Load imbalance and capacity factor 512 tokens routed to 8 experts with a skewed router (simulated with a fixed seed).
Capacity per expert = capacity factor ร— tokens รท experts (Switch Transformer). Tokens beyond capacity are dropped (skip the layer via the residual path) or the expert's GPU simply takes longer. Balancing losses or DeepSeek-V3's auxiliary-loss-free bias correction push the bars flat.
Routing strategy
  • Top-k / top-1
  • Load-balancing loss or bias
  • Capacity limits
Expert placement
  • Distribute experts across GPUs and nodes
  • Balance compute and memory
  • Replicate hot experts
All-to-all communication
  • Efficient token dispatch (e.g. DeepEP)
  • Overlap with compute
  • Limit nodes per token
Inference optimizations
  • Expert caching
  • Large expert-parallel serving
  • Fewer cross-node hops
Documented example โ€” DeepSeek-V3: each token is routed to at most 4 nodes to bound cross-node traffic, and the DualPipe schedule overlaps the all-to-all communication with computation. DeepSeek open-sourced DeepEP, its expert-parallel communication library, in February 2025.

05 Training Reliability at Scale

At frontier scale, hardware failure is not an exception โ€” it is the weather. Meta's Llama 3 paper reports the numbers from a 54-day snapshot of pre-training on 16,384 H100 GPUs:

โ–ถ 419 unexpected interruptions in 54 days Root causes reported in the Llama 3 paper (Table 5). One interruption about every 3 hours.
Of 466 job interruptions, 47 were planned (e.g. firmware upgrades) and 419 unexpected; about 78% of the unexpected ones were confirmed or suspected hardware issues. Despite this, Meta reports over 90% effective training time, thanks to automation.

Because failures are certain, the question is how often to checkpoint. Checkpoint too often and you spend all your time writing; too rarely and each failure throws away hours. Young's classic approximation gives the sweet spot:

โ–ถ How often should you checkpoint? Wasted time = time writing checkpoints + work lost on failure.
Waste fraction โ‰ˆ C/T + T/(2M) for checkpoint time C, interval T and mean time between failures M (each failure loses half an interval on average); the minimum is at T* = โˆš(2CM) (Young, 1974). Default M = 185 min โ‰ˆ 54 days รท 419 interruptions. Restart time is ignored. Asynchronous checkpointing shrinks C โ€” which is why it matters.
๐Ÿ“ฅ Fault-tolerant checkpointingSave model, optimizer and RNG state; sharded and asynchronous.
โ–ถ๏ธ ResumabilityDeterministic restart from the last checkpoint, including the data-loader position.
๐Ÿข Straggler handlingDetect and replace slow nodes: one slow GPU slows every synchronous step.
๐Ÿฉบ Cluster health monitoringWatch GPU, memory, network and temperature; drain bad hosts before they fail the job.
๐Ÿ“ˆ Rolling validationRun evals during training to catch loss spikes and regressions early.
โธ๏ธ Preemption resilienceHandle cloud preemptions and maintenance gracefully.

06 Inference Serving Architecture

A request's journey from user to streamed tokens passes through several layers. Press play:

โ–ถ Follow a request From the user's prompt to streamed tokens.
User โ†’ API gateway โ†’ router โ†’ scheduler โ†’ model workers โ†’ token stream

The scheduler is where the biggest gains are. Static batching runs a batch until its longest request finishes, leaving finished slots empty. Continuous batching (iteration-level scheduling, from Orca, OSDI 2022) admits a new request the moment another finishes. And paged KV cache (PagedAttention, from vLLM) stops reserving maximum-length memory per request. This simulator measures all four combinations on the same 400 requests:

"""Static vs continuous batching, contiguous vs paged KV cache: a tiny LLM-serving simulator.

Time advances in decode steps. Each step, every running request produces one token. A GPU has room for a
fixed number of KV-cache slots (tokens). Static batching waits for a whole batch to finish before admitting
new requests; continuous batching (Orca / vLLM style) admits a waiting request the moment one finishes.
Contiguous allocation reserves max_len slots per request up front; paged allocation takes 16-token blocks
as the sequence grows. Standard library only.
"""
import random

KV_SLOTS, MAX_LEN, BLOCK = 16_384, 2_048, 16

def workload(n, seed=7):
    rng = random.Random(seed)
    # prompt 50-1000 tokens, output 20-1000 tokens: long-tailed like real chat traffic
    return [(rng.randint(50, 1000), min(int(rng.expovariate(1 / 250)) + 20, 1000)) for _ in range(n)]

def simulate(reqs, continuous, paged):
    waiting, running, t, done, lat = list(enumerate(reqs)), [], 0, 0, []
    used_hist = []
    def need(p, gen):                                    # KV slots a request holds right now
        return -(-(p + gen) // BLOCK) * BLOCK if paged else MAX_LEN
    while waiting or running:
        if continuous or not running:                    # admission
            while waiting:
                i, (p, o) = waiting[0]
                if sum(need(r[1], r[3]) for r in running) + need(p, 1) > KV_SLOTS: break
                waiting.pop(0); running.append([i, p, o, 0])
        t += 1
        for r in running: r[3] += 1
        used_hist.append(sum(r[1] + r[3] for r in running) / KV_SLOTS)
        for r in [r for r in running if r[3] >= r[2]]:
            running.remove(r); done += 1; lat.append(t)
    toks = sum(o for _, o in reqs)
    return t, toks / t, sum(lat) / len(lat), sum(used_hist) / len(used_hist)

reqs = workload(400)
print(f"400 requests, {sum(o for _, o in reqs):,} output tokens, GPU KV budget {KV_SLOTS:,} token slots\n")
print("   scheduler                     steps   tokens/step   mean completion step   KV slots holding real tokens")
base = None
for name, c, pg in [("static batch + contiguous KV", False, False), ("continuous + contiguous KV", True, False),
                    ("static batch + paged KV", False, True), ("continuous + paged KV", True, True)]:
    t, tps, ml, util = simulate(reqs, c, pg)
    base = base or tps
    print(f"   {name:<29} {t:>6,}   {tps:>11.1f}   {ml:>20,.0f}   {util:>27.0%}   ({tps / base:.1f}x)")

Output (deterministic โ€” fixed random seed):

400 requests, 103,109 output tokens, GPU KV budget 16,384 token slots

   scheduler                     steps   tokens/step   mean completion step   KV slots holding real tokens
   static batch + contiguous KV  34,107           3.0                 17,227                           13%   (1.0x)
   continuous + contiguous KV    13,564           7.6                  6,606                           34%   (2.5x)
   static batch + paged KV       11,591           8.9                  5,551                           39%   (2.9x)
   continuous + paged KV          5,345          19.3                  2,314                           85%   (6.4x)
static batch + contiguous KV 3.0 tokens/step ยท 1.0ร— continuous + contiguous KV 7.6 tokens/step ยท 2.5ร— static batch + paged KV 8.9 tokens/step ยท 2.9ร— continuous + paged KV 19.3 tokens/step ยท 6.4ร—

The simulator counts decode steps, not seconds: in a real GPU a larger batch makes each step slightly slower, so real gains are smaller than 6.4ร— โ€” the vLLM paper reports 2โ€“4ร— throughput over prior systems at the same latency. The direction and the reason are the same: 13% of KV slots held real tokens with contiguous reservation and static batches, 85% with paging and continuous batching.

07 Prefill vs Decode: Two Different Workloads

Generating a response has two phases. Prefill reads the whole prompt in one parallel pass and builds the KV cache; it is compute-bound, and it determines time to first token. Decode generates one token at a time, and each step must read every weight from memory to do very little arithmetic per request; it is memory-bandwidth-bound and determines the time between tokens. Batching more requests per decode step is the main way to use a GPU's compute:

โ–ถ Compute-bound or memory-bound? A 70B-parameter BF16 model on one H100 SXM: 989 TFLOPS dense BF16, 3.35 TB/s memory bandwidth.
Step time โ‰ˆ max(2 ร— parameters ร— tokens รท FLOPS, weight bytes รท bandwidth) โ€” a roofline estimate that ignores attention and KV-cache reads (which grow with context) and assumes the model fits, which a 140 GB model does only when sharded. The crossover is at about FLOPS รท bandwidth โ‰ˆ 295 tokens: below it, reading weights dominates.
Prefill (context processing)
All prompt tokens in parallel ยท compute-bound (large matrix multiplies) ยท high GPU utilization ยท latency grows with prompt length ยท sets time to first token (TTFT).
Decode (token generation)
One new token per request per step ยท latency-sensitive and sequential ยท memory-bandwidth-bound ยท reads the KV cache built by prefill ยท sets inter-token latency (ITL / TPOT).

Mixing them on one GPU causes interference: a long prefill stalls every decoding request sharing its batch. Chunked prefill (Sarathi-Serve) splits long prompts into chunks that piggyback on decode steps; disaggregation separates them entirely.

08 Disaggregated Prefill and Decode

Disaggregated serving runs prefill and decode on separate GPU pools. A prefill worker processes the prompt, then transfers the KV cache over the network (RDMA or NVLink) to a decode worker, which streams the tokens. Each pool can use its own batch size, parallelism and even hardware, so TTFT and inter-token latency can be tuned independently.

โ–ถ Two pools, one request Press play to watch a prompt move from the prefill pool to the decode pool.
Prefill pool (throughput) โ†’ KV transfer โ†’ decode pool (latency)
Documented results: DistServe (OSDI 2024) reports serving 7.4ร— more requests, or meeting 12.6ร— tighter latency targets, than state-of-the-art colocated systems while keeping over 90% of requests within their latency limits. Splitwise (ISCA 2024) reports 1.4ร— higher throughput at 20% lower cost, or 2.35ร— more throughput for the same cost and power. Moonshot AI's Mooncake, which serves Kimi, is built around a disaggregated, KV-cache-centric design.
Production status, September 2026: vLLM's documentation still labels its disaggregated prefilling "experimental", with KV transfer through connectors such as NIXL; production deployments commonly run vLLM or SGLang under an orchestrator such as NVIDIA Dynamo or llm-d, which add routing and scheduling. NVIDIA TensorRT-LLM also supports disaggregated serving. Disaggregation improves goodput โ€” requests served within latency targets โ€” rather than raw peak throughput, and it costs network bandwidth for KV transfer.

09 KV-Cache Engineering

The KV cache โ€” keys and values of every earlier token โ€” is often the largest consumer of serving memory, and managing it decides how many requests fit at once. Before paging, systems reserved a contiguous region per request at maximum length; the vLLM paper measured that 60โ€“80% of KV memory in existing systems was wasted by fragmentation and over-reservation.

โ–ถ Contiguous vs paged allocation The same three growing requests in 48 memory slots. Press play.
Contiguous (reserve max length)
Paged (4-token blocks on demand)
Paged attention stores each sequence as fixed-size blocks anywhere in memory, found through a block table (logical โ†’ physical), like virtual-memory pages. Waste is limited to the last, partly filled block of each sequence.
๐Ÿ“„ Paged / block-based KV cacheAllocate on demand; no fragmentation (vLLM PagedAttention).
๐Ÿ” Prefix cachingReuse KV for shared prefixes โ€” system prompts, few-shot examples, conversation history. SGLang's RadixAttention keeps a radix tree of cached prefixes.
๐Ÿšš KV-cache transferMove KV between GPUs or nodes (prefill โ†’ decode) over RDMA; NIXL is the common library.
๐Ÿ—„๏ธ Tiering and offloadGPU HBM โ†’ CPU memory โ†’ NVMe for long or idle contexts (e.g. LMCache, Mooncake's KV store).
๐Ÿ’ฌ Multi-turn reuseKeep a conversation's cache between turns, with eviction when memory is tight.
๐Ÿ—œ๏ธ CompressionGQA, latent attention (MLA), FP8/FP4 KV โ€” see Part 1.

10 Speculative Decoding

Decode is memory-bound: checking five tokens costs the big model almost the same as generating one, because the weights are read once either way. Speculative decoding exploits this. A small, fast draft proposes k tokens; the large target model verifies all of them in a single forward pass, keeps the longest accepted prefix, and adds one token of its own. With the right acceptance rule the output distribution is exactly the target model's (Leviathan et al., 2023; Chen et al., 2023).

โ–ถ Draft, verify, accept Each round: the draft proposes k tokens, the target checks them in one pass.
Expected tokens per target pass = (1 โˆ’ ฮฑk+1) รท (1 โˆ’ ฮฑ), and speedup โ‰ˆ that รท (k ร— c + 1) for draft cost c relative to the target โ€” both from Leviathan et al. (2023), assuming independent acceptances. Note one correction to the common diagram: a round yields up to k + 1 tokens, not k, because the target always contributes one token after the accepted prefix.

Production variants avoid a separate draft model: Medusa adds extra decoding heads to the target; EAGLE drafts from the target's own hidden features (EAGLE-3 reports 3.0โ€“6.5ร— speedups); Apple's ReDrafter uses a recurrent draft head; n-gram / prompt lookup copies spans from the context; and DeepSeek-V3's multi-token-prediction module can serve as a draft. vLLM, SGLang and TensorRT-LLM all support several of these. Gains shrink at large batch sizes, where decode becomes less memory-bound.

11 Kernel and Execution Optimization

A GPU computes far faster than it can move data to and from its main memory (HBM). Many operations in a transformer โ€” layer norms, activations, softmax โ€” do little arithmetic per byte, so their cost is the memory round trip. Kernel fusion runs several of them in one kernel, keeping intermediate results in fast on-chip memory:

โ–ถ Unfused vs fused Four element-wise operations on one tensor: how many times does it travel through HBM?
๐Ÿ”— Fused kernelsFuse attention, norms, activations and MLP pieces to cut memory traffic.
โšก FlashAttentionComputes attention in tiles held in on-chip SRAM, never writing the full n ร— n score matrix to HBM (Dao et al., 2022). FlashAttention-3 targets H100 features such as asynchronous tensor cores and FP8.
๐Ÿ“ธ CUDA GraphsCapture a sequence of kernel launches once and replay it, removing CPU launch overhead that dominates small decode steps.
๐Ÿ› ๏ธ Compilers & JITtorch.compile, TensorRT, Triton, XLA generate optimized, fused kernels automatically.
๐Ÿงฑ Hardware-aware kernelsVendor libraries (cuBLAS, CUTLASS, NVIDIA, AMD and custom ASIC stacks) tuned for each chip.
๐Ÿ”€ Asynchronous schedulingOverlap compute, communication and data transfer so no unit waits.

12 Quantization and Precision Engineering

Fewer bits per weight means less memory, less bandwidth per decode step, and faster math on hardware that supports the format. The trade-off is accuracy, managed with calibration, per-channel or per-block scaling, and quantization-aware training.

โ–ถ How many GPUs just to hold the weights? Weights only; KV cache and activations need more.
Weight memory = parameters ร— bits รท 8; GPUs = memory รท 80 GB, rounded up. Decode speed on a memory-bound GPU scales roughly with bytes read, so halving the bits can nearly double decode throughput per GPU.
PrecisionTypical useBenefitsTrade-offs
FP32Master weights, optimizer stateHighest precision4 bytes per value; slow
BF16Standard training and inferenceFP32's range, stable trainingTwice FP8's memory
FP16Training (with loss scaling) and inferenceGood precisionNarrow range โ†’ overflow risk
FP8Large-scale training (DeepSeek-V3) and inferenceHalf of BF16's memory, ~2ร— tensor throughput on H100Needs fine-grained scaling; hardware support (Hopper and later)
INT8Inference (weights and/or activations)Lower memory, higher throughputPossible quality loss; outliers need handling (SmoothQuant)
INT4 / FP4Aggressive inference (GPTQ, AWQ; NVFP4 on Blackwell)Very low memory, high decode throughputLarger quality risk; more careful calibration

13 Production Metrics

Users feel two latencies: how long until the first token appears (TTFT), and how fast the rest streams (inter-token latency). Operators watch throughput, utilization and cost. Their relationship:

โ–ถ What the user waits for End-to-end latency = TTFT + (output tokens โˆ’ 1) ร— TPOT.
TTFTTime to first token ยท lower is better ยท driven by queueing + prefill
TPOT / ITLTime per output token (average) / inter-token latency (per gap) ยท lower is better
ThroughputTokens per second across all users ยท higher is better
GoodputRequests or tokens per second that meet the latency SLOs ยท higher is better
GPU utilizationCompute (MFU) and memory-bandwidth use ยท higher is better
Queue timeTime waiting in the scheduler ยท lower is better
KV-cache hit rateShare of prompt tokens served from prefix cache ยท higher is better
Tail latencyp95 / p99 โ€” the experience of the unluckiest users ยท lower is better
Cost per 1M tokensUsually priced separately for input and output tokens ยท lower is better

14 Reliability and Deployment

New model versions roll out like any risky software change โ€” gradually, with metrics gates and a fast way back:

โ–ถ Canary rollout Traffic shifts to the new version in steps; a metric regression triggers rollback.
๐Ÿ”ฅ Model warm-upLoad the model, compile kernels, fill caches before routing traffic.
๐Ÿค Canary & rolling deploysShift traffic gradually while watching metrics.
๐Ÿ“ˆ AutoscalingAdd and remove replicas with load; slow model loads make it harder than for web servers.
๐Ÿ”€ FallbacksRoute to a backup model or region when one fails.
๐Ÿ“ Capacity planningPlan for peak load, memory and cost.
๐Ÿšฆ Backpressure & rate limitsPrevent overload and cascading failures.
๐Ÿšจ Alerting & incident responseMonitor, alert and auto-recover common failures.

15 The End-to-End Production Path

1Model checkpointtrained weights
2Shard / quantize / compileTensorRT-LLM, vLLM, custom
3Load workerson a GPU cluster
4Warm upcompile, cache, health check
5Route requestscheduler
6Prefillprocess prompt
7Store KV cachepaged blocks
8Decodegenerate tokens
9Stream responseto the user
10Observe metricslogs, traces, dashboards
11Autoscale / recoverscale out, replace failed nodes, retry
The production engineering principle: optimize the whole system, not just model FLOPs. Bottlenecks move between compute, memory bandwidth, KV transfer, networking and scheduling as workloads change โ€” measure, profile and optimize end to end.
Production engineering in one paragraph
Training a frontier model means splitting it five ways across thousands of GPUs, sharding its 16 bytes per parameter of state, overlapping MoE all-to-all traffic with compute, and checkpointing often enough to survive a failure every few hours. Serving it means recognizing that prefill is compute-bound and decode memory-bound, then attacking decode from every side: continuous batching and paged KV cache to fit more requests, prefix caching to skip repeated work, speculative decoding to get several tokens per weight read, fused kernels and lower precision to move fewer bytes, and disaggregated pools so the two phases stop interfering. Then it is operated like any critical service โ€” measured by TTFT, inter-token latency, goodput and cost per million tokens, rolled out behind canaries, and designed to fail over.

Next: The Frontier Model Market Landscape โ€” who builds, trains, serves and sells these models, and what they cost.

โš ๏ธ Notes

The source infographic is accurate overall; corrections and refinements made here: (1) speculative decoding โ€” it shows "accept m โ‰ค k tokens"; a round actually yields up to k + 1 tokens because the target model adds one token after the accepted prefix; (2) production metrics โ€” "Cost per 1M tokens" appears twice; it is usually quoted separately for input and output tokens, which may have been the intent; (3) FP8 is labelled "training (emerging)", but it has been used for frontier-scale training since DeepSeek-V3 (December 2024) and is standard for inference on Hopper-class GPUs; (4) TPOT and ITL are shown as one metric โ€” TPOT is usually the average per request, ITL the individual gaps (whose tail matters for streaming smoothness). Its statement that disaggregation "does not inherently increase overall throughput" is fair: the published gains are in goodput under latency targets, although Splitwise also reports throughput-per-cost gains. The simulator in section 06 is a simplified model; the other interactive figures follow the stated formulas.

๐Ÿ”— References

Llama Team, Meta โ€” The Llama 3 Herd of Models (arXiv:2407.21783) Shoeybi et al. โ€” Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (arXiv:1909.08053) Huang et al. โ€” GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (arXiv:1811.06965) Rajbhandari et al. โ€” ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054) Zhao et al. โ€” PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (arXiv:2304.11277) Micikevicius et al. โ€” Mixed Precision Training (arXiv:1710.03740) Chen et al. โ€” Training Deep Nets with Sublinear Memory Cost (arXiv:1604.06174) Fedus et al. โ€” Switch Transformers (capacity factor) (arXiv:2101.03961) DeepSeek-AI โ€” DeepSeek-V3 Technical Report (FP8 training, DualPipe, node-limited routing) (arXiv:2412.19437) DeepSeek โ€” DeepEP expert-parallel communication library Yu et al. โ€” Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) Kwon et al. โ€” Efficient Memory Management for LLM Serving with PagedAttention (arXiv:2309.06180) Zheng et al. โ€” SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) (arXiv:2312.07104) Agrawal et al. โ€” Sarathi-Serve: Taming Throughput-Latency Tradeoff (chunked prefill) (arXiv:2403.02310) Zhong et al. โ€” DistServe: Disaggregating Prefill and Decoding (arXiv:2401.09670) Patel et al. โ€” Splitwise: Efficient Generative LLM Inference Using Phase Splitting (arXiv:2311.18677) Qin et al. โ€” Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079) vLLM documentation โ€” Disaggregated Prefilling (experimental) llm-d โ€” Disaggregated serving with vLLM NVIDIA Dynamo โ€” Disaggregated serving design Leviathan et al. โ€” Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192) Chen et al. โ€” Accelerating LLM Decoding with Speculative Sampling (arXiv:2302.01318) Cai et al. โ€” Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads (arXiv:2401.10774) Li et al. โ€” EAGLE-3: Scaling up Inference Acceleration via Training-Time Test (arXiv:2503.01840) Cheng et al. โ€” Recurrent Drafter for Fast Speculative Decoding (ReDrafter) (arXiv:2403.09919) Dao et al. โ€” FlashAttention (arXiv:2205.14135) Shah et al. โ€” FlashAttention-3 (arXiv:2407.08608) Xiao et al. โ€” SmoothQuant (arXiv:2211.10438) Lin et al. โ€” AWQ: Activation-aware Weight Quantization (arXiv:2306.00978) NVIDIA โ€” H100 Tensor Core GPU specifications