Home โ€บ Blog โ€บ The Great Flip
AI Compute Stack ยท Part 11 ยท Workloads

The Great Flip: Why AI Compute Is Shifting from Training to Inference

For a decade the story of AI compute was the training run: ever-bigger clusters building ever-bigger models. In 2026 the balance tipped. Forecasters now expect most AI compute to go to inference, answering questions, running agents and "thinking" before replying, rather than training. That shift changes which chips win, how data centres are designed, and who earns the money. This guide explains why it is happening, what happens inside a single request, and what the flip means for the rest of the AI compute stack.

FL
FrontierAGI Team
โš ๏ธ How this was researched This draws on web searches run up to 23 October 2026. The headline split (inference about two-thirds of AI compute in 2026) is a Deloitte forecast, not a measurement; Gartner's figures measure cloud spending rather than compute, and nobody publishes a measured market-wide split. Epoch AI's numbers on OpenAI's compute are estimates built from press reports. Details of Nvidia's GTC 2026 roadmap (Rubin CPX removed, Groq LPX added) come from trade-press coverage. The calculators use simplified physics with illustrative hardware numbers. See Reading the numbers.

๐Ÿ”„ The flip

Deloitte's 2026 technology predictions estimate that inference accounts for roughly two-thirds of all AI compute in 2026, up from about half in 2025 and a third in 2023. Gartner, measuring spending on AI-optimised cloud infrastructure rather than compute, puts inference at 55% in 2026, rising past 65% by 2029, and reports that inference spending ($23.3 billion) overtook training ($19 billion) in 2025. Different methods, same direction.

๐Ÿ“Š Inference's share of AI compute and spending

Solid = Deloitte compute estimates; hatched = Gartner spending on AI-optimised cloud infrastructure. Forecasts, not measurements.

Deloitte TMT Predictions 2026 and Gartner, via trade-press summaries. Gartner 2025: inference $23.3B vs training $19B (โ‰ˆ55%).

๐Ÿ’ก A useful counterweight Inside the frontier labs, "training" is mostly experiments. Epoch AI estimates that of OpenAI's roughly $5 billion of 2024 R&D compute, only about $500 million (โ‰ˆ10%) went to the final training runs of released models, with about $2 billion of compute on inference. By 2025 it describes OpenAI's compute as split roughly evenly between R&D and inference. So labs remain hungry for research compute even as the market as a whole tilts to serving.

โš–๏ธ Training vs inference, in plain terms

๐Ÿ” Two very different jobs

Pick a property.

TrainingInference
What happensThe model learns from trillions of tokens, adjusting its weightsThe finished model answers requests; weights are fixed
WhenOnce per model (plus many experiments), weeks to monthsContinuously, for as long as the model is in use
Shape of the workOne giant synchronised job across tens of thousands of chipsMillions of small independent requests, spiky through the day
Main limitRaw compute and the network between chipsMemory bandwidth and capacity (to read weights and conversation history)
What users feelNothing until the model shipsLatency: time to first word, words per second
Where it can runOne huge campus, ideally where power is cheapMany sites, closer to users; also phones and laptops
EconomicsA capital cost: pay up front, hope the model earns it backAn operating cost per token, directly tied to revenue

โฑ๏ธ When does serving overtake training?

A neat rule of thumb makes the flip intuitive. Training a dense transformer costs about 6 ร— parameters ร— training tokens floating-point operations; generating a token costs about 2 ร— parameters. So the model size cancels out: serving matches the whole training run once the model has processed three times as many tokens as it was trained on. A popular model can get there in weeks.

๐Ÿงฎ Days until inference compute exceeds training compute

Uses the 6ND / 2N rule. Ignores experiments, fine-tuning and serving overheads, so treat it as a teaching estimate.

Frontier models are trained on tens of trillions of tokens
From a niche model to a top consumer and API model

๐Ÿš€ Why it is happening

1. Models that think
Reasoning models such as OpenAI's o1 (September 2024) and DeepSeek-R1 (January 2025) generate long chains of hidden "thinking" tokens before answering. Research such as Snell et al. (2024) showed that spending more compute at answer time can beat training a bigger model for some tasks. Every hard question now costs many more tokens.
2. Agents
An agent completing a coding or research task makes dozens of model calls, re-reads large contexts and checks its own work. As The Token Price Collapse showed, agent and reasoning workloads use roughly 5โ€“30ร— more tokens per task than chat.
3. Usage
Hundreds of millions of people and most large companies now use AI daily, and cheaper tokens unlock still more demand.
4. Training is lumpy
Big training runs happen a few times a year per lab; inference grows every day a model is in use. Once a large base of models is deployed, serving them dominates.

๐Ÿ”ฌ Inside a single request: prefill and decode

Every request to a language model runs in two phases with opposite bottlenecks.

โ‘  Prefill (reading)

The model reads the whole prompt at once, all tokens in parallel, and builds the KV cache: a stored summary of every token's keys and values that later steps reuse. Thousands of tokens are processed together, so the chip's maths units stay busy. Compute-bound. It determines the time to first token.

โ‘ก Decode (writing)

The model writes the answer one token at a time. Each new token requires reading all the model's weights and the KV cache from memory, but does very little maths. Memory-bandwidth-bound. It determines tokens per second. Serving many users at once (batching) shares each weight read across many requests.

This is why inference cares so much about memory (see The Memory Wall): during decode, a GPU's expensive maths units often sit mostly idle, waiting on HBM.

๐ŸŽ›๏ธ Request simulator

โš™๏ธ Serve a model on one accelerator

Simplified roofline model of one Blackwell-class GPU (illustrative: ~2.5 PFLOP/s usable at FP8, 8 TB/s HBM). Model weights at 8 bits; KV-cache reads ignored; assumes the model fits.

Reasoning models can write thousands of hidden tokens

Prefill time = 2 ร— params ร— prompt tokens รท compute. Decode step = max(weights รท bandwidth, 2 ร— params ร— batch รท compute). Real systems add KV-cache traffic, attention cost on long contexts, multi-GPU splitting and scheduling overheads.

โœ‚๏ธ Splitting the work: disaggregated serving

Because prefill and decode want different things, running both on the same GPU wastes it: a long prompt arriving mid-conversation stalls everyone else's decode. Research systems Splitwise and DistServe (2024) showed the fix: run prefill on one pool of GPUs, decode on another, and pass the KV cache between them over fast links. Software such as vLLM (whose PagedAttention manages KV-cache memory like an operating system manages pages) and SGLang made this practical, and Nvidia's Dynamo 1.0 (March 2026) brought disaggregated serving to production, later splitting multimodal requests into encode, prefill and decode stages.

๐Ÿ”€ Disaggregated serving

Click a component.

Hardware is following. In September 2025 Nvidia announced Rubin CPX, a prefill chip using cheaper GDDR7 memory instead of HBM. At GTC in March 2026 CPX was absent from the keynote and roadmap; in its place Nvidia presented Groq 3 LPX racks (256 Groq LPUs with 128 GB of on-chip SRAM) to run alongside Vera Rubin NVL72 racks, with GPUs handling prefill and attention and LPUs speeding up decode. An Nvidia executive was reported as saying "we've pulled CPX. It's still a good idea."

๐Ÿ The inference specialists

Training is Nvidia's fortress. Inference is where challengers see an opening, because it rewards memory bandwidth, low latency and cost per token rather than giant synchronised clusters.

๐Ÿ“‹ Who is building for inference

Click a chip family.

๐Ÿข How data centres change

๐Ÿ—๏ธ A training campus vs an inference fleet

Toggle between the two designs.

๐Ÿงญ What the flip means

๐Ÿ•ฐ๏ธ Timeline

Sep 2023 ยท vLLM and PagedAttention paper: efficient KV-cache management for serving.
2024 ยท Splitwise and DistServe papers show the benefit of separating prefill and decode.
Augโ€“Sep 2024 ยท Snell et al. on scaling test-time compute; OpenAI releases o1, the first widely used reasoning model.
Jan 2025 ยท DeepSeek-R1 shows open reasoning models trained with reinforcement learning.
2025 ยท Gartner: inference spending ($23.3B) overtakes training ($19B). Agents spread across coding and research.
Sep 2025 ยท Nvidia announces Rubin CPX, a prefill-specialised chip.
Dec 2025 ยท Nvidia takes a non-exclusive licence to Groq's technology and hires key staff ($13B at closing plus $4B within a year, per Nvidia's annual report).
Mar 2026 ยท Nvidia Dynamo 1.0. At GTC, CPX disappears from the roadmap and Groq 3 LPX racks join Vera Rubin.
Apr 2026 ยท Google announces TPU 8i, an inference-focused TPU alongside the training-focused 8t.
May 2026 ยท Cerebras goes public, raising $5.55B, with a $20B, 750 MW inference agreement with OpenAI.
2026 ยท Deloitte: inference โ‰ˆ two-thirds of AI compute.

๐Ÿ”ญ What to watch

๐ŸŽฏ The bottom line Training builds a model once; inference runs it forever, and reasoning models and agents have multiplied the tokens each task needs. That is why most AI compute is shifting to serving. Inference is a different engineering problem, bound by memory and latency rather than raw compute and giant clusters, so it rewards different chips, software and data-centre designs. Nvidia still rules training, but the inference era is where custom silicon, Groq and Cerebras-style designs, and smarter serving software have their best chance, and where the revenue that pays for the whole build-out is actually earned.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References and papers