The Great Flip: Why AI Compute Is Shifting from Training to Inference
For a decade the story of AI compute was the training run: ever-bigger clusters building ever-bigger models. In 2026 the balance tipped. Forecasters now expect most AI compute to go to inference, answering questions, running agents and "thinking" before replying, rather than training. That shift changes which chips win, how data centres are designed, and who earns the money. This guide explains why it is happening, what happens inside a single request, and what the flip means for the rest of the AI compute stack.
๐ The flip
Deloitte's 2026 technology predictions estimate that inference accounts for roughly two-thirds of all AI compute in 2026, up from about half in 2025 and a third in 2023. Gartner, measuring spending on AI-optimised cloud infrastructure rather than compute, puts inference at 55% in 2026, rising past 65% by 2029, and reports that inference spending ($23.3 billion) overtook training ($19 billion) in 2025. Different methods, same direction.
๐ Inference's share of AI compute and spending
Solid = Deloitte compute estimates; hatched = Gartner spending on AI-optimised cloud infrastructure. Forecasts, not measurements.
Deloitte TMT Predictions 2026 and Gartner, via trade-press summaries. Gartner 2025: inference $23.3B vs training $19B (โ55%).
โ๏ธ Training vs inference, in plain terms
๐ Two very different jobs
Pick a property.
| Training | Inference | |
|---|---|---|
| What happens | The model learns from trillions of tokens, adjusting its weights | The finished model answers requests; weights are fixed |
| When | Once per model (plus many experiments), weeks to months | Continuously, for as long as the model is in use |
| Shape of the work | One giant synchronised job across tens of thousands of chips | Millions of small independent requests, spiky through the day |
| Main limit | Raw compute and the network between chips | Memory bandwidth and capacity (to read weights and conversation history) |
| What users feel | Nothing until the model ships | Latency: time to first word, words per second |
| Where it can run | One huge campus, ideally where power is cheap | Many sites, closer to users; also phones and laptops |
| Economics | A capital cost: pay up front, hope the model earns it back | An operating cost per token, directly tied to revenue |
โฑ๏ธ When does serving overtake training?
A neat rule of thumb makes the flip intuitive. Training a dense transformer costs about 6 ร parameters ร training tokens floating-point operations; generating a token costs about 2 ร parameters. So the model size cancels out: serving matches the whole training run once the model has processed three times as many tokens as it was trained on. A popular model can get there in weeks.
๐งฎ Days until inference compute exceeds training compute
Uses the 6ND / 2N rule. Ignores experiments, fine-tuning and serving overheads, so treat it as a teaching estimate.
๐ Why it is happening
- 1. Models that think
- Reasoning models such as OpenAI's o1 (September 2024) and DeepSeek-R1 (January 2025) generate long chains of hidden "thinking" tokens before answering. Research such as Snell et al. (2024) showed that spending more compute at answer time can beat training a bigger model for some tasks. Every hard question now costs many more tokens.
- 2. Agents
- An agent completing a coding or research task makes dozens of model calls, re-reads large contexts and checks its own work. As The Token Price Collapse showed, agent and reasoning workloads use roughly 5โ30ร more tokens per task than chat.
- 3. Usage
- Hundreds of millions of people and most large companies now use AI daily, and cheaper tokens unlock still more demand.
- 4. Training is lumpy
- Big training runs happen a few times a year per lab; inference grows every day a model is in use. Once a large base of models is deployed, serving them dominates.
๐ฌ Inside a single request: prefill and decode
Every request to a language model runs in two phases with opposite bottlenecks.
โ Prefill (reading)
The model reads the whole prompt at once, all tokens in parallel, and builds the KV cache: a stored summary of every token's keys and values that later steps reuse. Thousands of tokens are processed together, so the chip's maths units stay busy. Compute-bound. It determines the time to first token.โก Decode (writing)
The model writes the answer one token at a time. Each new token requires reading all the model's weights and the KV cache from memory, but does very little maths. Memory-bandwidth-bound. It determines tokens per second. Serving many users at once (batching) shares each weight read across many requests.This is why inference cares so much about memory (see The Memory Wall): during decode, a GPU's expensive maths units often sit mostly idle, waiting on HBM.
๐๏ธ Request simulator
โ๏ธ Serve a model on one accelerator
Simplified roofline model of one Blackwell-class GPU (illustrative: ~2.5 PFLOP/s usable at FP8, 8 TB/s HBM). Model weights at 8 bits; KV-cache reads ignored; assumes the model fits.
Prefill time = 2 ร params ร prompt tokens รท compute. Decode step = max(weights รท bandwidth, 2 ร params ร batch รท compute). Real systems add KV-cache traffic, attention cost on long contexts, multi-GPU splitting and scheduling overheads.
โ๏ธ Splitting the work: disaggregated serving
Because prefill and decode want different things, running both on the same GPU wastes it: a long prompt arriving mid-conversation stalls everyone else's decode. Research systems Splitwise and DistServe (2024) showed the fix: run prefill on one pool of GPUs, decode on another, and pass the KV cache between them over fast links. Software such as vLLM (whose PagedAttention manages KV-cache memory like an operating system manages pages) and SGLang made this practical, and Nvidia's Dynamo 1.0 (March 2026) brought disaggregated serving to production, later splitting multimodal requests into encode, prefill and decode stages.
๐ Disaggregated serving
Click a component.
Hardware is following. In September 2025 Nvidia announced Rubin CPX, a prefill chip using cheaper GDDR7 memory instead of HBM. At GTC in March 2026 CPX was absent from the keynote and roadmap; in its place Nvidia presented Groq 3 LPX racks (256 Groq LPUs with 128 GB of on-chip SRAM) to run alongside Vera Rubin NVL72 racks, with GPUs handling prefill and attention and LPUs speeding up decode. An Nvidia executive was reported as saying "we've pulled CPX. It's still a good idea."
๐ The inference specialists
Training is Nvidia's fortress. Inference is where challengers see an opening, because it rewards memory bandwidth, low latency and cost per token rather than giant synchronised clusters.
๐ Who is building for inference
Click a chip family.
๐ข How data centres change
๐๏ธ A training campus vs an inference fleet
Toggle between the two designs.
๐งญ What the flip means
- For Nvidia: its moat (NVLink, CUDA, giant training clusters) matters most in training. Inference opens room for custom silicon, Cerebras and Groq-style chips, which is why Nvidia licensed Groq's technology and folded LPUs into its own roadmap.
- For the clouds: inference is steady, usage-based revenue that grows with adoption, which makes data centres easier to finance than one-off training bets (see Who Pays for the AI Build-Out?).
- For prices: the cost per token keeps falling as serving gets more efficient, but the number of tokens per task keeps rising (The Token Price Collapse).
- For power and location: inference can be spread across many sites closer to users, easing the single-campus power bottleneck, but its demand follows daily usage peaks.
- For memory: long contexts and agents make KV-cache capacity a first-class problem, increasing demand for HBM and new memory tiers.
๐ฐ๏ธ Timeline
๐ญ What to watch
- Groq LPX availability (slated for the second half of 2026) and whether a CPX-style prefill chip returns.
- Cerebras' first results as a public company and how much of its revenue depends on OpenAI.
- TPU 8i, Maia 200 and MTIA deployments: how much inference moves off Nvidia inside the hyperscalers.
- Reasoning efficiency. Research on shorter chains of thought and adaptive "thinking budgets" could slow token growth.
- Training's comeback. New scaling methods (reinforcement learning on agents, synthetic data) could push training spend back up.
๐ข Reading the numbers
- "Two-thirds" is a Deloitte forecast. No one publishes a measured market-wide split, and definitions differ (compute vs spending, lab R&D vs commercial serving).
- Gartner's figures cover AI-optimised cloud infrastructure spending, not all compute; they come via secondary summaries.
- Epoch AI's OpenAI estimates are built from press reports; OpenAI does not disclose its allocation.
- GTC 2026 roadmap details (CPX removal, LPX specs, the quoted remark) come from trade press, not Nvidia's own materials.
- Groq deal terms are from Nvidia's annual report; the widely reported $20 billion figure was never confirmed.
- Calculators use textbook approximations (6ND, 2N, roofline) and illustrative hardware figures.
๐ References and papers
- Papers
- Snell et al., "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters" (2024)
- DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025)
- Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (vLLM, 2023)
- Patel et al., "Splitwise: Efficient Generative LLM Inference Using Phase Splitting" (2024)
- Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving" (2024)
- Kaplan et al., "Scaling Laws for Neural Language Models" (source of the 6ND / 2N approximations)
- Data and analysis
- Epoch AI: Most of OpenAI's 2024 compute went to experiments
- Epoch AI: Final training runs account for a minority of R&D compute spending
- Epoch AI: AI chip users explorer
- Summary of Deloitte's inference two-thirds forecast
- Training vs inference: what the data shows (Gartner, Deloitte, Epoch compared)
- Hardware and software
- Tom's Hardware: Nvidia removes Rubin CPX from its roadmap; Groq 3 LPUs take centre stage
- StorageReview: GTC 2026 โ Rubin GPUs, Groq LPUs, Vera CPUs
- Nvidia: When to use encode-prefill-decode disaggregation
- Nvidia fiscal 2026 annual report (Groq licence terms)
- Futurum: Cerebras S-1 teardown
- TrendForce: OpenAI reportedly discontent with Nvidia GPUs for inference
- Related on this site
- The Memory Wall ยท Custom Silicon vs Nvidia ยท The Token Price Collapse ยท Who Pays for the AI Build-Out?