Reasoning Models & Test-Time Compute:
The Second Scaling Law

Our AI Chips & Compute Infrastructure post ended with a note that test-time compute adds "a second, independent axis" of demand beyond training. This post is about that axis in full: what it actually is, why it works, and why it's arguably the biggest architectural shift in frontier AI since the Transformer paper itself. For a decade, capability scaled almost entirely with training-time compute — bigger models, more data, more pretraining FLOPs, all fixed before a user ever sends a query. Since late 2024, a second lever has proven just as powerful: spending more compute at inference time, letting a model deliberate before answering, produces reliable capability gains without touching model size at all. This is the story of how "just think longer" became a legitimate scaling law.

🕰️ From Prompting Tricks to a Second Scaling Law

Reasoning models didn't appear out of nowhere — they're the culmination of three years of research into getting language models to "show their work," which eventually got formalized into training objectives rather than remaining a prompting trick.

📊 Why Test-Time Compute, Why Now

Pretraining scaling laws had begun showing diminishing returns by 2024 — each additional order of magnitude of training compute bought a smaller capability gain than the order of magnitude before it, and high-quality training data itself was becoming scarce (a theme we've touched on throughout this series, from the "internet of robot actions" data problem to voice/video training corpora). Test-time compute offered a way around this: instead of a fixed amount of computation per query regardless of difficulty, let the model spend variable compute depending on how hard the problem actually is. A simple factual question gets a fast, cheap answer; a hard proof or multi-step coding problem gets minutes of internal deliberation. OpenAI's o1 scaling charts were the first public evidence that this variable-compute approach scales just as predictably as pretraining compute did — a second lever, not a diminishing-returns patch on the first one.

🧩 A Taxonomy of Test-Time Compute

"Reasoning model" gets used loosely — but the underlying techniques for spending extra inference compute split into genuinely distinct families with different tradeoffs.

By Deliberation Strategy

By Training Approach

🗂️ The Reasoning Model Landscape

⚙️ How Reasoning Models Actually Work

The core mechanism behind every modern reasoning model is reinforcement learning applied to a language model that already has strong base capability from pretraining — RL doesn't teach the model new facts, it teaches the model when and how to deliberate before committing to an answer.

1. What a Reasoning Trace Actually Looks Like

Before producing a final answer, a reasoning model generates a long internal sequence of tokens exploring the problem — often including behavior that looks like backtracking ("wait, that's not right, let me reconsider"), self-verification, and trying alternative approaches. This isn't scripted; it emerges from RL training rewarding whatever reasoning patterns actually lead to correct verifiable answers.

Illustrative reasoning trace — note the thinking tokens vastly outnumber the final-answer tokens

2. Why Verifiable Domains Work Best

Math and code are the domains where reasoning-model RL has worked best, for one crucial reason: the reward signal is objective and automatic. A generated proof either checks out or doesn't; generated code either passes the test suite or doesn't. This means labs can generate essentially unlimited training rollouts without needing expensive human labeling — a self-scaling data source, in sharp contrast to the data-scarcity problems covered in our Voice AI and Physical Intelligence posts. Extending strong reasoning to less verifiable domains (open-ended writing, subjective judgment, ambiguous real-world decisions) remains a harder, less-solved problem — covered further in the Research Frontiers section below.

3. Test-Time Scaling in Practice

The empirical relationship that made o1 a landmark: plotting accuracy against the number of "thinking tokens" a model is allowed to generate produces a smooth, predictable curve — more thinking compute reliably buys more accuracy, following a similar log-linear pattern to classic training scaling laws, at least up to some plateau.

Illustrative accuracy-vs-thinking-tokens curve on a hard benchmark — diminishing but persistent returns

🏢 Major Players

🚀 Startups & Funding Activity

Unlike the model-building layer — which requires frontier-lab-scale RL infrastructure — the reasoning-model wave has created a distinct startup opportunity in inference infrastructure and tooling, since reasoning workloads have fundamentally different serving characteristics (long, bursty, latency-tolerant generations) than traditional chat inference.

Investor thesis patterns: Groq and similar low-latency inference chip companies (also covered in our AI Chips post) are direct beneficiaries of the reasoning-model wave — deterministic, ultra-fast token generation matters even more when a single response involves tens of thousands of thinking tokens. Together AI, Fireworks, and Baseten bet on the "serve open reasoning models efficiently" layer, capturing value from DeepSeek-R1's open-weights disruption without needing frontier-lab-scale RL training budgets themselves. Reflection AI and Prime Intellect represent an emerging category betting specifically on RL-training tooling and infrastructure as its own defensible layer, separate from either model-building or inference-serving.

📐 How Reasoning Is Benchmarked

Reasoning models saturated most older LLM benchmarks quickly, forcing the field toward much harder evaluation sets specifically designed to resist memorization and require genuine multi-step problem-solving.

🔨 How to Build a Reasoning Model From Scratch

Training a frontier-class reasoning model from a base pretrained LLM is now a well-documented recipe, largely thanks to DeepSeek-R1's published methodology — though matching frontier-lab results still requires significant RL infrastructure and compute.

💰 The Economics of Thinking

The connection to our AI Chips post: test-time compute is precisely the "second, independent axis" of compute demand flagged there. It changes hardware requirements in a specific way — reasoning workloads are less sensitive to first-token latency (the user expects to wait while the model thinks) but extremely sensitive to sustained token-generation throughput and cost-per-token at scale, since a single query can generate tens of thousands of thinking tokens before a visible answer appears. This is reshaping which chips win which workloads, with low-latency deterministic inference architectures like Groq's LPU finding a natural fit for reasoning-heavy serving.

🔬 Research Frontiers

🌐 Applications

⚖️ The Verdict: Is Test-Time Scaling a Durable Second Law?