RSI Series · Part 5 of 7 📊

Data vs. Reinforcement Learning

Ask five researchers what's actually driving 2025-2026's reasoning-model gains and you'll get five different weightings of the same two ingredients: more and better data, or reinforcement learning that reshapes how a model uses what it already knows. This piece looks at the actual evidence — a paper showing RL narrows a model's solution distribution rather than expanding it, DeepSeek-R1's own documented account of what pure RL could and couldn't do, and the data-exhaustion timeline that makes this argument matter for how far scaling alone can carry the field.

FrontierAGI Team

September 21, 2026 · 40 min read

Opening The Data Wall What RLVR Actually Is Does RL Teach or Sharpen? DeepSeek-R1's Own Account Quality Over Quantity? Model Collapse Why This Matters for RSI Expert Disagreement The Series So Far Readiness Checklist What's Missing References Videos Verdict

Two Ingredients, One Very Different Story About Each

Every frontier model released since 2024 has been trained with both more/better data and some form of reinforcement learning layered on top. That makes it tempting to treat the two as complementary and move on. But for a series about recursive self-improvement, the split matters a great deal: if progress is mostly a data story, then RSI's leverage point is generating or finding better training data faster — including AI-generated synthetic data, which raises its own risks. If progress is mostly an RL story, the leverage point is designing better reward signals and verification — a very different, and in some ways more controllable, kind of self-improvement loop.

This article doesn't resolve that split, because the evidence doesn't resolve it. What it can do is walk through the actual documented findings on each side — including one specific, fairly striking 2025 paper on what RL demonstrably does and doesn't do to a model's capabilities — so the reader has real data points instead of vibes.

The Data Wall: A Real Timeline, Not a Talking Point

"We're running out of training data" is often said as a throwaway line. Epoch AI's research on this is the closest thing to a rigorous, citable answer. Their analysis estimates the effective stock of human-generated public text at roughly 300 trillion tokens (with a wide 90% confidence interval of 100 trillion to 1,000 trillion, reflecting real uncertainty about what counts as usable text). Under normal, non-overtrained scaling patterns, their projection puts full utilization of that stock somewhere between 2026 and 2032, with a commonly cited conservative estimate around 2028. If labs pursue aggressive overtraining — training far past the compute-optimal point, which several 2024-2025 releases have done deliberately to shrink inference costs — that timeline compresses toward the earlier end of the range.

What "Running Out" Actually Means This isn't a claim that the internet will literally run out of text. It's a claim about running out of new, high-quality, not-already-used-in-training text — the specific resource that made 2018-2023-era scaling laws work as cleanly as they did. Once that specific well is drawn down, further gains have to come from somewhere else: synthetic data, multimodal data, RL, or architectural changes — which is exactly why this question is inseparable from the RL question this article covers next.

What RLVR Actually Is

Reinforcement Learning with Verifiable Rewards (RLVR) is the specific technique behind most of the reasoning-model gains attributed to "RL" in 2024-2026 coverage. Unlike RLHF, where a learned reward model trained on human preference judgments provides the training signal, RLVR uses an automated, deterministic verifier: a math answer is checked against the known correct value, a piece of code is checked by running its unit tests, a proof is checked by a formal verifier. This sidesteps the reward-model imperfections discussed in Part 2 of this series for exactly the domains — math, code, logic puzzles — where a ground-truth checker is available.

This is also precisely why RLVR's gains concentrate so heavily in verifiable domains and much less in open-ended writing, judgment calls, or anything without a checkable answer — there's no equivalent verifier for "is this essay well-argued" the way there is for "does this code pass its tests."

Does RL Teach New Capabilities, or Sharpen Existing Ones?

The single most important piece of evidence for this article comes from a 2025 paper, "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?", which ran a direct comparison most casual RL coverage skips: instead of only measuring pass@1 (does the model get the right answer on its first try), the researchers also measured pass@k for large k — does the correct answer appear anywhere among many sampled attempts.

RLVR-trained models beat their base models at pass@1. But at large k, base models matched or exceeded the RLVR-trained models — the correct reasoning paths RLVR models produce were already present in the base model's own sampling distribution. RL made the model more likely to sample the answer it could already reach; it didn't expand what the model could reach.

The same paper found something that cuts the other way: distillation — training on outputs from a stronger teacher model — did measurably expand the pass@k ceiling, introducing capabilities the student model's base distribution didn't already contain. That's a genuinely important asymmetry: two techniques that both improve benchmark scores, doing so through mechanistically different means, one of which is closer to "concentrating effort on already-latent ability" and the other closer to "actually importing new ability."

Why This Isn't a Reason to Dismiss RL "RL sharpens rather than teaches" doesn't mean RL is unimportant — sharpening is often exactly what's needed. A base model that can occasionally stumble onto a correct 20-step math proof among a thousand attempts is not useful in production; a model that reliably produces that proof on the first try is. RLVR's practical value doesn't depend on it expanding the model's raw capability ceiling — it depends on making the model reliably use the capability it has. But that's a different, more modest claim than "RL is teaching the model to reason in ways it couldn't before," and the distinction matters for how much RL alone can be expected to drive capability gains toward AGI-level reasoning versus gains in reliability at tasks the base model could already sometimes do.

DeepSeek-R1's Own Account of What Pure RL Could and Couldn't Do

DeepSeek's January 2025 technical report (later published in Nature) is unusually candid about this exact tension, because the team ran the experiment directly: DeepSeek-R1-Zero was trained with large-scale reinforcement learning applied directly to the DeepSeek-V3 base model, with no supervised fine-tuning step at all — a deliberate test of how far pure RL could go on its own.

DeepSeek-R1-Zero: The Pure-RL ExperimentJan 2025

What Worked: Reasoning capability — including behaviors like self-verification and reflection — emerged directly from RL without any SFT warm-start, and accuracy on math benchmarks like AIME improved substantially over the base model across training (DeepSeek's own report shows a large, multi-fold improvement in AIME pass@1 over the course of RL training; treat the exact percentage-point figures as approximate pending a direct check against the paper's own tables, since secondary sources cite slightly different numbers for different checkpoints).

What Broke: R1-Zero's outputs suffered from poor readability and language mixing — the model would blend languages within a single chain of reasoning, and its outputs were often unnatural for a human reader to follow, even when the final answer was correct.

DeepSeek's fix, which became the full R1 pipeline, is itself informative: a small "cold-start" set of curated examples (data) before RL begins, rejection sampling and a language-consistency reward during RL, and further SFT and RL stages afterward. In other words, DeepSeek's own published solution to pure RL's failure mode was to add data back in — not to abandon RL, but to combine both ingredients deliberately rather than picking one. That's arguably the most concrete real-world evidence available that "data vs. RL" is the wrong frame for how the labs that actually ship these models think about the problem — it's data and RL, sequenced carefully, with each compensating for the other's weaknesses.

Quality Over Quantity: The Phi Model Series

A separate strand of evidence concerns whether more data is even the right axis to optimize, versus better-curated data. Microsoft Research's "Textbooks Are All You Need" (June 2023) introduced phi-1, a 1.3-billion-parameter code model trained on a deliberately small, highly curated "textbook-quality" dataset — combining filtered web data with synthetically generated textbook-style examples — and reported HumanEval coding performance competitive with much larger models trained on generic web-scraped code. A follow-up, "Textbooks Are All You Need II," extended the same curated-data philosophy to general reasoning with phi-1.5.

⚠️ The Pushback, Stated Fairly Independent researchers subsequently raised a real concern: analysis in "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples" found that some of phi-1's synthetic training data contained near-duplicates of actual HumanEval test problems — close enough in substance to inflate benchmark scores without being caught by simple text-overlap contamination checks. This doesn't mean the "quality over quantity" thesis is false, but it means phi-1's specific benchmark numbers are a weaker piece of evidence for that thesis than they first appeared, and any claim built on citing phi-1's raw scores should carry this caveat.

Model Collapse: The Risk of Training on Your Own Output

If human-generated data is running out and synthetic (AI-generated) data is the substitute, a documented failure mode becomes directly relevant: a 2024 Nature paper by Shumailov, Shumaylov, Zhao, Papernot, Anderson, and Gal showed that recursively training generative models on their own — or their predecessors' — synthetic output causes progressive degeneration, termed model collapse. Both theoretically (using a Gaussian estimation model) and empirically (with language models), the paper demonstrated a narrowing effect: successive generations lose the tails of the original data distribution, drift toward an increasingly narrow mode, and can eventually produce degenerate or repetitive output.

The Important Caveat This finding is specifically about recursive, uncurated, fully-synthetic retraining — each generation trained only on the previous generation's raw output, with no human data retained and no curation applied. Follow-up work examining this framing has argued it may overstate real-world risk, since production pipelines that use synthetic data (including DeepSeek-R1's own rejection-sampling approach above) typically retain human data, apply filtering, and don't naively retrain generation after generation on unfiltered self-output. Model collapse is a real, demonstrated phenomenon under specific conditions — it is not evidence that all synthetic data use is dangerous, and conflating the two overstates the finding.

Why This Matters Specifically for Recursive Self-Improvement

An AI system that's automating its own research — the subject of Part 4 — has to make exactly this data-vs-RL tradeoff at machine speed and, potentially, with less human oversight of the choice. If the "RL sharpens rather than teaches" finding generalizes, an RSI loop that only relies on RL against its own outputs may hit a ceiling defined by what's already latent in its base model — real acceleration of research throughput, but not necessarily an escape from the base model's fundamental capability boundary. Genuinely expanding that boundary, per the same paper's finding about distillation, would require injecting new signal from outside the model's own distribution — meaning a purely self-referential RSI loop, one that only trains a model on evaluations of its own outputs without new external data or a stronger outside teacher, has a specific, evidenced reason to plateau rather than compound indefinitely.

Model collapse adds a second, related concern: if that same self-referential loop generates its own training data for the next round without careful curation, it inherits the exact conditions under which model collapse has been demonstrated. This doesn't predict that any real RSI system will collapse — it identifies a specific, named failure mode that any system relying on self-generated data needs to actively design against, rather than a hypothetical risk with no documented precedent.

Expert Disagreement

The "sharpening vs. teaching" paper cited in this article is a single, if well-designed, 2025 study — not a settled consensus finding replicated across many independent labs. Some researchers argue that even "sharpening" existing capability is functionally equivalent to teaching new capability once a system is composed into longer agentic chains, since reliably executing a 50-step correct reasoning path that was only occasionally sampled before is a real capability gain in any practical sense, whatever the underlying mechanism. Others maintain the mechanistic distinction matters a great deal for predicting whether RL alone can drive continued frontier progress once data growth slows. No formal survey of researcher opinion on this specific question was located; this section reflects real, visible disagreement in papers and commentary, not polled consensus.

The Series So Far

Four Parts Published, Two to Go

Part 1 · Published

The Primer

Where RSI comes from, what's real today, and why every major lab's safety framework now names it as a tracked risk category.

Part 2 · Published

The Objective-Specification Problem

Why you can't write down what you actually want, and why that gap compounds once AI starts writing objectives for the next AI.

Part 3 · Published

Distillation, Centralization, and the Moat Question

Whether a capability lead survives contact with distillation — the OpenAI-DeepSeek accusation and Epoch AI's real catch-up-time data.

Part 4 · Published

AI Automating AI Research

Real September 2026 systems (Dream-RSI, ModularRSI, ScienceBuddy) that speed up AI research today, plus a fabricated claim caught and corrected.

Readiness Checklist

Can you explain the difference between pass@1 and pass@k, and why that distinction is central to the "does RL teach or sharpen" question?

Can you explain what RLVR is and why its gains concentrate in math and code rather than open-ended writing?

Can you describe what DeepSeek-R1-Zero's pure-RL experiment showed, and what specific problem the full R1 pipeline was built to fix?

Can you state the model-collapse finding precisely enough to know it does NOT mean "all synthetic data is dangerous"?

⚠️ What's Missing or Uncertain

Several figures in this article carry specific caveats worth restating. Epoch AI's data-exhaustion timeline (2026-2032) is itself a projection with a wide confidence interval, not a precise date — treat any single-year citation of it elsewhere as a simplification of a genuinely uncertain range. The exact AIME pass@1 percentage-point figures for DeepSeek-R1-Zero vary slightly across secondary sources reporting on the same technical report; this article describes the direction and scale of the improvement without asserting a single precise number pending a direct primary-source check. The "RL sharpens rather than teaches" finding comes from one 2025 paper, not a replicated multi-lab consensus, and the Expert Disagreement section above should be read as a real limitation on how far this article's central claim can be generalized. The exact origin/coinage of the term "RLVR" could not be confirmed — it is real and in wide use, but this article makes no claim about who first named it.

🔗 Reference Links

🎥 Recommended Videos

But What Is a Neural Network?

3Blue1Brown — foundational context for how training data shapes a model's learned distribution

🧭 Closing — It Was Never Actually a Contest

🎯 The Bottom Line

The strongest evidence in this article — DeepSeek's own account of fixing R1-Zero's failures by adding curated data back into an RL pipeline — undercuts the premise of "data vs. RL" as a real contest with a winner. Data supplies what a model can potentially know and do; RL, at least in its current RLVR form, appears to concentrate a model's effort on reliably producing what it can already sometimes do, per the sharpening-versus-teaching evidence covered above. Neither one substitutes for the other, and every documented frontier pipeline this article examined uses both, sequenced deliberately rather than chosen between. The genuinely open question for RSI is not which ingredient wins — it's whether a self-improving system can keep supplying itself with the kind of genuinely new signal that data (or a stronger external teacher) provides, once the easy human-generated sources run low, without falling into the recursive-collapse failure mode this article also documented.