Deep Dive · 3 of 5
Training on What Search Finds
Search can discover a correct answer that was already somewhere in a model's distribution. Training is how that discovery becomes something the model produces reliably on the first try. This deep dive expands the field handbook's coverage of STaR, GRPO, and DAPO with verified citations and original diagrams for each.
On the citations below: STaR's identity is confirmed against the official NeurIPS 2022 proceedings page — high confidence, including a genuine conference-talk recording. GRPO is introduced inside the DeepSeekMath paper rather than as a standalone paper — its arXiv ID and core mechanism are high confidence, but its complete author roster wasn't independently verified byte-for-byte. DAPO's core identity and reported results are high confidence via ByteDance Seed's own publications page; its full ~30-person author list and its exact peer-review status weren't independently confirmed. arXiv itself wasn't directly browsable in this research environment for any of the three — treat specific numbers as accurately transcribed from secondary sources, not independently recomputed.
From Search to Weights
STaR
GRPO
The Zero-Variance Problem
DAPO
Comparing the Three
Confidence Notes & Gaps
Full Reference List
From Search to Weights
Everything in Deep Dives 1 and 2 spends extra computation at inference time — sampling more, searching a tree, running parallel branches — to get a better answer to one specific problem. None of that computation carries over to the next problem unless something captures what worked and bakes it into the model's weights. The three techniques below are three different ways of doing that capture, in increasing order of sophistication: bootstrapping training data from the model's own successful attempts, comparing sampled responses against each other instead of an external value function, and stabilizing that comparison-based training over very long reasoning chains.
STaR: Bootstrapping Reasoning With Reasoning
STaR: Bootstrapping Reasoning With Reasoning HIGH
Bootstraps a model's reasoning ability without a large hand-labeled rationale dataset. The loop: generate chain-of-thought rationales for many questions via few-shot prompting; keep only the rationales whose final answer is correct; for questions the model got wrong, use "rationalization" — show the model the correct answer and have it construct a rationale that leads there; fine-tune on the pooled correct-and-rationalized examples; repeat with the improved model. This lets a model improve iteratively on reasoning tasks using only its own generations as training data, evaluated on tasks spanning arithmetic/symbolic reasoning and commonsense QA.
STaR's loop: correct rationales are kept directly; incorrect attempts get a second chance via rationalization before the whole pool is used to fine-tune the next iteration.
GRPO: Comparing Within a Group Instead of Learning a Critic
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models HIGH identity full author list unconfirmed
Introduces Group Relative Policy Optimization (GRPO) as its RL algorithm — a critic-free variant of PPO. For each prompt, a group of responses is sampled and each scored by a reward model; the advantage used to update the policy is computed relative to that group's own mean and standard deviation, rather than from a separately learned value/critic model, cutting memory and compute versus standard PPO. DeepSeekMath 7B, continued-pretrained on roughly 120B math-related tokens, reportedly reached 51.7% on the MATH benchmark without external tools or majority voting — approaching the level of much larger closed models on that benchmark at the time.
📄 arXiv:2402.03300
📺 No official author-produced video confirmed; third-party explainers exist (e.g. AI Papers Academy's "GRPO Reinforcement Learning Explained")
GRPO scores a group of sampled responses against each other, not against a learned value model — the advantage is purely relative standing within the batch.
The Zero-Variance Problem
Group-relative methods like GRPO run into an obvious failure case: if every response in a sampled group is wrong, or every one is right, there's no meaningful relative signal — reinforcing "above the group average" is meaningless when the whole group scored the same. Training efficiency then depends heavily on how many prompts actually produce a mix of successes and failures across the sampled group, which is part of what motivates the dynamic sampling technique inside DAPO below: rather than wasting compute on prompts whose group happens to be uniform, filter them out and resample prompts more likely to produce a useful gradient.
DAPO: Stabilizing RL Over Long Reasoning Chains
DAPO: An Open-Source LLM Reinforcement Learning System at Scale HIGH
Extends the GRPO-style approach with four techniques aimed specifically at long-chain-of-thought RL stability: Clip-Higher (decoupled, asymmetric upper/lower clipping ranges that preserve exploration and resist entropy collapse — where the model's outputs narrow onto a small set of "safe" behaviors); Dynamic Sampling (filtering out prompts whose sampled group is entirely correct or entirely incorrect, directly addressing the zero-variance problem above); Token-Level Policy Gradient Loss (argued to matter specifically in long-CoT settings, where response-level loss averaging can distort learning); and Overlong Reward Shaping (reducing reward noise introduced by responses that get truncated for running too long). Using a Qwen2.5-32B base model, DAPO reportedly reaches 50 points on AIME 2024, said to outperform a previously reported DeepSeek-R1-Zero-Qwen-32B result using roughly half the training steps. The team fully open-sourced their training code (built on the verl RL framework) and curated dataset.
DAPO's four techniques each target a different failure mode of naive group-relative RL once reasoning chains get long.
Comparing the Three
| Technique | What it captures into weights | Core mechanism | Where it fits |
|---|---|---|---|
| STaR | Self-generated rationales that reach correct answers | Filter-then-fine-tune, plus rationalization for near-misses | Bootstrapping reasoning ability with no large labeled rationale dataset |
| GRPO | Which sampled responses beat their own group's average | Critic-free RL, group-relative advantage estimation | Efficient RL for reasoning without a separate value model |
| DAPO | Same as GRPO, made stable over much longer chains | Decoupled clipping, dynamic sampling, token-level loss, reward shaping | Long-chain-of-thought RL at scale without entropy collapse |
🎯 The Bottom Line
These three sit on one continuous line, not three unrelated options. STaR shows that a model can generate its own training data if you can verify correctness; GRPO shows you can turn that same self-generated-and-verified data into an RL signal without a separate critic model; DAPO shows what breaks when you push that RL signal over reasoning chains long enough for entropy collapse and zero-variance batches to actually matter, and fixes each failure mode individually. None of the three tells you anything about whether the model actually got more capable versus just more reliable at reproducing strategies it could already sometimes find — that distinction, raised in the field handbook, is a separate question these training methods don't resolve on their own.
Continuing the deep dives? Deep Dive 4 — Competitive Programming and Retrieval Agents covers AlphaCode's two generations and two techniques for making search itself part of the reasoning loop.
⚠️ Confidence Notes and Gaps
STaR's identity and its NeurIPS 2022 talk recording are confirmed against official sources — high confidence. GRPO's core mechanism and arXiv ID are high confidence since it's a well-documented part of the widely-discussed DeepSeekMath paper, but its complete author roster wasn't independently verified byte-for-byte against the arXiv abstract page in this research pass. DAPO's identity, reported AIME 2024 score, and open-source release are high confidence via ByteDance Seed's own publications page, but its full ~30-person author list and exact peer-review status (versus simply being publicized alongside ICLR 2025 activity) were not independently confirmed. None of the three papers were directly fetched from arxiv.org in this research environment — verify specific figures directly before citing them formally.