Course Guide · Standalone Handbook 📘

Self-Improving AI Agents: A Field Handbook

A standalone handbook distilling Stanford's CS329A, "Self-Improving AI Agents" — the course's framing, the case for why verification (not generation) is the real bottleneck, test-time compute, agentic feedback loops, planning as search, training models to reproduce what search discovers, evaluating agents that take hundreds of steps, and where the field still has open problems — organized as one reference rather than a lecture-by-lecture sequence.

FrontierAGI Team

September 23, 2026 · 32 min read

On sourcing: this handbook was written from a detailed independent synthesis of the course's material, cross-referenced against third-party course-note mirrors on GitHub where topics overlapped (YouTube itself was unreachable in this research environment). It is our own original explanation of these concepts, not a reproduction of any source document. Where a technique doesn't have a specific paper link included below, that's deliberate — this pass didn't independently verify a specific arXiv ID or repository for every name mentioned, and we'd rather name a technique without a citation than attach one we haven't checked. Treat names like LATS, SPRINT, SWiRL, Search-O1, Search-R1, GDPval, and DeepScholar-Bench as pointers to look up yourself, not confirmed bibliographic entries.

Course Framing Test-Time Compute & Verification Agentic Feedback Loops Planning as Search Training on What Search Finds Case Study: Competitive Programming Retrieval & Research Agents Evaluating Long-Horizon Agents What's Still Missing Putting It Together Confidence Notes & Gaps Full Reference List

Course Framing

CS329A organizes the last several years of AI progress into four phases: scale (bigger models, more data, more compute — improvement encoded directly in the weights), post-training (RLHF and instruction tuning, teaching an already-trained model to actually follow instructions and align with a reward signal), test-time compute ("let it think longer" — spending more computation at inference instead of only improving the model itself), and agentic loops (goal → plan → act → observe → self-correct, wrapping a model in a loop where it takes actions and adjusts rather than producing one answer and stopping).

Read as a whole, those four phases describe improvement moving outward — from the model's weights, into how the model is used at inference, and then into the scaffolding built around it. The course's central organizing claim ties all of it together: once a system can generate many candidate solutions, the bottleneck is no longer generation quality. It's whether you can reliably tell which candidate is actually correct. A system that produces brilliant answers alongside plausible-looking wrong ones, with no way to distinguish them, gets no real benefit from generating more.

The pattern that recurs at every scale in this handbook: generate something, check it, keep what works, and eventually train the system to produce the good version directly — applied first to single answers, then to multi-step plans, then to entire training runs, and (in the field's genuinely unsolved cases) to the verifiers themselves.

Test-Time Compute & Verification

The first concrete application of the course's central claim is the simplest one: hold a model's weights fixed and spend more compute at inference time instead. Sample a problem many times, and the chance that at least one attempt is correct climbs — sometimes dramatically, since generating more candidates raises the ceiling of what's reachable even when any single attempt is unreliable. That's a coverage curve, not an accuracy curve: it tells you the right answer is somewhere in your samples, not that you can find it. Inference compute can be spent two different ways — parallel (generate many independent candidates and pick the best, i.e. best-of-n) or sequential (generate one answer, then critique and revise it repeatedly) — and which is more efficient depends on how hard the specific problem is, which is why some frameworks (Archon among them) treat the choice of inference architecture itself as something to search over automatically, rather than fix in advance.

None of that generation advantage matters without a way to select the right candidate, which is where verification comes in. The field's verifiers have followed a clear progression: outcome reward models judge only the final answer, with no credit for the reasoning that got there — a chain of reasoning can be entirely wrong and still get full credit for stumbling onto the right number, or entirely sound and get zero credit for one arithmetic slip at the end. Process reward models fix this by scoring each step of a reasoning chain individually, catching a bad step even when the final answer happens to be right — work like "Let's Verify Step by Step" and its PRM800K dataset established this approach using large-scale human step-level annotation, and later work (Math-Shepherd among it) explored deriving that same kind of step-level signal automatically instead of requiring expensive human annotation for every example. The most recent step in this progression, sometimes called Weaver, fuses many weaker, cheaper verification signals into one stronger judgment rather than relying on one large expensive verifier — the same "cheap-and-many vs. expensive-and-few" tradeoff that shows up in generation, applied to verifiers themselves.

Why this matters beyond single answers: a verifier that only judges final answers can't tell a multi-step agent where it went wrong — only whether to keep or discard an entire attempt. A step-level verifier can. That distinction is the difference between a system that can only get better at picking good full attempts, and one that can actually learn from and correct its own partial mistakes — a prerequisite for everything in the rest of this handbook.

Agentic Feedback Loops

A language model that only produces text in one shot can't correct a mistake it doesn't yet know it's making. The systems covered in this section give it a way to find out.

ReAct — reasoning interleaved with acting. Instead of a model reasoning entirely inside its own head and then producing a final answer, ReAct-style agents alternate between a reasoning step and an action step — calling a search tool, querying a database, running a calculation — then folding the result of that action back into the next round of reasoning. The effect is that reasoning stays grounded in real, checkable information instead of drifting into confident-sounding guesses.

RLEF — reinforcement learning from execution feedback. Code is unusually well-suited to this kind of loop because its correctness can often be checked automatically: generate code, run it, see whether it passes tests, and use that pass/fail (or partial-credit) signal as a training reward. Because execution feedback is objective and doesn't require a human in the loop, it scales far more easily than most other reward sources — one reason coding has become one of the most productive domains for this style of training.

Constitutional AI — feedback without an executable check. Not everything can be graded by running code. Constitutional-AI-style methods instead give the model a written set of principles, have it critique its own response against those principles, revise accordingly, and use that critique-and-revise process as a training signal. It's a weaker, less objective feedback source than execution — how well it works depends heavily on how well the model actually applies the stated principles — but it extends the same generate-check-improve pattern to behaviors that can't be unit-tested.

Want the expanded version? Agentic Feedback Loops in Practice (Deep Dive 1 of 5) covers all three techniques with verified paper citations and an original diagram for each loop.

Planning as Search

A single chain of reasoning commits to one line of thought from the start. Harder, longer-horizon tasks often benefit from considering several possible paths before committing to one — which turns planning into a search problem over possible futures, not just a longer version of ordinary reasoning.

Tree search over reasoning and actions. Rather than a single reasoning chain, the agent builds a tree: each node is a state or partial plan, each branch is an alternative next step, and the system evaluates branches to decide where to spend more computation — closer to how game-playing search algorithms explore a move tree than how a chatbot answers a question. This buys the ability to backtrack out of an early mistake instead of being stuck with it, at the real cost of much higher inference compute.

The irreversible-action problem. Tree search assumes exploring a bad branch is cheap to abandon. That assumption breaks the moment an agent's actions have real consequences — sending a message, spending money, deleting a file, modifying a production system. A reliable agent architecture needs a clear boundary between planning (where mistakes are recoverable) and commitment (where they aren't), enforced through mechanisms like simulating an action before taking it, requiring explicit approval for high-stakes steps, or setting a much higher confidence bar before anything irreversible is allowed to execute.

Parallelizing reasoning. Ordinary chain-of-thought reasoning is sequential by construction — each step depends on the last, which adds latency. Some more recent approaches instead decompose a problem into genuinely independent sub-plans that can be reasoned about simultaneously and then combined, trading the difficulty of finding a valid decomposition for the ability to use parallel compute instead of paying for depth serially.

Learning tool use from synthetic trajectories. Training an agent to use tools well normally means letting it interact with real, live environments during training — slow, expensive, and sometimes unsafe. An alternative is generating synthetic multi-step tool-use trajectories offline, filtering them for quality, and training on those instead of on live rollouts. The open question with this approach is always the same one any simulation-to-reality method faces: does a policy trained on synthetic trajectories actually generalize to environments it never saw during training?

Want the expanded version? Planning as Search (Deep Dive 2 of 5) covers LATS, SPRINT, and SWiRL with verified citations, original diagrams, and a closer look at the irreversible-action problem.

Training the Model to Reproduce What Search Finds

Search and sampling can discover a correct answer that was already somewhere in the model's distribution. Training is how that discovery gets turned into something the model produces reliably on the first try, without needing to search for it again.

The bootstrapping pattern. One influential family of methods (in the spirit of what's often called self-taught reasoning) works like this: when a model is given a question and a correct final answer but no worked explanation, have it generate its own reasoning trace toward that answer; keep the traces that actually reach the right answer; fine-tune on those. The model is, in effect, teaching itself to produce good rationales using only answer-level supervision — with the obvious caveat that a reasoning trace which happens to land on the right final answer isn't guaranteed to be reasoning correctly along the way, which is exactly the outcome-vs-process distinction covered above for verifiers.

Why full policy-gradient RL got expensive. Classic reinforcement-learning setups for LLM alignment (Proximal Policy Optimization being the best-known) typically need a policy model, a separate reference model, a reward model, and often a value/critic model running simultaneously — a lot of memory and compute overhead once the base model itself is already large.

Group-relative optimization as a cheaper alternative. A more recent line of work (associated with DeepSeek's math-focused RL training) removes the separate critic model entirely. Instead of learning an absolute value estimate, the model generates a group of candidate responses to the same prompt, scores them, and reinforces responses that scored above the group's own average while discouraging ones that scored below it — a purely relative comparison within each batch of samples, not a learned external value function.

The zero-variance problem. Group-relative methods run into an obvious failure case: if every sampled response in a batch is wrong, or every one is right, there's no meaningful relative signal to learn from. Training efficiency ends up depending heavily on how many prompts actually produce a mix of successes and failures across the sampled group — which is itself a reason difficulty-aware curricula and sampling strategies matter.

Stabilizing RL over long reasoning chains. As reasoning traces get longer, a few specific problems show up repeatedly in this line of research: asymmetric treatment of how much a policy is allowed to shift depending on the direction of the update; the risk that low-signal prompts (the zero-variance case above) waste compute unless they're filtered or resampled; length imbalances between short and long responses distorting the loss; and entropy collapse, where the model's output distribution narrows onto a small set of "safe" successful behaviors at the cost of the diversity needed to discover anything new later.

A subtlety worth sitting with: RL applied this way often improves how reliably a model produces a strategy it could already sometimes produce, more than it discovers a genuinely new capability. If a model solves a hard problem 15% of the time before training and 75% of the time after, that's a real and valuable improvement — but it may mean the training mostly made an existing good strategy more probable, rather than expanding what the model is fundamentally capable of. Whether self-improvement loops can reliably push outward on capability, not just reliability, remains one of the more contested open questions in this literature.

Want the expanded version? Training on What Search Finds (Deep Dive 3 of 5) covers STaR, GRPO, and DAPO with verified citations and original diagrams for each.

Case Study: Competitive Programming as a Search Problem

Competitive programming is a useful case study because it makes the generate-verify-select loop concrete and because two generations of the same underlying system illustrate the shift toward learned selection. Early approaches to this problem leaned heavily on brute-force generation: produce a very large number of candidate programs for a given problem, filter out ones that are obviously invalid, cluster the survivors to avoid submitting many near-duplicate solutions, and only then submit a small diverse set. Diversity management — making sure your limited number of submission attempts aren't all the same idea in different clothing — turns out to matter almost as much as raw generation volume.

A later generation of the same approach shifted weight away from brute-force sampling and toward stronger generation models paired with a learned scoring/ranking step: generate, score, rank, select — spending relatively more effort on picking the right candidate out of a smaller, higher-quality set instead of relying on overwhelming numbers of low-quality attempts. It's the same lesson from the verification section above showing up again: as generation gets better and cheaper, the system's success increasingly hinges on the quality of selection, not the quantity of candidates.

Retrieval and Research Agents

The limits of retrieve-everything-up-front. Standard retrieval-augmented generation fetches documents once, stuffs them into context, and generates an answer. This is simple but indiscriminate — a context window can fill up with material that's only weakly related to the actual reasoning problem, diluting the genuinely useful evidence.

Conditional, in-the-loop search. A more recent pattern treats search as an action the agent takes only when it notices a gap in its own knowledge mid-reasoning, rather than a fixed first step — reason, detect uncertainty, search specifically for what's missing, extract the relevant evidence, and continue. This keeps retrieval targeted to what's actually needed instead of front-loading everything that might be relevant.

Learning when searching is worth it. Beyond prompting a model to search whenever it feels uncertain, some approaches train the search-or-not decision itself via reinforcement learning — implicitly weighing the expected value of new information against the real cost of getting it (latency, compute, the risk of retrieving something misleading). Search stops being a fixed pipeline stage and becomes a resource-allocation decision the agent makes repeatedly.

Retrieval doesn't end the problem — it creates a new one. Fetching a whole document doesn't guarantee anything useful was actually found in it. A capable research agent still has to separate relevant passages from irrelevant ones, distinguish real evidence from speculation, and tell primary sources from secondary summaries — a reasoning problem layered on top of the retrieval problem, not a substitute for it.

Want the expanded version? Competitive Programming and Retrieval Agents (Deep Dive 4 of 5) covers AlphaCode's two generations plus Search-O1 and Search-R1 with verified citations and original diagrams.

Evaluating Agents That Take Hundreds of Steps

Traditional benchmarks — short questions, fixed correct answers, one isolated reasoning problem at a time — don't capture what matters about an agent that has to complete a task made of dozens or hundreds of sequential decisions, using tools, under ambiguous instructions, with no single "check the final answer" moment.

Measuring capability as a time horizon (the METR approach). Rather than asking what fraction of fixed questions a model gets right, this approach asks a different question: how long a duration of real human professional work can an agent reliably complete end to end? Reframing capability as a task-length horizon exposes something benchmark-accuracy scores hide — that small per-step error rates compound multiplicatively across a long task. A 98%-per-step success rate sounds excellent, but across 100 sequential critical steps, the chance of completing the whole task successfully drops to roughly 13% — which is why local competence on short tasks doesn't guarantee reliability on long ones.

Reliability is its own capability, separate from "can it ever succeed." An agent that completes a hard task 20% of the time and one that completes it 95% of the time both demonstrate the task is within reach — but only the second is something you could actually delegate to. Evaluation needs to distinguish "this is within the model's capability frontier" from "I can safely hand this over," which are very different practical claims.

Evaluating economic usefulness directly (GDPval). Rather than academic benchmark accuracy, this style of evaluation measures performance on real professional tasks spanning many occupations and sectors — work that comes with ambiguous instructions, hidden assumptions, domain-specific conventions, and communication requirements that clean, single-answer benchmarks tend to strip away.

Evaluating research synthesis specifically (DeepScholar-style benchmarks). A literature-review or research-synthesis agent has to search, retrieve, read, extract, compare, synthesize, cite, and write — with failure possible at every one of those stages, and no simple unit-test equivalent the way code has. Common failure modes named in this space include missing important sources, retrieving low-quality ones, misreading evidence, incomplete coverage of relevant facts, unsupported claims, and outright wrong citations.

Across long-horizon agents generally, a recognizable set of failure modes keeps showing up regardless of the specific domain: poor upfront planning, choosing the wrong tool despite having the right one available, declaring a task finished prematurely, getting stuck in repetitive unproductive loops, gradually losing track of the original requirements (context drift), a small early mistake contaminating everything downstream (cascading errors), a verifier that wrongly accepts a bad intermediate result, and simply continuing to spend compute well past the point of diminishing returns. That last one points to a genuinely underrated design problem: a capable agent needs a sensible stopping policy — not just knowing how to think, but knowing when continuing to think stops being worth the cost.

Want the expanded version? Evaluation and the Open Problems (Deep Dive 5 of 5, the final part in this series) covers METR, GDPval, and DeepScholar-Bench with verified citations, plus a closer look at the three problems below.

What's Still Missing

Meta-verification. Nearly every system covered so far depends on an external reference to check against — a known answer, a passing test, a human label. A more autonomous system needs some way to judge reasoning quality even when no reference answer exists at all: is this internally coherent, does the cited evidence actually support the conclusion, can an independent model reproduce the result, is the verifier itself trustworthy. Pushed far enough, this creates a verifier-of-the-verifier regress that nobody has fully solved — building a useful hierarchy without an infinite chain of evaluators checking evaluators remains open.

Self-generated tasks and curricula. If humans have to keep supplying every new training problem by hand, improvement is bottlenecked on human task-writing capacity. A more autonomous loop would have the system generate its own next task, attempt it, verify the outcome, train on what worked, and generate a harder task next — with the real difficulty being keeping generated tasks in the productive zone: hard enough to teach something, not so hard that every attempt fails and there's no learning signal at all.

Continual learning and memory. Most deployed systems still follow a rigid train-then-freeze-then-deploy cycle. A system that kept learning from its own deployment experience could adapt to users, pick up new tools, and incorporate new information over time — but this opens a long list of hard problems: catastrophic forgetting, unstable updates from noisy or adversarial feedback, model drift, and the practical question of what a long-lived agent should actually remember versus discard, across working memory, episodic memory of past experiences, generalized semantic knowledge, and learned procedural strategies.

The economics and infrastructure of spending more compute. If future systems routinely spend large amounts of inference compute per task — many parallel branches, search trees, multiple verifiers, tool execution — that's a genuinely different computational profile than today's serving infrastructure was built for, which optimizes for low-latency single responses. There's also a raw efficiency question underneath all of this: capability-per-unit-of-energy, not just capability in isolation, since a system that's extremely capable but extremely energy-hungry doesn't scale to widespread use. Both point toward routing — sending easy tasks to smaller/local/cheaper models and only escalating genuinely hard ones to the most expensive available compute — becoming a real design problem in its own right, not just an implementation detail.

Putting It Together

Strip away the individual technique names and nearly everything in this handbook is one loop applied at different scales: generate candidates, check them against some form of verification, keep what's good, and eventually train the system to produce the good version directly next time — repeated for single answers, for multi-step plans, for entire training runs, and (in the still-unsolved cases above) for the verifiers themselves.

Layer	What gets generated	What checks it	How the win gets kept
Single answer	Sampled candidate responses	Outcome/process reward models, ensembles (see above)	Best-of-n selection, or distillation into a cheaper model
Multi-step plan	Branches in a reasoning/action tree	Step-level scoring, execution feedback, tool results	Search selects the branch; irreversible steps gated separately
Model weights	Reasoning traces / rollouts from the current model	Verifiers, execution results, reward signals	Fine-tuning or RL updates on the successful traces
The verifier itself	A judgment about correctness or quality	Largely unsolved — meta-verification, above	Open research problem

If there's one idea to take away from this whole handbook: the scarce resource in self-improving AI isn't generation, and increasingly isn't even compute — it's trustworthy verification. Every layer in the table above works fine as long as the "checked by" column is reliable, and breaks down in exactly the way you'd expect when it isn't: an agent that games a weak verifier, a training run that reinforces a lucky-but-wrong reasoning trace, a research agent that confidently synthesizes claims its sources don't actually support. That's the same conclusion the practitioner's primer reached from the paper-literature side (self-improvement strength tracks verifier quality) — this course arrives at it independently from the systems-and-training side.

🎯 The Bottom Line

CS329A isn't really nine separate lectures — it's one argument told nine times from different angles. Generation is cheap and getting cheaper; verification is expensive and stays the bottleneck, whether that verification is a reward model checking an answer, a test suite checking code, an AI judging a critique, or a tree search deciding which branch to keep exploring. Every technique in this handbook is a different answer to "how do I get a trustworthy check on this specific kind of output," and every open problem in the "What's Still Missing" section is a case where nobody has a good answer yet — most pointedly, checking the checker itself. If you take one framework out of this course, it's that: before building the agent, design what tells you it's actually working.

⚠️ Confidence Notes and Gaps

This handbook explains concepts and names techniques at a conceptual level. The verification section above (outcome vs. process reward models, PRM800K, Math-Shepherd, Weaver) carries specific cited work; most of the later material (planning/search methods, the RL techniques, AlphaCode, the retrieval methods, GDPval, and DeepScholar-style benchmarks) does not carry specific performance figures or paper citations, because those weren't independently re-verified against primary sources in this pass. If you plan to cite a specific claim, technique name, or number from this piece in your own work, look up the primary paper or benchmark directly rather than relying on this summary. The overall shape of the course's later material — agentic loops, planning-as-search, RL for reasoning, competitive programming as a case study, retrieval agents, and long-horizon evaluation — is presented with reasonable confidence; specific figures and exact paper attributions are not.

🔗 Full Reference List

A Practitioner's Primer on RSI (independent convergent conclusion on verifier quality) CS329A YouTube playlist (not independently fetchable in this research environment)