Deep Dive · 1 of 5
Agentic Feedback Loops in Practice
The field handbook's "Agentic Feedback Loops" section introduced ReAct, RLEF, and Constitutional AI in a few paragraphs each. This is the expanded version — the real papers, verified citations, and an original diagram for each loop so you can see exactly where information flows and where the feedback signal actually comes from.
On the citations below: all three papers' titles, authors, arXiv IDs, and venues were independently verified through direct research rather than carried over from any single source. Two small caveats are flagged where they apply: RLEF's exact co-author ordering and Constitutional AI's full ~50-person author list could not be byte-for-byte confirmed against the primary arXiv page in this research pass (arxiv.org itself was not directly browsable), though the paper identity, arXiv ID, and core claims are high confidence, cross-corroborated across multiple independent sources. Video links are only included where a genuine, specifically-identifiable recording was found — no fabricated links.
Why These Three Together
ReAct
RLEF
Constitutional AI
Comparing the Three Loops
Confidence Notes & Gaps
Full Reference List
Why These Three Together
All three techniques answer the same underlying question — how does a language model get feedback on something more complex than "is this single answer right or wrong" — but they source that feedback from three genuinely different places: ReAct grounds reasoning in real-world tool results, RLEF grounds it in code execution, and Constitutional AI grounds it in the model's own critique against a written set of principles, with an AI (not a human) judging the result. Seeing the three loops side by side makes clear that "agentic feedback" isn't one mechanism — it's a design choice about where the ground truth for "good" comes from.
ReAct: Reasoning Grounded in Tool Results
ReAct: Synergizing Reasoning and Acting in Language Models HIGH
Proposes prompting an LLM to interleave chain-of-thought reasoning with discrete actions — API or tool calls to external sources — and the observations those actions return, rather than doing pure reasoning or pure acting alone. On HotpotQA and FEVER, grounding reasoning in retrieved facts this way reduces hallucination and error propagation compared to reasoning with no external check. On interactive decision-making benchmarks (ALFWorld, WebShop), it substantially outperforms imitation- and RL-trained baselines using only one or two in-context examples.
The ReAct loop: each action's real-world observation feeds the next thought, instead of reasoning happening in one unchecked pass.
RLEF: Reinforcement Learning from Execution Feedback
RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning HIGH
Trains code-generating LLMs end-to-end with reinforcement learning (PPO) to actually use execution feedback: the model writes code, that code runs against public test cases, and any failure output is fed back into the conversation for the model to repair — with the reward based on held-out private tests the model never sees directly. This teaches genuine iterative self-correction, not just better first-try accuracy, and the paper reports state-of-the-art results on competitive programming benchmarks with both 8B and 70B models, using an order of magnitude fewer samples than prior approaches to reach comparable performance.
📄 arXiv:2410.02089
📺 ICML 2025 poster session exists; no specific named recorded talk independently confirmed
The RLEF loop: execution results, not a learned judge, decide the reward — the same objective feedback pattern the practitioner's primer highlighted as a domain advantage for code.
Constitutional AI: Feedback from a Written Constitution, Judged by AI
Constitutional AI: Harmlessness from AI Feedback HIGH
Trains a helpful-and-harmless assistant using no human harm labels at all. In a supervised phase, the model critiques and revises its own outputs against a written set of principles (the "constitution"). In an RL phase — this is the part often shorthanded RLAIF, reinforcement learning from AI feedback — an AI model, not human raters, judges pairs of outputs to build a preference dataset, and the assistant is trained against that AI-generated preference model. The paper reports the resulting model was preferred by human crowdworkers over prior RLHF-only harmlessness training, while being less evasive and more willing to actually engage with a question.
📄 arXiv:2212.08073
📺 No dedicated technical-talk recording of this specific paper independently confirmed
Constitutional AI's two phases: a supervised critique-and-revise loop, then an RL phase where an AI — not a human — supplies the preference signal.
Comparing the Three Loops
| Technique | Where feedback comes from | How objective is it | Best-suited domain |
|---|---|---|---|
| ReAct | Real tool/API results returned mid-reasoning | High — the observation is a real external fact | Question answering, retrieval-grounded tasks, interactive environments |
| RLEF | Code execution against test cases | Very high — tests pass or fail, no judgment call | Code generation and repair specifically |
| Constitutional AI | Written principles + an AI judge's preference | Lower — depends on how well the model applies the constitution | Behaviors that can't be unit-tested (tone, harmlessness, style) |
The pattern underneath all three: the more objectively "correct" the feedback source is, the less the loop depends on trusting a judge — ReAct and RLEF both ground themselves in something the world (or a test suite) directly confirms, while Constitutional AI's RL phase depends on the AI judge itself being reliable, which is exactly the "who verifies the verifier" problem the field handbook's meta-verification section flags as still largely unsolved.
🎯 The Bottom Line
If you're choosing which of these three to build with first, choose based on how objective your feedback source can be, not on which technique sounds most sophisticated. RLEF is the strongest default wherever code execution is available — the reward is unambiguous and doesn't require trusting a judge. ReAct is the right pattern the moment a task needs real-world facts a model can't reliably know from training alone. Constitutional AI is the fallback for everything else — behaviors that can't be unit-tested — but it inherits a weakness the other two mostly avoid: its RL phase's entire signal depends on the AI judge being trustworthy, which is exactly the meta-verification problem the field still hasn't solved. Pick the loop whose ground truth you'd actually bet on.
Continuing the deep dives? Deep Dive 2 — Planning as Search covers LATS's tree search, SPRINT's parallel reasoning, SWiRL's synthetic-trajectory training, and the irreversible-action problem tree search quietly assumes away.
⚠️ Confidence Notes and Gaps
All three papers' core identity (title, arXiv ID, venue, central claims) are high confidence, cross-corroborated across multiple independent sources during research for this piece. Two narrower details are flagged individually above rather than presented as fully certain: RLEF's exact co-author ordering and Constitutional AI's complete ~50-person author list could not be verified byte-for-byte against the primary arXiv abstract page in this research pass, since arxiv.org itself wasn't directly browsable in this environment. If you're citing either paper formally, pull the author list directly from the arXiv page rather than from this summary.