AGI Researcher Foundations · Article 6 of 11 🎮

Reinforcement Learning Foundations: The Math Underneath RLHF

Articles 1, 2, and this series' "After Transformers" companion all reference RLHF as a working technique without explaining the theory underneath it. This article gives you that theory — value functions, the Bellman equation, policy gradients, and the exploration-exploitation tradeoff — plain-language first, with the real papers and the direct line from classical RL to PPO, the exact algorithm used inside modern RLHF.

FrontierAGI Team

September 14, 2026 · 75 min read

Opening The MDP Framework Value Functions & Bellman Exploration vs. Exploitation Policy Gradients From RL to RLHF Key Papers Courses to Complete Real Scenarios Self-Assessment What's Missing Where This Series Goes Videos References Verdict

Why RLHF Needs Its Own Theory Article

Article 2's self-assessment and "How Each Network Architecture Actually Learned"'s Era 8 both described RLHF mechanically — generate outputs, rank them, train a reward model, update the policy — without explaining the reinforcement learning theory that makes "update the policy" a well-defined mathematical operation rather than a vague gesture. This article fills that gap: reinforcement learning (RL) predates deep learning by decades as its own field, with its own vocabulary, and RLHF is a specific, relatively recent application of ideas that were developed for a very different original purpose — teaching agents to play games and control robots.

The scoping rule from Article 3 applies again here: this article covers the RL concepts that show up directly in RLHF and in this series' other RL references (Berkeley CS285, DeepMind x UCL, both cited in Article 1), not the full breadth of classical RL research.

1957 Year Richard Bellman's dynamic programming work introduced the equation underlying every value function in this article

2016 Year AlphaGo, an RL system, beat a world champion Go player — RL's most famous pre-LLM result

PPO The 2017 policy-gradient algorithm that InstructGPT and most RLHF systems actually use under the hood

2 Core RL problems this article covers: learning what's valuable, and learning what to do about it

Part 1 — The Basic Framework

Agents, States, Actions, and Rewards

Every RL problem, from a robot arm to an LLM being fine-tuned on human preferences, is described using the same handful of ingredients, formalized as a "Markov Decision Process" (MDP):

State — a snapshot of the current situation the agent is in (a board position in chess; the conversation so far, for an LLM).

Action — a choice the agent can make from that state (a chess move; the next word to generate).

Reward — a number received after taking an action, signaling how good or bad that action was (winning a game; a human rater's preference score, in RLHF).

Policy — the agent's strategy: a (possibly learned) rule for which action to take in each state. Training an RL agent means improving this policy over time.

The word "Markov" refers to a simplifying assumption: the current state contains all the information needed to decide what to do next — you don't need the entire history, just where you are right now. This assumption is what makes the math in Part 2 tractable, and it's also a real, sometimes-violated simplification worth being aware of (a long conversation's full context may matter more than an LLM's compressed internal state fully captures).

Part 2 — Value Functions and the Bellman Equation

Learning What's Actually Valuable, Not Just What Feels Good Right Now

A reward is immediate — but a good action might have a low immediate reward and a great long-term payoff (a chess move that looks passive but sets up a winning position ten moves later). A value function estimates the total future reward an agent can expect from a given state (or state-action pair), not just the next reward — this is the mathematical tool that lets an agent reason about delayed consequences.

Richard Bellman's foundational insight, from his 1957 work on dynamic programming, is that a state's value can be defined recursively: the value of being in a state equals the immediate reward you'd get, plus the (discounted) value of whatever state you land in next.

Value(state) = immediate_reward + discount × Value(next_state)

Plain terms: "how good is this situation?" equals "what do I get right now?" plus "roughly how good will things be after that, counted a little less than getting the same value immediately" — the discount factor makes future rewards worth slightly less than immediate ones, which is both mathematically convenient and matches real intuition (a reward now is worth more than the same reward later).

This recursive relationship — the Bellman equation — is the mathematical backbone of nearly every value-based RL method, including Q-learning (Watkins, 1989) and the Deep Q-Network that famously learned to play Atari games directly from pixels (Mnih et al., Nature, 2015).

A value function is the mathematical answer to "was that actually a good move, even though it didn't pay off immediately?" — exactly the kind of judgment a good researcher (Article 2, Article 5) also has to make about a training run that hasn't converged yet.

Part 3 — Exploration vs. Exploitation

The Tradeoff Every RL Agent Has to Manage

An agent that only ever repeats the action it currently believes is best (exploitation) can get permanently stuck on a mediocre strategy, never discovering something better it hasn't tried. An agent that only ever tries new things (exploration) never capitalizes on what it's already learned. Balancing these two is a genuinely hard, actively studied problem, most cleanly illustrated by the "multi-armed bandit" problem: imagine several slot machines with unknown, different payout rates — how do you decide when to keep pulling the machine that's paid off so far, versus trying an unexplored one that might be better? Auer et al.'s UCB algorithm (2002) is a classic, mathematically principled answer: favor actions that are either high-value or under-tried, with a formula that naturally shifts toward exploitation as more evidence accumulates.

Strategy	Risk	Where It Shows Up
Pure exploitation	Gets stuck on a locally-good, globally-suboptimal strategy	An RLHF-tuned model that only ever produces "safe," previously-rewarded response styles
Pure exploration	Never converges, wastes resources on already-known-bad options	An agent that never settles into a usable, predictable strategy
UCB / principled balance	More complex to implement correctly	Classical bandit algorithms; conceptually related to how RLHF's KL penalty (Part 4) keeps a policy from drifting too erratically

Part 4 — Policy Gradients

Learning a Strategy Directly, Instead of Learning Values First

Value-based methods (Part 2) learn how good states are, then derive a policy from that (pick the action leading to the highest-value state). Policy gradient methods take a more direct approach: represent the policy itself as a trainable function (in modern RLHF, a neural network — the same LLM being fine-tuned), and directly adjust its parameters to make high-reward actions more likely and low-reward actions less likely.

Williams' 1992 REINFORCE algorithm is the classical starting point: run the current policy, observe the resulting reward, and push up the probability of whatever actions were taken, scaled by how good the outcome turned out to be. Sutton et al.'s 1999 policy gradient theorem formalized why this works mathematically, giving the field a principled foundation for a whole family of methods that followed.

nudge parameters ∝ (reward received) × (gradient of the probability of the action taken)

Plain terms: "if this action led to a good outcome, make the policy more likely to choose it again in similar situations; if it led to a bad outcome, make it less likely" — directly echoing Article 3's chain-rule intuition, just applied to actions and rewards instead of a labeled loss.

The specific algorithm used in modern RLHF is Schulman et al.'s Proximal Policy Optimization (PPO, 2017) — a refinement of the basic policy-gradient idea that adds a safeguard against updating the policy too aggressively in any single step (a real, practical failure mode of naive policy gradients, where one bad update can destabilize the entire policy). This is the literal algorithm cited in InstructGPT's methodology ("How to Read a Paper Like a Researcher").

Part 5 — From Classical RL to RLHF

Mapping RLHF's Pieces Onto This Article's Vocabulary

State = the conversation so far (the prompt plus whatever the model has generated up to this point).

Action = generating the next token.

Reward = the learned reward model's score (Article 2, Era 8 of "How Each Network Architecture Actually Learned"), standing in for a real human preference signal that can't be queried live for every single training step.

Policy = the LLM itself, and PPO is the specific policy-gradient algorithm used to update it.

The exploration-exploitation tradeoff, in RLHF form — RLHF typically includes a KL-divergence penalty (Article 3's information theory) keeping the fine-tuned policy close to its original pretrained behavior, which is functionally an exploitation-favoring safeguard: it stops the policy from "exploring" too far into behaviors the reward model might score well but that don't reflect genuine human preference (reward hacking, "After Transformers").

Seen this way, RLHF isn't a mysterious new invention — it's a specific, well-motivated application of decades-old RL machinery (Bellman's recursive value idea is decades older, PPO is from 2017) to a problem (aligning a language model's behavior) that classical RL was never originally designed for, but turns out to map onto cleanly.

Part 6 — Papers & Courses

Key Papers to Read First

Paper	Why It's Foundational	Link
Williams — "Simple Statistical Gradient-Following Algorithms" — REINFORCE (1992)	The classical policy-gradient starting point	Springer
Sutton et al. — "Policy Gradient Methods..." (1999)	Formalizes why policy gradients work — the theoretical foundation for PPO and everything after	NeurIPS
Mnih et al. — "Human-Level Control Through Deep RL" — DQN (2015)	The landmark deep-RL result combining value functions (Part 2) with deep networks	Nature
Silver et al. — "Mastering the Game of Go..." — AlphaGo (2016)	RL's most famous pre-LLM public result, combining value functions with tree search	Nature
Schulman et al. — "Proximal Policy Optimization Algorithms" — PPO (2017)	The literal algorithm used inside InstructGPT and most modern RLHF systems	arXiv:1707.06347
Christiano, Leike et al. — "Deep RL from Human Preferences" (2017, previously cited in Article 1)	The foundational RLHF paper this entire article builds toward explaining	arXiv:1706.03741

Courses to Complete

Sutton & Barto — Reinforcement Learning: An Introduction

Free · The canonical RL textbook

Covers everything in this article — MDPs, value functions, policy gradients — in full mathematical rigor, freely available online, written by two of the field's founding researchers.

Berkeley CS285 — Deep Reinforcement Learning

Free · Previously cited in Article 1

The deepest freely available RL course, taught by Sergey Levine — the natural next step after this article for anyone pursuing RL-heavy research.

DeepMind x UCL — Reinforcement Learning Course

Free · Previously cited in Article 1

A lab-grade RL course covering this article's material in more depth.

Caveat, as noted in Article 1: no confirmed recent refresh — treat as an evergreen reference.

Part 7 — Real Scenarios

Real Scenario Walkthroughs

🎯Scenario A — Diagnosing an RLHF Run That "Reward Hacks"

"After Transformers" named reward hacking as RLHF's core limitation; this article's vocabulary makes it precise: the policy (Part 5) has found actions that score highly according to the learned reward model (an imperfect stand-in for genuine human preference) without those actions actually reflecting what a human wants — exactly the exploitation failure mode from Part 3, taken to an extreme where the "reward" being exploited is itself flawed. Recognizing this as a specific, nameable RL failure mode — not a vague "the model got weird" — is what lets a researcher diagnose it methodically: check whether the reward model's judgments still track real human preference on the specific outputs the policy has started favoring.

The lesson: reward hacking isn't mysterious once you have the value-function and exploitation vocabulary from Parts 2–3 — it's exploitation of a flawed reward signal, which is a well-studied RL failure mode with a name.

🎲Scenario B — Why AlphaGo Needed Both Value Functions and Search

Silver et al.'s AlphaGo combined a learned value function (estimating how good a board position is) with tree search (explicitly trying out several possible future move sequences before committing) — neither alone was sufficient. This foreshadows a genuine open question for LLM reasoning models ("After Transformers," Part 1): whether today's test-time compute methods are effectively a lighter-weight form of the same search-plus-value-estimate combination that worked for Go, applied to token sequences instead of board positions.

The lesson: the combination of "estimate how good this is" (value) and "actually try a few options before deciding" (search) recurs across very different RL applications — recognizing the pattern helps you evaluate new methods faster.

Self-Assessment Checklist

Can you state, in your own words, what state/action/reward/policy each mean for a general RL problem, and specifically for RLHF?

Can you explain the Bellman equation's recursive idea without needing to write the formula down first?

Can you explain the exploration-exploitation tradeoff using a concrete example (a slot machine, a restaurant choice) rather than only in the abstract?

Do you understand why PPO adds a safeguard against large policy updates, and what could go wrong without one?

Can you explain reward hacking as a specific RL concept (exploiting a flawed reward signal) rather than a vague description of "bad AI behavior"?

⚠️ What's Missing or Uncertain in This Article

This article does not cover: model-based RL (learning an explicit model of the environment to plan with — directly relevant to the world-models open bet in "After Transformers," but left for that article's own treatment), multi-agent RL, or the full mathematical derivation of PPO's clipped objective function. It also does not resolve the open question in Scenario B above (whether test-time reasoning is meaningfully "RL-flavored search") — that remains a genuinely unsettled characterization as of September 2026, mentioned here as a connection worth noticing, not a settled fact.

Where This Series Goes Next

Article 7 moves from RL foundations to interpretability — how researchers actually look inside a trained model to understand what it's doing, building directly on this site's Interpretability Gap article and Anthropic's superposition research (referenced in "After Transformers"). From there, the series continues through alignment, world models, systems, and research methodology, before a capstone article ties every foundational concept back into one unified map.

🔗 Additional Reference Links

🎥 Recommended Videos

The Spelled-Out Intro to Neural Networks and Backpropagation

Andrej Karpathy — gradient intuition underlying Part 4

But What Is a Neural Network?

3Blue1Brown — background for the policy network in Part 5

🧭 Closing — RLHF Is Old Math, Applied to a New Problem

🎯 The Bottom Line

Value functions trace to Bellman's 1957 dynamic programming work; policy gradients trace to Williams' 1992 REINFORCE; PPO, the algorithm literally running inside InstructGPT-style RLHF, is from 2017 — none of this is new math invented for language models. RLHF's real contribution was recognizing that a decades-old toolkit, built for games and robotics, maps cleanly onto the problem of aligning a language model's behavior with human preference. Understanding that mapping — state/action/reward/policy, the exploration-exploitation tradeoff, why reward hacking is a named, well-studied failure mode rather than a mysterious AI quirk — is what separates using RLHF as a black box from actually understanding what it's doing.