How Post-Training Works: SFT, RLHF, DPO and RLAIF
A pretrained model can write fluently but doesn't reliably do what you ask, and nothing in next-token prediction makes it helpful, honest or safe. Post-training is how that changes: supervised fine-tuning, then learning from preferences — with a reward model and reinforcement learning, or directly with DPO — then evaluation, deployment and a feedback loop. This piece walks the whole pipeline, and runs SFT, DPO and RLHF side by side on a small model so you can see how they actually differ.
From Base Model to Assistant
A base model's capabilities are real — broad knowledge, fluent language, good next-token prediction — but so are its limits. It may not follow instructions, may be verbose or unhelpful, can produce unsafe or biased content, and was never optimized for what people actually prefer. Post-training addresses those limits in four stages:
1 · Supervised fine-tuning
- Train on demonstrations of good responses
- Teaches the model to follow instructions
2 · Preference optimization
- Collect comparisons between responses
- Optimize with RLHF, DPO or RLAIF
3 · Evaluation and safety testing
- Capability, safety and human evaluation
- Automated red teaming
4 · Deployment and monitoring
- Serve efficiently
- Collect feedback and iterate
The recipe was established by OpenAI's InstructGPT: supervised fine-tuning on demonstrations written by labelers, a reward model trained on labelers' rankings of the model's outputs, and reinforcement learning (PPO) against that reward model. The payoff was striking. In human evaluations, outputs from the 1.3B-parameter InstructGPT were preferred over those of the original 175B GPT-3 — a model 100× larger — and InstructGPT produced truthful and informative answers about twice as often.
Stage 1: Supervised Fine-Tuning
SFT trains the base model on (instruction, response) pairs with the ordinary next-token loss, counted only on the response:
SFT's strength is also its limit: it teaches the model to imitate the demonstrations — including their flaws. Our experiment makes this concrete. We fine-tuned a small model on demonstrations where half the answers were a real answer and half were "I don't know." The resulting SFT model faithfully learned to say "I don't know" 44% of the time. Nothing in the SFT loss says one response is better than another; to teach that, you need comparisons.
Stage 2: Preference Data
Preference data is built in three steps: generate several responses to the same prompt from the SFT model, have a judge — human labelers or an AI model — compare or rank them, and store the result as pairs (x, yw, yl): a prompt, the preferred ("winner") response and the less preferred ("loser") one. A ranking of K responses yields many pairs; InstructGPT's labelers ranked several model outputs per prompt this way.
Why comparisons instead of more demonstrations? Judging which of two answers is better is far easier and more consistent than writing the ideal answer, so it scales to more data and captures qualities — tone, honesty, the right level of detail — that are hard to specify but easy to recognize. The standard way to turn comparisons into a learning signal is the Bradley–Terry model: the probability that yw is preferred is σ(r(x, yw) − r(x, yl)) for some score r, where σ is the logistic function. Both RLHF and DPO start from this assumption.
RLHF: A Reward Model, Then Reinforcement Learning
Step 1 — train a reward model. Take the SFT model, replace its output layer with a single number, and train it to score preferred responses above rejected ones:
Step 2 — optimize the policy against it. Generate responses, score them, and use reinforcement learning to make high-scoring responses more likely, with a penalty for drifting from the SFT model:
That KL term does real work. A reward model is only a proxy for human judgment, and optimizing any proxy hard enough eventually finds its blind spots — Goodhart's law. OpenAI measured this directly: optimizing against a proxy reward model first raises and then lowers the score from a "gold-standard" judge, with predictable curves that depend on reward-model size. The symptoms, often called reward hacking, include responses that are longer, more flattering or more formulaic than they should be. PPO itself is a policy-gradient method with a clipped objective that limits how far each update can move the policy; for language models it also trains a value network (a critic) to estimate expected reward, which adds another model to the setup.
DPO: Skipping the Reward Model
Direct Preference Optimization starts from the same KL-regularized objective and a mathematical observation: its optimal policy can be written in closed form in terms of the reward, which means the reward can be rewritten in terms of the policy. The reward model becomes implicit — β · log(πθ(y|x) / πref(y|x)) — and training reduces to a classification-style loss on preference pairs, with no reward model and no sampling during training:
Explore the loss. The margin is the bracketed term: how much more the policy prefers yw over yl than the reference did. The gradient weight shows how hard DPO pushes on a pair — pairs the model already ranks correctly get little weight:
DPO is simpler, cheaper and more stable to run than PPO — which is why it spread quickly through open-model post-training — but it has its own failure mode, and our experiment hit it. Because the loss only cares about the gap between chosen and rejected responses, it can widen the gap by pushing both down. In our run, DPO lowered the log-probability of the rejected answer by 15.96 but also lowered the chosen answer by 1.17. Pal et al. showed this theoretically and proposed DPO-Positive to prevent it; they found it most common when the chosen and rejected responses differ in only a few tokens.
RLAIF: AI Feedback Instead of Human
Human preference labels are slow and expensive. RLAIF replaces the human judge with a capable model: generate responses, ask a judge model which is better (often against a written set of principles), and train with DPO or RL exactly as before. Anthropic's Constitutional AI introduced the approach for harmlessness, with human oversight provided only through a list of principles: a supervised phase where the model critiques and revises its own responses, then an RL phase using AI preference labels. Google's comparison found RLAIF performed comparably to RLHF across summarization, helpful dialogue and harmless dialogue; on harmlessness it scored higher (88% harmless, versus 76% for RLHF and 64% for the SFT baseline). The main risk is that the policy inherits the judge's biases, so AI feedback is usually combined with some human review.
Beyond the Diagram: RL with Verifiable Rewards
For tasks with checkable answers — math with a known result, code with test cases — the reward doesn't need to be learned at all. A rule-based checker says right or wrong, and there's nothing to hack in the way a learned reward model can be hacked. DeepSeek trained its R1 reasoning models this way, with rewards based on the correctness of final answers (and a compiler with test cases for programming). Their optimizer, GRPO, drops PPO's critic entirely: it samples a group of responses per prompt and uses the group's average reward as the baseline. Our experiment's RLHF stage uses that same group-relative baseline, with a learned reward model instead of a checker.
Comparing the Methods
| Method | Training data | What's trained | Objective | Strengths | Weaknesses |
|---|---|---|---|---|---|
| SFT | (x, y) demonstrations | Policy θ | Next-token cross-entropy | Simple, stable; teaches instruction following | Imitates demonstrations, flaws included; no notion of "better" |
| RLHF | (x, yw, yl) pairs + sampled rollouts | Reward model φ, then policy θ (and a critic, with PPO) | Bradley–Terry, then reward − β·KL | Flexible; learns from its own outputs; can target complex goals | Several models to run; less stable; reward hacking |
| DPO | (x, yw, yl) pairs | Policy θ (reference frozen) | Direct preference loss | No reward model, no sampling; cheap and stable | Limited to fixed pairs; can lower the chosen response's likelihood |
| RLAIF | AI-labeled pairs | As DPO or RLHF | As DPO or RLHF | Cheap, fast, scalable labels | Only as good as the judge; inherits its biases |
| RLVR | Prompts with checkable answers | Policy θ | Verified reward (e.g. GRPO) | No reward model to hack; drove recent reasoning gains | Only works where answers can be checked |
What Our Experiment Shows
Same starting point for all three: the SFT model that says "I don't know" about half the time. DPO and RLHF each trained on preference pairs for 8 of the 12 questions, preferring the real answer over "I don't know." We then sampled 40 answers per question from each model and sorted them into correct, "I don't know," and malformed. "Held-out" means the four questions not used in preference training (SFT had seen all twelve).
Keep this in proportion. This is an 843K-parameter model with 8 preference pairs, and DPO's hyperparameters weren't tuned beyond a single setting. At real scale, well-tuned DPO performs very well, and many strong open models use it. The durable lessons are the mechanisms: preference optimization can generalize a behavior from few examples; DPO can lower the likelihood of what you prefer; and on-policy RL corrects its own mistakes at the cost of more machinery.
Evaluation and Safety Testing
A post-trained model is evaluated on four fronts before release:
Deployment and Monitoring
Serving the model efficiently is its own engineering discipline, covered in depth in How LLM Inference Works. The standard techniques: a KV cache so each new token doesn't recompute attention over the whole context; continuous batching of many users' requests, with memory managers like vLLM's PagedAttention that store the KV cache in pages the way an operating system manages memory, cutting waste by up to 96%; quantization of weights to 8 or 4 bits; speculative decoding, where a small draft model proposes several tokens and the large model verifies them in one pass — 2–3× faster with identical outputs in the original paper; and memory-efficient attention kernels like FlashAttention.
Post-training doesn't end at release. User feedback (ratings, reports) and logged outputs feed automatic analysis that detects failures and unsafe responses, finds common issues and clusters problematic prompts. Those become new training data — hard examples, updated preference pairs, safety data — for the next round of SFT and preference optimization. The pipeline is a loop.
Run the Experiment
The full comparison in one script, about three minutes on a laptop CPU: pretrain a small model; SFT it on demonstrations where half the answers are "I don't know"; build preference pairs; run DPO against the frozen SFT reference; train a Bradley–Terry reward model and run policy-gradient RL with a KL penalty and a group-relative baseline; then sample from every model and measure behavior and drift. Needs PyTorch 2.x.
"""Post-training in miniature: SFT -> preference pairs -> DPO, and SFT -> reward model -> RL with a KL penalty.""" import copy, itertools, random import torch import torch.nn as nn import torch.nn.functional as F random.seed(0); torch.manual_seed(0); torch.set_num_threads(4) # ---------------- 1. A pretrained base model (condensed from the pretraining article) ---------------- facts = [ ("The cat sat on the mat and watched the rain.", "Where did the cat sit?", "On the mat."), ("The dog chased a red ball across the park.", "What did the dog chase?", "A red ball."), ("Water boils at one hundred degrees Celsius.", "When does water boil?", "At one hundred degrees."), ("The sun rises in the east every morning.", "Where does the sun rise?", "In the east."), ("Bees collect nectar and turn it into honey.", "What do bees make?", "Honey."), ("The river flows into the quiet lake.", "Where does the river flow?", "Into the quiet lake."), ("Children read books in the library after school.", "Where do children read?", "In the library."), ("Trains leave the station every hour.", "How often do trains leave?", "Every hour."), ("The farmer harvested the wheat in autumn.", "When was the wheat harvested?", "In autumn."), ("Birds build their nests in tall trees.", "Where do birds build nests?", "In tall trees."), ("The moon pulls on the ocean and causes the tides.", "What causes the tides?", "The moon."), ("Snow covered the mountain before the hikers left.", "What covered the mountain?", "Snow."), ] UNHELPFUL = "I don't know." EOT, V, T = 256, 257, 128 enc = lambda s: list(s.encode()) dec = lambda ids: bytes(i for i in ids if i < 256).decode(errors="replace") pages = [" ".join(f for f, _, _ in c) for c in itertools.permutations(facts, 3)] random.shuffle(pages) stream = torch.tensor([t for p in pages[:600] for t in enc(p) + [EOT]]) d, H, L = 128, 4, 4 class Block(nn.Module): def __init__(self): super().__init__() self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d) self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d) self.up, self.down = nn.Linear(d, 4 * d), nn.Linear(4 * d, d) def forward(self, x): B_, T_, _ = x.shape q, k, v = self.qkv(self.ln1(x)).split(d, dim=-1) q, k, v = (t.view(B_, T_, H, d // H).transpose(1, 2) for t in (q, k, v)) x = x + self.proj(F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(B_, T_, d)) return x + self.down(F.gelu(self.up(self.ln2(x)))) class GPT(nn.Module): def __init__(self): super().__init__() self.tok, self.pos = nn.Embedding(V, d), nn.Embedding(T, d) self.blocks = nn.Sequential(*[Block() for _ in range(L)]) self.ln_f, self.head = nn.LayerNorm(d), nn.Linear(d, V, bias=False) self.head.weight = self.tok.weight for m in self.modules(): if isinstance(m, (nn.Linear, nn.Embedding)): nn.init.normal_(m.weight, std=0.02) if isinstance(m, nn.Linear) and m.bias is not None: nn.init.zeros_(m.bias) def hidden(self, idx): return self.ln_f(self.blocks(self.tok(idx) + self.pos(torch.arange(idx.shape[1])))) def forward(self, idx): return self.head(self.hidden(idx)) base = GPT() opt = torch.optim.AdamW(base.parameters(), lr=2e-3, weight_decay=0.1) for step in range(500): i = torch.randint(0, len(stream) - T - 1, (32,)) x = torch.stack([stream[j:j + T] for j in i]); y = torch.stack([stream[j + 1:j + T + 1] for j in i]) loss = F.cross_entropy(base(x).reshape(-1, V), y.reshape(-1)) opt.zero_grad(); loss.backward(); torch.nn.utils.clip_grad_norm_(base.parameters(), 1.0); opt.step() # ---------------- helpers: batching (prompt, response) pairs, log-probs, sampling ---------------- prompt_ids = lambda q: enc(f"Q: {q}\nA: ") def make_batch(pairs): """pairs of (question, response) -> input ids, labels with the prompt masked to -100.""" rows = [(prompt_ids(q), enc(r) + [EOT]) for q, r in pairs] seqs = [p + r for p, r in rows] width = max(len(s) for s in seqs) - 1 X = torch.tensor([s[:-1] + [EOT] * (width - len(s) + 1) for s in seqs]) Y = torch.tensor([([-100] * len(p) + r)[1:] + [-100] * (width - len(p) - len(r) + 1) for p, r in rows]) return X, Y def seq_logprob(model, X, Y): """Sum of log-probabilities of the response tokens: log pi(y | x).""" logp = model(X).log_softmax(-1) mask = Y != -100 tok = logp.gather(-1, Y.clamp(min=0).unsqueeze(-1)).squeeze(-1) return (tok * mask).sum(-1) @torch.no_grad() def sample(model, q, n): """n responses at temperature 1.""" ids = torch.tensor([prompt_ids(q)] * n) done = torch.zeros(n, dtype=torch.bool) out = [[] for _ in range(n)] for _ in range(30): nxt = torch.multinomial(model(ids)[:, -1].softmax(-1), 1).squeeze(-1) for k in range(n): if not done[k]: if nxt[k].item() == EOT: done[k] = True else: out[k].append(nxt[k].item()) if done.all(): break ids = torch.cat([ids, nxt[:, None]], dim=1) return [dec(o) for o in out] def behavior(model, qa, n=40): """Fraction of sampled answers that are correct / 'I don't know.' / anything else.""" torch.manual_seed(123) c = {"correct": 0, "unhelpful": 0, "other": 0} for _, q, a in qa: for r in sample(model.eval(), q, n): c["correct" if r == a else "unhelpful" if r == UNHELPFUL else "other"] += 1 total = n * len(qa) return {k: f"{100 * v / total:.0f}%" for k, v in c.items()} train_qa, heldout_qa = facts[:8], facts[8:] # ---------------- 2. SFT on imperfect demonstrations: half the answers are unhelpful ---------------- sft = copy.deepcopy(base) X, Y = make_batch([(q, a) for _, q, a in facts] + [(q, UNHELPFUL) for _, q, a in facts]) opt = torch.optim.AdamW(sft.parameters(), lr=3e-4) for step in range(400): loss = F.cross_entropy(sft.train()(X).reshape(-1, V), Y.reshape(-1), ignore_index=-100) opt.zero_grad(); loss.backward(); opt.step() print(f"SFT loss {loss.item():.3f} (ln 2 = 0.693 would mean a pure coin flip between the two answers)") print("SFT model train prompts:", behavior(sft, train_qa), " held-out:", behavior(sft, heldout_qa)) # ---------------- 3. Preference pairs: (x, y_w = the real answer, y_l = "I don't know.") ---------------- Xw, Yw = make_batch([(q, a) for _, q, a in train_qa]) Xl, Yl = make_batch([(q, UNHELPFUL) for _, q, a in train_qa]) ref = copy.deepcopy(sft).eval().requires_grad_(False) # frozen reference = the SFT model with torch.no_grad(): ref_w, ref_l = seq_logprob(ref, Xw, Yw), seq_logprob(ref, Xl, Yl) # ---------------- 4a. DPO ---------------- beta = 0.1 dpo = copy.deepcopy(sft) opt = torch.optim.AdamW(dpo.parameters(), lr=1e-5) for step in range(60): pi_w, pi_l = seq_logprob(dpo.train(), Xw, Yw), seq_logprob(dpo, Xl, Yl) logits = (pi_w - pi_l) - (ref_w - ref_l) # how much MORE the policy prefers y_w than ref does loss = -F.logsigmoid(beta * logits).mean() opt.zero_grad(); loss.backward(); opt.step() if step in (0, 59): print(f"DPO step {step:>2}: loss {loss.item():.3f} implicit reward margin {beta * logits.mean().item():+.2f}") with torch.no_grad(): d_w = (seq_logprob(dpo.eval(), Xw, Yw) - ref_w).mean().item() d_l = (seq_logprob(dpo, Xl, Yl) - ref_l).mean().item() print(f" log p(chosen) moved {d_w:+.2f}, log p(rejected) moved {d_l:+.2f} (both relative to the reference)") print("DPO model train prompts:", behavior(dpo, train_qa), " held-out:", behavior(dpo, heldout_qa)) torch.manual_seed(5) odd = [r for _, q, a in train_qa for r in sample(dpo, q, 5) if r not in (a, UNHELPFUL)] print(" examples of 'other':", odd[:4]) # ---------------- 4b. RLHF: reward model, then policy gradient with a KL penalty ---------------- class RewardModel(nn.Module): """The SFT network with a scalar head reading the final token's hidden state.""" def __init__(self, lm): super().__init__() self.lm, self.score = copy.deepcopy(lm), nn.Linear(d, 1) def forward(self, X, Y): last = (Y != -100).sum(-1) + (Y == -100).long().argmin(-1) - 1 # index of the final response token h = self.lm.hidden(X) return self.score(h[torch.arange(len(X)), last]).squeeze(-1) rm = RewardModel(sft) opt = torch.optim.AdamW(rm.parameters(), lr=1e-4) for step in range(100): loss = -F.logsigmoid(rm(Xw, Yw) - rm(Xl, Yl)).mean() # Bradley-Terry: r(y_w) should beat r(y_l) opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): acc = (rm(Xw, Yw) > rm(Xl, Yl)).float().mean().item() print(f"reward model: pairwise loss {loss.item():.3f}, ranks {acc:.0%} of training pairs correctly") rl = copy.deepcopy(sft) opt = torch.optim.AdamW(rl.parameters(), lr=1e-5) kl_coef, group = 0.05, 8 for step in range(40): qs = [q for _, q, _ in train_qa] pairs = [(q, r) for q in qs for r in sample(rl.eval(), q, group)] # a group of samples per prompt Xs, Ys = make_batch(pairs) with torch.no_grad(): reward = rm(Xs, Ys) - kl_coef * (seq_logprob(rl, Xs, Ys) - seq_logprob(ref, Xs, Ys)) r = reward.view(len(qs), group) adv = ((r - r.mean(1, keepdim=True)) / (r.std(1, keepdim=True) + 1e-6)).view(-1) # group-relative baseline loss = -(adv * seq_logprob(rl.train(), Xs, Ys)).mean() # REINFORCE opt.zero_grad(); loss.backward(); opt.step() print("RLHF model train prompts:", behavior(rl, train_qa), " held-out:", behavior(rl, heldout_qa)) # ---------------- 5. How far did each model move from the SFT reference? ---------------- @torch.no_grad() def kl_from_ref(model, qa, n=20): torch.manual_seed(7) pairs = [(q, r) for _, q, _ in qa for r in sample(model.eval(), q, n)] Xs, Ys = make_batch(pairs) return (seq_logprob(model, Xs, Ys) - seq_logprob(ref, Xs, Ys)).mean().item() print(f"KL from SFT reference (nats per response): DPO {kl_from_ref(dpo, train_qa):.2f}, RLHF {kl_from_ref(rl, train_qa):.2f}")
Output (PyTorch 2.14 on CPU; sampling and multithreaded CPU math make exact percentages vary between runs — the RLHF numbers shifted by a few points across our own runs, while the qualitative pattern held):
SFT loss 0.058 (ln 2 = 0.693 would mean a pure coin flip between the two answers)
SFT model train prompts: {'correct': '48%', 'unhelpful': '44%', 'other': '8%'} held-out: {'correct': '45%', 'unhelpful': '50%', 'other': '5%'}
DPO step 0: loss 0.693 implicit reward margin +0.00
DPO step 59: loss 0.214 implicit reward margin +1.47
log p(chosen) moved -1.17, log p(rejected) moved -15.96 (both relative to the reference)
DPO model train prompts: {'correct': '41%', 'unhelpful': '0%', 'other': '59%'} held-out: {'correct': '81%', 'unhelpful': '0%', 'other': '19%'}
examples of 'other': ['s aut onendred the moun.', 's aut oneley.', 's aumn.', '\x7fn the mat.']
reward model: pairwise loss 0.000, ranks 100% of training pairs correctly
RLHF model train prompts: {'correct': '78%', 'unhelpful': '2%', 'other': '20%'} held-out: {'correct': '85%', 'unhelpful': '6%', 'other': '9%'}
KL from SFT reference (nats per response): DPO 4.54, RLHF 1.20⚠️ Notes
trainers.py, which also suggests β of 0.1–0.5); Constitutional AI's two phases; RLAIF's comparable performance and 88/76/64% harmlessness figures; reward-model overoptimization; DPO's chosen-likelihood failure mode (Pal et al.); GRPO and DeepSeek-R1's correctness rewards; PPO's clipped objective; the benchmark descriptions; PagedAttention's up-to-96% waste reduction; speculative decoding's 2–3× speedup with identical outputs; FlashAttention. The example preference pair is illustrative. All experiment numbers are real outputs of the included script, from one run; the DPO-vs-RLHF comparison reflects a single untuned setting at toy scale and is presented as a mechanism demonstration, not a ranking of methods.