Home โ€บ Blog โ€บ Post-Training
Producing Intelligence ยท Part 6 ยท Post-training

Post-Training: How Llama 3 405B Became an Assistant

At the end of Part 5, Meta had a 405-billion-parameter base model that could continue any text but could not hold a conversation, follow an instruction or say no. Turning it into Llama 3.1 405B Instruct took a much smaller amount of compute but a very different kind of engineering: human preference data, a reward model, rejection sampling, supervised fine-tuning and Direct Preference Optimization, run in repeated rounds. This part walks through that loop, the data that fed it and why Meta chose DPO over the PPO-based RLHF that made ChatGPT.

FL
FrontierAGI Team
โš ๏ธ How this was researched Facts come from section 4 of Meta's paper "The Llama 3 Herd of Models" (arXiv 2407.21783), mostly as quoted in search results, and from Meta's Llama Guard 3 model card. A few details are taken only from secondary summaries such as course slides; these are marked โš ๏ธ secondary. The round counts in the loop diagram, the best-of-N simulator and the DPO loss explorer are illustrations of the method, not Meta's measurements. See Reading the numbers.
4 stepsper round: reward model โ†’ rejection sampling โ†’ SFT โ†’ DPO
6 roundsof the loop โš ๏ธ secondary (course slides)
ฮฒ = 0.1DPO, learning rate 1e-5
52.66%of SFT examples are general English chat
14hazard categories in Llama Guard 3

1. Base model vs assistant

Pre-training teaches a model to predict the next token of internet-scale text. That gives it knowledge and skill, but its default behaviour is to continue a document, not to answer the person writing it. Ask a base model a question and it may produce more questions, a forum thread, or a plausible but unhelpful paragraph. Post-training changes the model's default behaviour: answer helpfully, follow the format asked for, use tools when given them, and refuse what it should refuse.

The same prompt, before and after

Illustrative responses showing typical behaviour, not real outputs from Llama 3.

Compared with pre-training it is cheap. Pre-training of the 405B model consumed 30.84M H100-hours, while post-training runs over millions of examples, not trillions of tokens. Meta did not publish a GPU-hour figure for post-training in our sources. It is still heavy engineering: every round needs fresh human annotations, a newly trained reward model, and a large inference job that generates and scores candidate answers.

2. The round loop

Meta describes post-training as an iterative procedure. In each round it collects new human preference data, trains a reward model on it, uses that reward model to pick the best of several sampled answers (rejection sampling), fine-tunes on the result (SFT), and then aligns further with DPO. The best model from the round produces the samples for the next one, so data quality and model quality improve together. Course slides on the paper put the count at six rounds โš ๏ธ secondary.

One round of Llama 3 post-training

Click a step to see what happens there. Move the slider to step through rounds.

Illustrative: Meta does not publish per-round statistics in our sources.
๐Ÿ’ก Why rounds? Preference labels are most informative when they compare answers the current model actually produces. Old labels compare responses from a weaker model, so the reward model learns to separate mistakes the new model no longer makes. Collecting fresh data each round keeps the comparisons close to where the model currently is.

3. Preference data

Human annotators were shown a prompt and two responses, sampled from different models, and asked which they preferred and by how much, on four levels: significantly better, better, slightly better, or marginally better. They could also edit the chosen response to make it better, which gives three ranked responses for that prompt: edited > chosen > rejected. For multi-turn conversations, the comparison was made turn by turn.

An annotation, as a reward model sees it

Pick a preference strength. Summaries of the paper say pairs where the two responses were close were dropped from reward-model training โš ๏ธ secondary.

Response A

To reverse a list in Python use my_list[::-1], which returns a new list, or my_list.reverse(), which changes it in place.

Response B

You can reverse a list by writing a loop that walks backwards from the last index and appends each element to a new list.

4. The reward model and rejection sampling

The reward model starts from the pre-trained checkpoint and learns to score a (prompt, response) pair so that preferred responses score higher. It is never shown to users. Its job is to be a cheap stand-in for a human judge. Meta then uses it for rejection sampling: for each human-written prompt, generate several answers from the current best model, score them all, keep the highest-scoring one, and add it to the SFT training set.

That is the "best-of-N" idea, and it has diminishing returns: the expected best of N samples rises roughly with the square root of the logarithm of N, while generation cost rises linearly with N.

Best-of-N simulator

Each answer's reward is drawn from a normal distribution (illustrative). The bar shows the average reward of the kept answer, the cost line the number of generations per prompt.

Illustrative scale.
โš ๏ธ Reward hacking A reward model is an imperfect copy of human judgment. The harder you optimise against it, the more likely the policy finds answers that score well but are not actually better, such as longer, more confident or more flattering responses. Best-of-N is a gentle form of optimisation, which is one reason it is popular; the reward model is also refreshed every round.

5. Supervised fine-tuning and the data mix

SFT trains the model on (prompt, ideal response) pairs with the ordinary cross-entropy loss, but only on the response tokens: the prompt tokens are masked so the model is not trained to predict its own instructions. For the largest models Meta used a learning rate of 1e-5 for about 8.5Kโ€“9K steps. The data combines rejection-sampled answers, human-annotated data and synthetic data targeting specific capabilities.

What the SFT set is made of

Share of examples by category, from Table 7 of the paper. Switch to see average turns where reported.

Source: Llama 3 paper, Table 7 (as quoted). Shares are of examples, not of tokens; a long-context example contains far more tokens than a chat turn.

Two things stand out. General English chat is more than half of the examples, so the bulk of SFT teaches the general helpful-assistant style. And "long context" is only 0.11% of examples: Meta reported that a small share of long-context data was enough to keep long-context ability while mixing it in with short data, without hurting short-context benchmarks.

6. DPO instead of PPO

The classic recipe, made famous by InstructGPT and ChatGPT, is RLHF with PPO: train a reward model, then run an online reinforcement-learning loop where the model generates answers, the reward model scores them, and PPO updates the model while a KL penalty keeps it close to the SFT starting point. That loop needs four models in memory (policy, reference, reward and value) and a generate-then-train cycle at every step.

Direct Preference Optimization (Rafailov et al., 2023) gets a similar effect from a simple offline loss on preference pairs. It raises the log-probability of the chosen response relative to the rejected one, measured against a frozen reference model. Meta reports that DPO needed less compute for large models and performed better, especially on instruction-following benchmarks such as IFEval. Its settings: learning rate 1e-5, ฮฒ = 0.1. Two changes made it stable:

The DPO loss, one pair at a time

Margin = how much more the model (relative to the reference) prefers the chosen answer over the rejected one, in log-probability. The loss is โˆ’log ฯƒ(ฮฒ ร— margin).

Llama 3 used ฮฒ = 0.1.
RLHF with PPODPO (Llama 3)
DataFresh samples from the policy at each stepFixed set of preference pairs per round
Models in memoryPolicy, reference, reward, valuePolicy, reference (log-probs can be precomputed)
LoopGenerate โ†’ score โ†’ update, onlineOrdinary supervised-style training
Main riskInstability, reward hacking, many hyperparametersOverfitting to the pairs; chosen-response probability can fall
UsersInstructGPT, early ChatGPT, Llama 2Llama 3, Tulu 3 (with other steps), many open models

7. Data for specific capabilities

General chat preferences do not teach everything. The paper describes dedicated data work for a list of capabilities. The tab text below summarises what each capability needs in general; Meta's exact pipelines are described in section 4.3 of the paper and not reproduced here in detail โš ๏ธ not verified in our sources.

Capability by capability

8. Model averaging

At each of the reward-model, SFT and DPO stages, Meta trained several variants with different data or hyperparameters and averaged their weights. Models fine-tuned from the same starting point tend to sit in the same low-loss basin, so their average is usually at least as good as the best member and more robust. It is the post-training cousin of the Polyak averaging used at the end of pre-training in Part 5.

Averaging three fine-tuned variants

Illustrative scores. Toggle variants into the average.

9. Safety and Llama Guard 3

Safety tuning runs inside the same loop: adversarial and borderline prompts get their own preference and SFT data, so the model learns both to refuse harmful requests and to avoid refusing harmless ones that only look risky. Meta also released a separate safeguard model with Llama 3.1 on 23 July 2024: Llama Guard 3, an 8B model fine-tuned from Llama 3.1 that classifies prompts and responses as safe or unsafe across 14 hazard categories based on the MLCommons taxonomy. It is a tool for developers and runs alongside the main model, not inside it.

A guarded deployment

Click a stage.

10. What changed since

Llama 3's recipe is a snapshot of mid-2024. Post-training has moved quickly since, mostly toward reinforcement learning on tasks with checkable answers.

2022 ยท InstructGPT. SFT, reward model, PPO. The template for ChatGPT.
2023 ยท Llama 2. RLHF with rejection sampling and PPO over several rounds.
2023 ยท DPO. Preference optimisation without an RL loop.
Jul 2024 ยท Llama 3. Rounds of reward model โ†’ rejection sampling โ†’ SFT โ†’ DPO, plus model averaging.
Nov 2024 ยท Tulu 3 (AI2). Open recipe: SFT, then DPO, then RL with verifiable rewards (RLVR) on maths and instruction-following checks.
Jan 2025 ยท DeepSeek-R1. Large-scale RL with rule-based rewards (GRPO) produces long chains of reasoning, bringing online RL back to the centre of post-training.

The infrastructure changed with it. DPO training looks like ordinary fine-tuning, but online RL needs a generation fleet (typically vLLM or SGLang) and a training job working in lockstep, coordinated by frameworks such as TRL, OpenRLHF or verl. Post-training compute has grown from a rounding error next to pre-training to a significant share of a frontier lab's budget.

๐Ÿ—‚๏ธ Llama 3 case file: post-training

Input
Llama 3 405B base model (Part 5)
Output
Llama 3.1 405B Instruct, released 23 July 2024
Loop
Reward model โ†’ rejection sampling โ†’ SFT โ†’ DPO, repeated (6 rounds โš ๏ธ secondary)
Preferences
4 strength levels; edited > chosen > rejected; per-turn for multi-turn
SFT
LR 1e-5, ~8.5Kโ€“9K steps, prompt tokens masked; 52.66% general English
DPO
LR 1e-5, ฮฒ 0.1, formatting tokens masked, NLL term (0.2 โš ๏ธ secondary)
Extra
Model averaging at each stage; capability-specific data; Llama Guard 3 8B released alongside
Verdict Llama 3's post-training is deliberately conservative: it avoids the instability of online RL and leans on data. Most of the work is in the inputs (fresh human preferences every round, a reward model to filter samples, targeted synthetic data) and the training itself is close to ordinary fine-tuning. That made it repeatable at 405B scale. The field has since added RL on verifiable tasks on top, but the round structure, preference data and averaging remain common building blocks.

๐Ÿ‘€ What to watch

๐Ÿ”ข Reading the numbers

๐Ÿ“š References