Post-Training: How Llama 3 405B Became an Assistant
At the end of Part 5, Meta had a 405-billion-parameter base model that could continue any text but could not hold a conversation, follow an instruction or say no. Turning it into Llama 3.1 405B Instruct took a much smaller amount of compute but a very different kind of engineering: human preference data, a reward model, rejection sampling, supervised fine-tuning and Direct Preference Optimization, run in repeated rounds. This part walks through that loop, the data that fed it and why Meta chose DPO over the PPO-based RLHF that made ChatGPT.
1. Base model vs assistant
Pre-training teaches a model to predict the next token of internet-scale text. That gives it knowledge and skill, but its default behaviour is to continue a document, not to answer the person writing it. Ask a base model a question and it may produce more questions, a forum thread, or a plausible but unhelpful paragraph. Post-training changes the model's default behaviour: answer helpfully, follow the format asked for, use tools when given them, and refuse what it should refuse.
The same prompt, before and after
Illustrative responses showing typical behaviour, not real outputs from Llama 3.
Compared with pre-training it is cheap. Pre-training of the 405B model consumed 30.84M H100-hours, while post-training runs over millions of examples, not trillions of tokens. Meta did not publish a GPU-hour figure for post-training in our sources. It is still heavy engineering: every round needs fresh human annotations, a newly trained reward model, and a large inference job that generates and scores candidate answers.
2. The round loop
Meta describes post-training as an iterative procedure. In each round it collects new human preference data, trains a reward model on it, uses that reward model to pick the best of several sampled answers (rejection sampling), fine-tunes on the result (SFT), and then aligns further with DPO. The best model from the round produces the samples for the next one, so data quality and model quality improve together. Course slides on the paper put the count at six rounds โ ๏ธ secondary.
One round of Llama 3 post-training
Click a step to see what happens there. Move the slider to step through rounds.
3. Preference data
Human annotators were shown a prompt and two responses, sampled from different models, and asked which they preferred and by how much, on four levels: significantly better, better, slightly better, or marginally better. They could also edit the chosen response to make it better, which gives three ranked responses for that prompt: edited > chosen > rejected. For multi-turn conversations, the comparison was made turn by turn.
An annotation, as a reward model sees it
Pick a preference strength. Summaries of the paper say pairs where the two responses were close were dropped from reward-model training โ ๏ธ secondary.
To reverse a list in Python use my_list[::-1], which returns a new list, or my_list.reverse(), which changes it in place.
You can reverse a list by writing a loop that walks backwards from the last index and appends each element to a new list.
4. The reward model and rejection sampling
The reward model starts from the pre-trained checkpoint and learns to score a (prompt, response) pair so that preferred responses score higher. It is never shown to users. Its job is to be a cheap stand-in for a human judge. Meta then uses it for rejection sampling: for each human-written prompt, generate several answers from the current best model, score them all, keep the highest-scoring one, and add it to the SFT training set.
That is the "best-of-N" idea, and it has diminishing returns: the expected best of N samples rises roughly with the square root of the logarithm of N, while generation cost rises linearly with N.
Best-of-N simulator
Each answer's reward is drawn from a normal distribution (illustrative). The bar shows the average reward of the kept answer, the cost line the number of generations per prompt.
5. Supervised fine-tuning and the data mix
SFT trains the model on (prompt, ideal response) pairs with the ordinary cross-entropy loss, but only on the response tokens: the prompt tokens are masked so the model is not trained to predict its own instructions. For the largest models Meta used a learning rate of 1e-5 for about 8.5Kโ9K steps. The data combines rejection-sampled answers, human-annotated data and synthetic data targeting specific capabilities.
What the SFT set is made of
Share of examples by category, from Table 7 of the paper. Switch to see average turns where reported.
Source: Llama 3 paper, Table 7 (as quoted). Shares are of examples, not of tokens; a long-context example contains far more tokens than a chat turn.
Two things stand out. General English chat is more than half of the examples, so the bulk of SFT teaches the general helpful-assistant style. And "long context" is only 0.11% of examples: Meta reported that a small share of long-context data was enough to keep long-context ability while mixing it in with short data, without hurting short-context benchmarks.
6. DPO instead of PPO
The classic recipe, made famous by InstructGPT and ChatGPT, is RLHF with PPO: train a reward model, then run an online reinforcement-learning loop where the model generates answers, the reward model scores them, and PPO updates the model while a KL penalty keeps it close to the SFT starting point. That loop needs four models in memory (policy, reference, reward and value) and a generate-then-train cycle at every step.
Direct Preference Optimization (Rafailov et al., 2023) gets a similar effect from a simple offline loss on preference pairs. It raises the log-probability of the chosen response relative to the rejected one, measured against a frozen reference model. Meta reports that DPO needed less compute for large models and performed better, especially on instruction-following benchmarks such as IFEval. Its settings: learning rate 1e-5, ฮฒ = 0.1. Two changes made it stable:
- Formatting tokens masked. Special header and termination tokens are excluded from the loss for both chosen and rejected responses. These tokens appear in both, so pushing one up and the other down creates a conflicting signal that can make the model repeat itself or stop abruptly.
- An added NLL term. A negative log-likelihood loss on the chosen responses keeps their probability from falling, which DPO alone can let happen. Course slides give a weight of 0.2 โ ๏ธ secondary.
The DPO loss, one pair at a time
Margin = how much more the model (relative to the reference) prefers the chosen answer over the rejected one, in log-probability. The loss is โlog ฯ(ฮฒ ร margin).
| RLHF with PPO | DPO (Llama 3) | |
|---|---|---|
| Data | Fresh samples from the policy at each step | Fixed set of preference pairs per round |
| Models in memory | Policy, reference, reward, value | Policy, reference (log-probs can be precomputed) |
| Loop | Generate โ score โ update, online | Ordinary supervised-style training |
| Main risk | Instability, reward hacking, many hyperparameters | Overfitting to the pairs; chosen-response probability can fall |
| Users | InstructGPT, early ChatGPT, Llama 2 | Llama 3, Tulu 3 (with other steps), many open models |
7. Data for specific capabilities
General chat preferences do not teach everything. The paper describes dedicated data work for a list of capabilities. The tab text below summarises what each capability needs in general; Meta's exact pipelines are described in section 4.3 of the paper and not reproduced here in detail โ ๏ธ not verified in our sources.
Capability by capability
8. Model averaging
At each of the reward-model, SFT and DPO stages, Meta trained several variants with different data or hyperparameters and averaged their weights. Models fine-tuned from the same starting point tend to sit in the same low-loss basin, so their average is usually at least as good as the best member and more robust. It is the post-training cousin of the Polyak averaging used at the end of pre-training in Part 5.
Averaging three fine-tuned variants
Illustrative scores. Toggle variants into the average.
9. Safety and Llama Guard 3
Safety tuning runs inside the same loop: adversarial and borderline prompts get their own preference and SFT data, so the model learns both to refuse harmful requests and to avoid refusing harmless ones that only look risky. Meta also released a separate safeguard model with Llama 3.1 on 23 July 2024: Llama Guard 3, an 8B model fine-tuned from Llama 3.1 that classifies prompts and responses as safe or unsafe across 14 hazard categories based on the MLCommons taxonomy. It is a tool for developers and runs alongside the main model, not inside it.
A guarded deployment
Click a stage.
10. What changed since
Llama 3's recipe is a snapshot of mid-2024. Post-training has moved quickly since, mostly toward reinforcement learning on tasks with checkable answers.
The infrastructure changed with it. DPO training looks like ordinary fine-tuning, but online RL needs a generation fleet (typically vLLM or SGLang) and a training job working in lockstep, coordinated by frameworks such as TRL, OpenRLHF or verl. Post-training compute has grown from a rounding error next to pre-training to a significant share of a frontier lab's budget.
๐๏ธ Llama 3 case file: post-training
- Input
- Llama 3 405B base model (Part 5)
- Output
- Llama 3.1 405B Instruct, released 23 July 2024
- Loop
- Reward model โ rejection sampling โ SFT โ DPO, repeated (6 rounds โ ๏ธ secondary)
- Preferences
- 4 strength levels; edited > chosen > rejected; per-turn for multi-turn
- SFT
- LR 1e-5, ~8.5Kโ9K steps, prompt tokens masked; 52.66% general English
- DPO
- LR 1e-5, ฮฒ 0.1, formatting tokens masked, NLL term (0.2 โ ๏ธ secondary)
- Extra
- Model averaging at each stage; capability-specific data; Llama Guard 3 8B released alongside
๐ What to watch
- How much of a frontier model's compute goes to post-training and RL compared with pre-training.
- Whether preference optimisation (DPO and variants) stays a standard step once RL with verifiable rewards matures.
- Synthetic data and AI feedback replacing human annotators for more capabilities.
- Next in this series: evaluation, the benchmarks and red-teaming that decide whether a model is ready.
๐ข Reading the numbers
- Loop structure, four preference levels, edited/chosen/rejected, SFT settings, DPO settings and masking, DPO vs PPO rationale, Table 7 shares and model averaging come from the Llama 3 paper as quoted in search results.
- Six rounds, the 0.2 NLL weight and dropping close pairs come from secondary summaries (course slides, blog explainers) and are marked โ ๏ธ.
- Llama Guard 3 facts come from Meta's model card.
- The base-vs-instruct responses, round slider, best-of-N simulator, DPO loss explorer and averaging scores are illustrations.
- Post-training GPU-hours, annotation volumes (Table 6) and synthetic-data pipeline details are not reproduced because we could not verify them.
๐ References
- Llama 3
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024), section 4, Table 7
- Meta: Llama Guard 3 model card
- JHU course slides on Llama 3 (secondary: rounds, NLL weight)
- Papers Explained 187b: Llama 3.1 (secondary)
- Oxen.ai: How Meta Trained Llama 3.1 (secondary)
- Methods and related work
- Rafailov et al.: "Direct Preference Optimization" (2023)
- Ouyang et al.: "Training language models to follow instructions with human feedback" (InstructGPT, 2022)
- Schulman et al.: "Proximal Policy Optimization Algorithms" (2017)
- Touvron et al.: "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023)
- Lambert et al.: "Tulu 3: Pushing Frontiers in Open Language Model Post-Training" (2024)
- DeepSeek-AI: "DeepSeek-R1" (2025)
- Wortsman et al.: "Model soups" (2022)
- Related on this site
- Part 5: Running the Run ยท Part 2: The Data Refinery ยท Part 1: The Software Stack