After Pre-training: The Software That Turns a Base Model into a Reasoning Assistant
Pre-training makes a model that predicts text. Post-training makes it follow instructions, prefer good answers and, since 2024, reason through hard problems with reinforcement learning. That last step is a systems puzzle: every training step first runs an inference job to generate answers, then scores them, then runs a training job, and the two halves want very different hardware layouts. This part explains the algorithms, the architectures and the open stacks (TRL, verl, OpenRLHF, NeMo RL, slime, AReaL, torchforge) that labs use to do it.
1. Case file
2. Timeline
Post-training, 2022 โ 2026
Tap a milestone. Colour: algorithms, open stacks, models, environments.
3. The post-training stages
Post-training is a sequence of stages, each with its own data and software needs. Producing Intelligence Part 6 follows Llama 3 through them; here the focus is on the tools.
From base model to agent
Tap a stage. Compute shares are qualitative: they vary hugely by lab and have shifted toward RL since 2024.
4. Why RL is a systems problem
Supervised fine-tuning looks like pre-training with different data. Reinforcement learning does not. Each step must generate many answers from the current model (an inference workload, best served by an engine such as vLLM or SGLang), score them (run tests, check maths, or call a reward model), compute advantages (which answers were better than expected) and finally update the weights (a training workload using FSDP or Megatron, Part 3). Then the new weights must be copied back to the inference engine before the next round.
One RL step
Step through, then compare where time goes. Time shares are illustrative: generation is usually the largest share and grows with answer length.
Two facts make this hard. First, generation is autoregressive and memory-bound (Part 4's roofline), so it runs at a fraction of the GPUs' compute peak, while training is compute-bound. Second, answers vary wildly in length: in a synchronous step every GPU waits for the longest answer in the batch. The HybridFlow paper measured one older baseline, NeMo-Aligner (which lacked a KV cache in generation), spending up to 81.2% of each RLHF iteration on generation, which shows why the rollout engine matters so much.
5. Algorithms and the models they need
How many models must be in memory?
Tap an algorithm. Memory assumes a 70B policy in BF16 with Adam training state for trainable models (~16 bytes/parameter) and weights only (2 bytes/parameter) for frozen ones. Our arithmetic, illustrative.
- PPO (used for InstructGPT, 2022): an actor (the policy), a critic that predicts value, a reward model and a frozen reference model to keep the policy from drifting. Four models, two of them trained.
- DPO (May 2023): learns directly from pairs of preferred and rejected answers, offline, with only the policy and a reference. Cheap and stable; no generation during training.
- GRPO (DeepSeekMath, Feb 2024; used for DeepSeek-R1, Jan 2025): drops the critic by sampling a group of answers to each prompt and scoring each against the group average. Pairs naturally with verifiable rewards (right or wrong).
- Later variants: DAPO (ByteDance, 2025, built on verl), GSPO โ ๏ธ and others adjust clipping, sampling and length handling. The stacks race to support each new one within weeks.
6. GRPO calculator
Group-relative advantages
One prompt, eight sampled answers. Tap an answer to toggle correct (reward 1) or wrong (0). GRPO's advantage is (reward โ group mean) รท group standard deviation.
Notice what happens when all eight answers are right, or all wrong: every advantage is zero and the prompt teaches nothing. That is why RL pipelines filter prompts to ones the model sometimes solves (DAPO's "dynamic sampling" does this) and why curriculum matters. It is also why GRPO needs many samples per prompt, which multiplies the generation cost.
7. Three architectures
Where do generation and training run?
Eight GPUs over three RL steps. Our own simplified timelines. Tap a design.
8. The stacks
Almost every serious RL stack is now a glue layer: it borrows a training engine (FSDP2, Megatron-Core or DeepSpeed), a generation engine (vLLM or SGLang) and usually Ray to place them on GPUs, and adds the RL logic, weight synchronisation and data flow. The differences are in how they place the two engines, how stale they allow samples to be, and how easily you can plug in new algorithms, rewards and multi-turn environments.
9. Feature matrix
From project documentation, October 2026. โ built in ยท โ partial or experimental ยท โ not a focus. Tap a cell.
10. Rewards and environments
Reward hacking is the standing risk: the policy finds ways to score well without doing the task, such as special-casing tests or writing confident but wrong answers a judge likes. The defences are partly in the stack (sandboxes that cannot be tampered with, held-out checks, length penalties) and partly in evaluation (Part 6).
11. Agentic RL
The frontier in 2025โ26 moved from single answers to multi-turn episodes: the model calls tools, reads results, runs code and browses over dozens of turns before a reward arrives. Rollouts become long and wildly uneven, need sandboxes and containers, and make asynchronous designs close to mandatory. Standards for environments have appeared: Meta and Hugging Face launched OpenEnv in October 2025, a Gym-style spec (reset, step, close) for container-based environments with a shared hub, integrated with TRL; Prime Intellect runs a community Environments Hub โ ๏ธ; and PyTorch announced torchforge (Oct 2025) as a PyTorch-native agentic RL library supporting any degree of asynchrony.
12. Who uses what
| Organisation | RL / post-training stack | Source |
|---|---|---|
| ByteDance Seed | verl (HybridFlow); DAPO built on it | Papers, repository |
| Zhipu / Z.ai (GLM) | slime: Megatron training + SGLang rollout; GLM-4.5 onward | Project documentation |
| Ant Group | AReaL (asynchronous) | Paper (NeurIPS 2025) |
| NVIDIA | NeMo RL (DTensor or Megatron backends) | Documentation |
| DeepSeek | In-house (GRPO, R1) | Papers |
| Meta / PyTorch | torchforge, OpenEnv; in-house for Llama | PyTorch blog |
| Hugging Face, many researchers | TRL | Documentation |
| OpenAI, Anthropic, Google, xAI | Not disclosed | โ |
13. Choosing a stack
Which stack fits?
An editorial guide.
14. Reading the numbers
- Primary
- HybridFlow/verl: 1.53รโ20.57ร throughput over baselines; averages 3.67ร vs DeepSpeed-Chat, 3.25ร vs OpenRLHF, 12.52ร vs NeMo-Aligner; NeMo-Aligner generation up to 81.2% of iteration time (paper, EuroSys 2025). AReaL: up to 2.77ร vs synchronous systems (NeurIPS 2025; an earlier version said 2.57ร). torchforge announced 22 Oct 2025 (PyTorch blog). OpenEnv launched 23 Oct 2025 (Hugging Face). xAI: Grok 4 RL on 200K-GPU Colossus at pretraining scale, 6ร compute-efficiency gain (xAI announcement).
- Secondary โ ๏ธ
- RL spend roughly equal to pre-training for Grok 4 (commentary). slime powering later GLM versions (third-party documentation). Prime Intellect Environments Hub and GSPO details not checked against primary sources.
- Our illustrations
- Stage compute shares, RL step time shares, model-memory totals (16 bytes/param trained, 2 bytes/param frozen, 70B everywhere), GPU timelines and the chooser are simplified.
- Papers
- InstructGPT (2022)
- Direct Preference Optimization (2023)
- DeepSeekMath: GRPO (2024)
- DeepSeek-R1 (2025)
- HybridFlow / verl (EuroSys 2025)
- AReaL (NeurIPS 2025)
- DAPO (2025)
- Projects and announcements
- Hugging Face TRL
- verl ยท OpenRLHF ยท slime
- NVIDIA NeMo RL
- PyTorch: introducing torchforge
- Hugging Face: introducing OpenEnv
- xAI Grok 4 announcement (reproduced)
- Part 4: kernels ยท Part 3: parallelism