Home โ€บ Blog โ€บ Post-training & RL stacks
Inside the ML Layer ยท Part 5 ยท Post-training & RL

After Pre-training: The Software That Turns a Base Model into a Reasoning Assistant

Pre-training makes a model that predicts text. Post-training makes it follow instructions, prefer good answers and, since 2024, reason through hard problems with reinforcement learning. That last step is a systems puzzle: every training step first runs an inference job to generate answers, then scores them, then runs a training job, and the two halves want very different hardware layouts. This part explains the algorithms, the architectures and the open stacks (TRL, verl, OpenRLHF, NeMo RL, slime, AReaL, torchforge) that labs use to do it.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are the InstructGPT, DPO, DeepSeekMath (GRPO), DeepSeek-R1, HybridFlow (verl), AReaL and DAPO papers, project documentation for TRL, NeMo RL, slime and OpenRLHF, the PyTorch blog (torchforge), Hugging Face's OpenEnv announcement and xAI's Grok 4 announcement. How OpenAI, Anthropic and Google run RL is not public. Throughput figures are from each project's own benchmarks. The step animator, GPU timelines and calculator are our own illustrations. See Reading the numbers.

1. Case file

๐ŸŽฏ
The post-training layer, October 2026Layer 4 of 8 in the ML layer map
Main open RL stacksTRL ยท verl ยท OpenRLHF ยท NeMo RL ยท slime ยท AReaL ยท torchforge
verl (HybridFlow, EuroSys 2025)1.53ร—โ€“20.57ร— throughput vs baselines (paper)
AReaL (fully asynchronous)up to 2.77ร— vs synchronous systems (paper)
Grok 4 (xAI, Jul 2025)RL "at pretraining scale" on Colossus (xAI)
Typical rollout enginevLLM or SGLang
Environment standardsOpenEnv (Meta + Hugging Face, Oct 2025)

2. Timeline

Post-training, 2022 โ†’ 2026

Tap a milestone. Colour: algorithms, open stacks, models, environments.

Tap a milestone.

3. The post-training stages

Post-training is a sequence of stages, each with its own data and software needs. Producing Intelligence Part 6 follows Llama 3 through them; here the focus is on the tools.

From base model to agent

Tap a stage. Compute shares are qualitative: they vary hugely by lab and have shifted toward RL since 2024.

4. Why RL is a systems problem

Supervised fine-tuning looks like pre-training with different data. Reinforcement learning does not. Each step must generate many answers from the current model (an inference workload, best served by an engine such as vLLM or SGLang), score them (run tests, check maths, or call a reward model), compute advantages (which answers were better than expected) and finally update the weights (a training workload using FSDP or Megatron, Part 3). Then the new weights must be copied back to the inference engine before the next round.

One RL step

Step through, then compare where time goes. Time shares are illustrative: generation is usually the largest share and grows with answer length.

Two facts make this hard. First, generation is autoregressive and memory-bound (Part 4's roofline), so it runs at a fraction of the GPUs' compute peak, while training is compute-bound. Second, answers vary wildly in length: in a synchronous step every GPU waits for the longest answer in the batch. The HybridFlow paper measured one older baseline, NeMo-Aligner (which lacked a KV cache in generation), spending up to 81.2% of each RLHF iteration on generation, which shows why the rollout engine matters so much.

5. Algorithms and the models they need

How many models must be in memory?

Tap an algorithm. Memory assumes a 70B policy in BF16 with Adam training state for trainable models (~16 bytes/parameter) and weights only (2 bytes/parameter) for frozen ones. Our arithmetic, illustrative.

6. GRPO calculator

Group-relative advantages

One prompt, eight sampled answers. Tap an answer to toggle correct (reward 1) or wrong (0). GRPO's advantage is (reward โˆ’ group mean) รท group standard deviation.

Notice what happens when all eight answers are right, or all wrong: every advantage is zero and the prompt teaches nothing. That is why RL pipelines filter prompts to ones the model sometimes solves (DAPO's "dynamic sampling" does this) and why curriculum matters. It is also why GRPO needs many samples per prompt, which multiplies the generation cost.

7. Three architectures

Where do generation and training run?

Eight GPUs over three RL steps. Our own simplified timelines. Tap a design.

GenerateScoreTrainSync weightsIdle

8. The stacks

Almost every serious RL stack is now a glue layer: it borrows a training engine (FSDP2, Megatron-Core or DeepSpeed), a generation engine (vLLM or SGLang) and usually Ray to place them on GPUs, and adds the RL logic, weight synchronisation and data flow. The differences are in how they place the two engines, how stale they allow samples to be, and how easily you can plug in new algorithms, rewards and multi-turn environments.

9. Feature matrix

From project documentation, October 2026. โ— built in ยท โ— partial or experimental ยท โ€” not a focus. Tap a cell.

Tap a cell.

10. Rewards and environments

Verifiable rewards (RLVR)Check the answer: a maths result, unit tests for code, a proof checker. Cheap, hard to fool, and the engine of the 2025 reasoning boom. xAI says Grok 4 expanded verifiable data "well beyond math and coding".
Learned rewardsA reward model trained on human preferences, or an LLM judge with a rubric. Needed for open-ended tasks (writing, helpfulness) but easier to exploit.

Reward hacking is the standing risk: the policy finds ways to score well without doing the task, such as special-casing tests or writing confident but wrong answers a judge likes. The defences are partly in the stack (sandboxes that cannot be tampered with, held-out checks, length penalties) and partly in evaluation (Part 6).

11. Agentic RL

The frontier in 2025โ€“26 moved from single answers to multi-turn episodes: the model calls tools, reads results, runs code and browses over dozens of turns before a reward arrives. Rollouts become long and wildly uneven, need sandboxes and containers, and make asynchronous designs close to mandatory. Standards for environments have appeared: Meta and Hugging Face launched OpenEnv in October 2025, a Gym-style spec (reset, step, close) for container-based environments with a shared hub, integrated with TRL; Prime Intellect runs a community Environments Hub โš ๏ธ; and PyTorch announced torchforge (Oct 2025) as a PyTorch-native agentic RL library supporting any degree of asynchrony.

๐Ÿ’ก RL is becoming the expensive partIn July 2025 xAI said Grok 4 ran reinforcement learning "at pretraining scale" on its 200,000-GPU Colossus cluster, with over an order of magnitude more RL compute than before. Commentators read this as RL spend approaching pre-training spend โš ๏ธ; xAI has not published figures. If that trend holds, the RL stack is no longer a post-training afterthought but a second AI-factory workload as large as the first (see Case Study 1).

12. Who uses what

OrganisationRL / post-training stackSource
ByteDance Seedverl (HybridFlow); DAPO built on itPapers, repository
Zhipu / Z.ai (GLM)slime: Megatron training + SGLang rollout; GLM-4.5 onwardProject documentation
Ant GroupAReaL (asynchronous)Paper (NeurIPS 2025)
NVIDIANeMo RL (DTensor or Megatron backends)Documentation
DeepSeekIn-house (GRPO, R1)Papers
Meta / PyTorchtorchforge, OpenEnv; in-house for LlamaPyTorch blog
Hugging Face, many researchersTRLDocumentation
OpenAI, Anthropic, Google, xAINot disclosedโ€”

13. Choosing a stack

Which stack fits?

An editorial guide.

VerdictPost-training has become its own systems discipline. For fine-tuning and DPO, TRL or a fine-tuning tool is enough. For reasoning RL at scale, verl is the most widely used open stack, with slime and NeMo RL strong on Megatron-scale models and AReaL and torchforge pushing asynchronous and agentic designs. The open stacks are converging on the same ingredients (Ray, vLLM/SGLang, FSDP/Megatron); the lasting differences will be in asynchrony, environment support and how cheaply they turn GPU-hours into reward.

14. Reading the numbers

Primary
HybridFlow/verl: 1.53ร—โ€“20.57ร— throughput over baselines; averages 3.67ร— vs DeepSpeed-Chat, 3.25ร— vs OpenRLHF, 12.52ร— vs NeMo-Aligner; NeMo-Aligner generation up to 81.2% of iteration time (paper, EuroSys 2025). AReaL: up to 2.77ร— vs synchronous systems (NeurIPS 2025; an earlier version said 2.57ร—). torchforge announced 22 Oct 2025 (PyTorch blog). OpenEnv launched 23 Oct 2025 (Hugging Face). xAI: Grok 4 RL on 200K-GPU Colossus at pretraining scale, 6ร— compute-efficiency gain (xAI announcement).
Secondary โš ๏ธ
RL spend roughly equal to pre-training for Grok 4 (commentary). slime powering later GLM versions (third-party documentation). Prime Intellect Environments Hub and GSPO details not checked against primary sources.
Our illustrations
Stage compute shares, RL step time shares, model-memory totals (16 bytes/param trained, 2 bytes/param frozen, 70B everywhere), GPU timelines and the chooser are simplified.