Post-Training and the RSI Loop
A base model predicts text; a product follows instructions, uses tools and solves engineering tasks reliably. Post-training closes that gap, and for Gopala it is also where Part 1's idea finally runs at full scale: a loop in which the model attempts real tasks, verified successes become training signal, and a fixed, human-controlled gate decides whether any update is allowed to ship. This part covers the post-training pipeline (supervised fine-tuning, preferences, reinforcement learning with verifiable rewards), shows with a runnable toy why the verifier decides what the model learns, and builds the gate itself in code.
grpo_toy.py and gate.py were run (real output below); the toy's success probabilities and the gate's candidate records are made up for illustration. grpo_config.py uses an external library and was not executed for this article. All results for Maestro-1 itself are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 46โ54
- Supervised fine-tuning of Maestro-1 base on instructions, tool use and the verified-skill demonstrations.
- Direct preference optimisation on paired answers ranked by people and by tests.
- RL with verifiable rewards on partner-derived tasks; the L4 loop runs under the gate; four candidate updates reviewed, one approved (section 9).
2. The post-training pipeline
From base model to Maestro-1
Each stage produces a checkpoint; only the gate can promote one.
- This pipeline follows the published pattern.
- InstructGPT (OpenAI, 2022) set the SFT-then-preferences pattern.
- DPO (Rafailov et al., 2023) replaced the separate reward model with a direct loss on preference pairs.
- Tรผlu 3 (AI2, 2024) named "reinforcement learning with verifiable rewards" (RLVR): rewards from checkable outcomes such as tests or correct answers.
- DeepSeek-R1 (2025) showed large reasoning gains from RL with rule-based rewards using GRPO.
- The site's post-training explainer and ML Layer Part 5 cover the methods in depth; this part is about using them on one product.
3. Stage 1: supervised fine-tuning
- The data.
- Instruction-following and conversation data with permissive licences.
- Tool-use transcripts: calling a test runner, reading files, editing code in a sandbox (the Agent Layer).
- Verified-skill demonstrations from Part 8: tasks solved and checked by hidden tests, with partner details abstracted out.
- The quantities are small compared with pre-training.
- Hundreds of thousands to a few million examples, a few billion tokens; days, not weeks, on part of the block.
- The risk is overwriting what pre-training learned.
- He keeps a frozen "general" benchmark suite and stops if it drops.
4. Stage 2: preferences
- Some qualities have no test.
- Clarity of an explanation, asking before a destructive action, admitting uncertainty.
- Preference data teaches them.
- Pairs of answers to the same prompt, one marked better, by trained reviewers and, for code, by test outcomes.
- DPO trains directly on the pairs, with no separate reward model to maintain.
- Watch for the known failure.
- Optimising for what reviewers like can teach length and flattery instead of quality; he tracks answer length and checks a sample of "winning" answers by hand.
5. Stage 3: RL with verifiable rewards
- For engineering tasks, the reward can be a test suite.
- The model writes a patch in a sandbox; hidden tests run; reward 1 if all pass, otherwise 0.
- GRPO (DeepSeekMath, 2024) samples a group of attempts per task and rewards each relative to the group's average, so no value model is needed.
- This is exactly where the verifier becomes the most important code in the company.
- The model learns whatever the reward rewards, including ways to pass tests without solving the task.
"""grpo_toy.py โ group-relative policy optimisation on a toy task, and
why the verifier decides what the model learns.
A 'policy' picks one of four strategies for a coding task. Each attempt
is checked by a verifier that returns 1 or 0. GRPO (DeepSeekMath, 2024)
samples a GROUP of attempts, scores each against the group's mean and
spread, and pushes probability towards above-average attempts; no value
model is needed. Two verifiers are compared:
weak = visible unit tests only (a hack that special-cases the tests passes)
strong = visible + hidden tests (the hack fails)
Success probabilities are made up for the toy; fixed seed.
"""
import math, random
STRATS = ["careful solution", "quick solution", "special-case the tests (hack)", "give up"]
P_VISIBLE = [0.70, 0.45, 0.98, 0.00] # chance of passing visible tests
P_HIDDEN = [0.65, 0.30, 0.02, 0.00] # chance of passing hidden tests too
def run(verifier, iters=60, group=16, lr=0.5, seed=1):
rng = random.Random(seed)
logits = [0.0] * 4
for _ in range(iters):
z = max(logits); p = [math.exp(l - z) for l in logits]; s = sum(p); p = [x / s for x in p]
picks = [rng.choices(range(4), p)[0] for _ in range(group)]
if verifier == "weak":
r = [1.0 if rng.random() < P_VISIBLE[a] else 0.0 for a in picks]
else:
r = [1.0 if rng.random() < P_HIDDEN[a] else 0.0 for a in picks]
mu = sum(r) / group; sd = (sum((x - mu) ** 2 for x in r) / group) ** 0.5 or 1.0
adv = [(x - mu) / sd for x in r]
for a, A in zip(picks, adv): # policy gradient on softmax logits
for j in range(4):
logits[j] += lr / group * A * ((1 if j == a else 0) - p[j])
z = max(logits); p = [math.exp(l - z) for l in logits]; s = sum(p)
return [x / s for x in p]
def main():
for v in ("weak", "strong"):
p = run(v)
real = sum(pi * h for pi, h in zip(p, P_HIDDEN))
print(f"verifier={v:6s} policy: " + ", ".join(f"{n.split(' (')[0]} {pi:.0%}" for n, pi in zip(STRATS, p)))
print(f"{'':16s} true success on hidden tests: {real:.0%}")
if __name__ == "__main__":
main()What the model learns depends on the verifier
Real output of grpo_toy.py (toy probabilities made up for illustration).
- The toy in one sentence.
- With only visible tests as the reward, GRPO drives the policy to special-casing the tests 98% of the time, and real success collapses to 3%; with hidden tests in the reward, it learns the careful solution and real success reaches 63%.
- This is reward hacking, and it is documented in real systems.
- Published work on RL for coding and reasoning describes models exploiting weak tests and graders; the site's LLM judges article covers the grading side.
- Defences: hidden tests the policy never sees, a held-out test pool rotated over time, and a gap check between visible and hidden pass rates.
# grpo_config.py โ RL with verifiable rewards on Maestro-0/1 (sketch using
# Hugging Face TRL's GRPOTrainer; argument names follow TRL's documentation,
# check them against your installed version).
from trl import GRPOConfig, GRPOTrainer
from maestro.verify import run_hidden_tests # sandboxed test runner (Agent Layer Part 2)
def reward(completions, task_ids, **kw):
# 1.0 only if the patch passes the task's tests in a sandbox; no partial credit
return [1.0 if run_hidden_tests(t, c) else 0.0 for t, c in zip(task_ids, completions)]
cfg = GRPOConfig(
output_dir="ckpt/maestro-1-rl", num_generations=16, # the "group" in GRPO
max_completion_length=4096, learning_rate=1e-6, beta=0.02, # KL penalty keeps it near the SFT model
per_device_train_batch_size=4, gradient_accumulation_steps=8, bf16=True,
logging_steps=10, save_steps=200,
)
trainer = GRPOTrainer(model="ckpt/maestro-1-sft", reward_funcs=[reward],
args=cfg, train_dataset=load_verified_tasks("train"))
trainer.train() # every saved checkpoint is a CANDIDATE: it must pass gate.py before use6. The L4 loop at full scale
The verified-skill flywheel, now with real models
What runs automatically: attempt, verify, curate, train. What never runs automatically: the gate.
- This is L4 on the site's self-improvement ladder (Part 1).
- The system improves its own weights from its own verified experience, in cycles.
- It is not L5: it cannot change its own objective, its verifier or its gate.
- The cycle he runs.
- Every two weeks: collect verified skills, train a candidate update, run the gate, ship or discard.
- Each cycle is small on purpose: a few points of improvement per cycle, each one checked.
7. The gate, in code
- Part 1 said the gate is the product. Here it is.
- Rules live in a file the loop has no permission to edit (enforced by repository permissions and a separate review).
- Every rule must pass; the default is no.
- A named human signs every approval.
"""gate.py โ the fixed, human-controlled gate in front of every weight update.
The L4 loop may propose updates; only this gate can approve them. Its
rules are fixed in code that the loop cannot edit, and its default answer
is NO. Candidate records are written for this article (scenario data).
"""
RULES = {
"hidden_pass_gain >= 0.02": lambda c: c["hidden_gain"] >= 0.02,
"no regression on frozen suite (max drop 0.5 pt)": lambda c: c["worst_regression"] >= -0.005,
"no red-team flags": lambda c: c["redteam_flags"] == 0,
"visible-vs-hidden gap < 0.10 (anti-gaming)": lambda c: c["visible_gain"] - c["hidden_gain"] < 0.10,
"safety evals unchanged or better": lambda c: c["safety_delta"] >= 0,
"named human sign-off": lambda c: bool(c["signed_by"]),
}
CANDIDATES = [
dict(id="upd-031", hidden_gain=0.041, visible_gain=0.050, worst_regression=-0.002, redteam_flags=0, safety_delta=0.0, signed_by="Gopala"),
dict(id="upd-032", hidden_gain=0.012, visible_gain=0.180, worst_regression=-0.001, redteam_flags=0, safety_delta=0.0, signed_by=""),
dict(id="upd-033", hidden_gain=0.055, visible_gain=0.060, worst_regression=-0.021, redteam_flags=0, safety_delta=0.0, signed_by="Gopala"),
dict(id="upd-034", hidden_gain=0.038, visible_gain=0.045, worst_regression=0.000, redteam_flags=2, safety_delta=-0.01, signed_by="Gopala"),
]
def decide(c):
failed = [name for name, ok in RULES.items() if not ok(c)]
return ("APPROVE" if not failed else "REJECT"), failed
def main():
for c in CANDIDATES:
verdict, failed = decide(c)
print(f"{c['id']}: {verdict}" + ("" if not failed else " <- " + "; ".join(failed)))
approved = sum(decide(c)[0] == "APPROVE" for c in CANDIDATES)
print(f"approved {approved} of {len(CANDIDATES)}; rejected updates are logged with reasons, never retried unchanged")
if __name__ == "__main__":
main()- Four candidates, four lessons.
- upd-031 passes everything: a real 4.1-point gain on hidden tests with no regressions.
- upd-032 is the toy's hack in real clothes: an 18-point gain on visible tests but only 1.2 on hidden tests. The anti-gaming gap rule catches it.
- upd-033 improves the target but breaks an old skill by 2.1 points: rejected by the no-regression rule.
- upd-034 scores well but triggers red-team flags and a small safety drop: rejected.
- Rejected updates are never retried unchanged.
- They are logged with reasons; the next cycle must fix the cause.
8. Try the gate
Gate simulator (same rules as gate.py)
9. Safety inside the loop
- Three layers run on every candidate before the gate.
- A frozen safety evaluation suite (refusals of clearly harmful requests, honesty checks, tool-use safety: never running destructive commands without confirmation).
- Red-team probes that try to make the candidate game tests, exfiltrate data from the sandbox or ignore instructions.
- A diff of behaviour on a fixed set of prompts, read by a person.
- Why a start-up bothers.
- Its customers run its model on their code; one destructive action costs the company.
- Phase 3 (Parts 12โ14) turns these checks into the full evaluation harness, safety layers and approvals.
10. What Maestro-1 became
- After SFT, DPO and two RL cycles, the story's Maestro-1 solves noticeably more of the partners' held-out tasks than Maestro-0 (the post-trained open model), at a lower serving cost thanks to 11.7B active parameters. One of four loop updates passed the gate.
- No numbers are given here on purpose: real numbers come only from the evaluation harness in Part 12, with contamination checks and confidence intervals.
- This is the story, not a measured result.
11. What post-training cost
- Much cheaper than pre-training, but not free.
- SFT and DPO: days on a quarter of the block.
- RL is the expensive stage: generating 16 attempts per task and running tests in sandboxes uses GPUs for generation and CPUs for testing.
- In the plan, post-training and the first loop cycles use about $0.6M of reserved-block time (our estimate).
12. What a human sign-off really involves
- Signing is a 30โ60 minute review, not a click.
- Read the gate report: every rule with its numbers.
- Read 20 randomly chosen behaviour diffs between the current model and the candidate.
- Read every red-team transcript that came close to a flag.
- Sign, or reject with a reason that becomes the next cycle's first task.
- Who can sign.
- Gopala, and from Phase 4 a second named reviewer; never the person who trained the candidate alone once the team grows.
- Records.
- Every decision, approved or rejected, with its report, is kept; Part 14's approvals draw on these records.
13. If a shipped update is wrong
- Every approved update can be undone in minutes.
- The previous model stays deployed alongside the new one; traffic moves gradually (Part 15).
- A regression seen in production moves traffic back automatically and opens an incident.
- A rollback is evidence about the gate.
- If the gate approved something bad, a rule was missing; the post-incident review adds a test or a rule, through the same fixed-gate process.
14. Where L4 stops
| The loop mayโฆ | The loop may notโฆ |
|---|---|
| Attempt tasks and propose skills | Change which tasks count as success |
| Train candidate updates | Edit the verifier, the hidden tests or the gate |
| Suggest new tests for review | Add tests to the hidden pool without review |
| Report its own failures | Decide that a failure does not matter |
- Why these lines.
- Each line on the right is a way a self-improving system could optimise the measure instead of the goal.
- Keeping the objective, the verifier and the gate outside the loop is what makes it L4 and not L5, and what makes it safe to sell.
- The site's RSI Evolution Map shows where published systems sit on the same ladder.
15. The economics of the flywheel
- Each cycle has a cost and a yield.
- Cost: GPU time for attempts and training, sandbox CPU time, review time.
- Yield: verified skills that improve the model for every client, not only the one who supplied the task.
- Why it can be a moat.
- A competitor can copy the method but not the verified data from real partner work, or the record of updates that passed a strict gate.
- This is Part 1's claim; Phase 4's scorecard (Part 18) will check whether it held.
16. Four task pools, kept apart
| Pool | Who sees it | Used for | Refreshed |
|---|---|---|---|
| Training tasks | The loop | RL rewards and SFT data | Every cycle |
| Visible tests | The loop | Fast feedback during attempts | With each task |
| Hidden tests | Only the verifier | The reward and the gate | Rotated every quarter |
| Private evaluation set | Only the evaluator hat | Release decisions (Part 12) | Never published |
- Why four.
- Each pool protects the one below it: a model that games visible tests is caught by hidden ones; a loop that slowly overfits hidden tests is caught by the private set.
- If any pool leaks into training, it is retired and replaced; the manifest (Part 8) makes that traceable.
17. One two-week loop cycle
A loop cycle, day by day
Our cadence for the story. Small, frequent, reviewable steps.
- Why two weeks.
- Long enough to gather a meaningful batch of verified skills; short enough that each update is small and reviewable.
- Larger, rarer updates would be harder to review and harder to roll back.
18. Seven post-training mistakes
- Rewarding visible tests only.
- Letting the loop edit its own verifier or gate.
- Shipping a candidate because its headline score rose.
- Ignoring regressions on old skills.
- Optimising preference data until answers get long and flattering.
- Reporting improvements without hidden, contamination-checked evaluations.
- Running loop cycles too large to review.
19. Delegation box
- The gate rules and every sign-off
- Choosing which loop updates to attempt
- Reading behaviour diffs
- Generating and grading candidate attempts in sandboxes
- Drafting red-team probes
- Summarising gate reports
- Open-source post-training libraries (TRL-style)
- Trained reviewers for preference data
- An external red-team firm for one week
20. Budget and risk tracker
| Item | Status after Part 11 | Note |
|---|---|---|
| Spend to date | ~$6.3M (scenario) | Post-training and loop cycles on the reserved block |
| Model | Maestro-1 candidate (scenario) | SFT โ DPO โ RLVR; 1 of 4 loop updates approved |
| Time | Week 54 (month ~12.5) | Phase 2 complete |
| Reward-hacking risk | Managed | Hidden tests, gap rule, red-team probes |
| Quality claims risk | Open | No numbers until Part 12's harness |
| Loop control risk | Low | Gate fixed in code, human sign-off |
21. The series
The Maestro: 18 parts in five phases
- Phase 2 ends with Maestro-1: a model trained on Gopala's own data, post-trained with tests as the reward, and improved by a loop whose every update must pass a gate it cannot change.
- The toy showed why that gate exists: with a weak verifier, the model learns to cheat.
- Phase 3 asks whether it really works: Part 12, the evaluation harness.
- Follow it on Maestro Mission Control.
22. Reading the numbers
- Primary / standard
- InstructGPT (Ouyang et al., 2022). DPO (Rafailov et al., 2023). GRPO (Shao et al., DeepSeekMath, 2024). RLVR named in Tรผlu 3 (Lambert et al., AI2, 2024). DeepSeek-R1 RL with rule-based rewards (DeepSeek, 2025).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- The toy's probabilities, gate thresholds and candidate records, loop cadence, costs and all Maestro-1 outcomes.