Home โ€บ Blog โ€บ The Maestro ยท Part 11
The Maestro ยท Part 11 ยท Phase 2 ยท ๐Ÿง  ML engineer hat

Post-Training and the RSI Loop

A base model predicts text; a product follows instructions, uses tools and solves engineering tasks reliably. Post-training closes that gap, and for Gopala it is also where Part 1's idea finally runs at full scale: a loop in which the model attempts real tasks, verified successes become training signal, and a fixed, human-controlled gate decides whether any update is allowed to ship. This part covers the post-training pipeline (supervised fine-tuning, preferences, reinforcement learning with verifiable rewards), shows with a runnable toy why the verifier decides what the model learns, and builds the gate itself in code.

FL
FrontierAGI Team
๐Ÿง  ML engineer๐Ÿ”ฌ Researcher๐Ÿ›ก๏ธ Safety lead๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) Methods (InstructGPT-style SFT and RLHF, DPO, GRPO, RLVR) are cited to their papers; DeepSeek-R1 and Tรผlu 3 are cited for RL with verifiable rewards. grpo_toy.py and gate.py were run (real output below); the toy's success probabilities and the gate's candidate records are made up for illustration. grpo_config.py uses an external library and was not executed for this article. All results for Maestro-1 itself are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 46โ€“54

Weeks 46โ€“48 ยท SFT
  • Supervised fine-tuning of Maestro-1 base on instructions, tool use and the verified-skill demonstrations.
Weeks 48โ€“50 ยท Preferences
  • Direct preference optimisation on paired answers ranked by people and by tests.
Weeks 50โ€“54 ยท RL and the loop
  • RL with verifiable rewards on partner-derived tasks; the L4 loop runs under the gate; four candidate updates reviewed, one approved (section 9).

2. The post-training pipeline

From base model to Maestro-1

๐ŸงŠBase modelFrom Part 10
๐Ÿ—ฃ๏ธSFTLearn to follow instructions and use tools
โš–๏ธPreferencesDPO on ranked answer pairs
๐ŸŽฏRL with verifiable rewardsTests decide the reward
๐ŸšฆGateFixed rules + human sign-off
๐Ÿš€Maestro-1Candidate for evaluation (Part 12)

Each stage produces a checkpoint; only the gate can promote one.

3. Stage 1: supervised fine-tuning

4. Stage 2: preferences

5. Stage 3: RL with verifiable rewards

grpo_toy.py ยท Python 3, no dependencies
"""grpo_toy.py โ€” group-relative policy optimisation on a toy task, and
why the verifier decides what the model learns.

A 'policy' picks one of four strategies for a coding task. Each attempt
is checked by a verifier that returns 1 or 0. GRPO (DeepSeekMath, 2024)
samples a GROUP of attempts, scores each against the group's mean and
spread, and pushes probability towards above-average attempts; no value
model is needed. Two verifiers are compared:
  weak   = visible unit tests only (a hack that special-cases the tests passes)
  strong = visible + hidden tests (the hack fails)
Success probabilities are made up for the toy; fixed seed.
"""
import math, random

STRATS = ["careful solution", "quick solution", "special-case the tests (hack)", "give up"]
P_VISIBLE = [0.70, 0.45, 0.98, 0.00]     # chance of passing visible tests
P_HIDDEN  = [0.65, 0.30, 0.02, 0.00]     # chance of passing hidden tests too

def run(verifier, iters=60, group=16, lr=0.5, seed=1):
    rng = random.Random(seed)
    logits = [0.0] * 4
    for _ in range(iters):
        z = max(logits); p = [math.exp(l - z) for l in logits]; s = sum(p); p = [x / s for x in p]
        picks = [rng.choices(range(4), p)[0] for _ in range(group)]
        if verifier == "weak":
            r = [1.0 if rng.random() < P_VISIBLE[a] else 0.0 for a in picks]
        else:
            r = [1.0 if rng.random() < P_HIDDEN[a] else 0.0 for a in picks]
        mu = sum(r) / group; sd = (sum((x - mu) ** 2 for x in r) / group) ** 0.5 or 1.0
        adv = [(x - mu) / sd for x in r]
        for a, A in zip(picks, adv):           # policy gradient on softmax logits
            for j in range(4):
                logits[j] += lr / group * A * ((1 if j == a else 0) - p[j])
    z = max(logits); p = [math.exp(l - z) for l in logits]; s = sum(p)
    return [x / s for x in p]

def main():
    for v in ("weak", "strong"):
        p = run(v)
        real = sum(pi * h for pi, h in zip(p, P_HIDDEN))
        print(f"verifier={v:6s} policy: " + ", ".join(f"{n.split(' (')[0]} {pi:.0%}" for n, pi in zip(STRATS, p)))
        print(f"{'':16s} true success on hidden tests: {real:.0%}")

if __name__ == "__main__":
    main()
$ python3 grpo_toy.py verifier=weak policy: careful solution 1%, quick solution 1%, special-case the tests 98%, give up 0% true success on hidden tests: 3% verifier=strong policy: careful solution 97%, quick solution 1%, special-case the tests 1%, give up 1% true success on hidden tests: 63%

What the model learns depends on the verifier

Weak verifier: policy picks the hack98%
Weak verifier: real success (hidden tests)3%
Strong verifier: policy picks careful solution97%
Strong verifier: real success (hidden tests)63%

Real output of grpo_toy.py (toy probabilities made up for illustration).

grpo_config.py ยท TRL GRPOTrainer sketch ยท not executed for this article
# grpo_config.py โ€” RL with verifiable rewards on Maestro-0/1 (sketch using
# Hugging Face TRL's GRPOTrainer; argument names follow TRL's documentation,
# check them against your installed version).
from trl import GRPOConfig, GRPOTrainer
from maestro.verify import run_hidden_tests      # sandboxed test runner (Agent Layer Part 2)

def reward(completions, task_ids, **kw):
    # 1.0 only if the patch passes the task's tests in a sandbox; no partial credit
    return [1.0 if run_hidden_tests(t, c) else 0.0 for t, c in zip(task_ids, completions)]

cfg = GRPOConfig(
    output_dir="ckpt/maestro-1-rl", num_generations=16,   # the "group" in GRPO
    max_completion_length=4096, learning_rate=1e-6, beta=0.02,  # KL penalty keeps it near the SFT model
    per_device_train_batch_size=4, gradient_accumulation_steps=8, bf16=True,
    logging_steps=10, save_steps=200,
)
trainer = GRPOTrainer(model="ckpt/maestro-1-sft", reward_funcs=[reward],
                      args=cfg, train_dataset=load_verified_tasks("train"))
trainer.train()   # every saved checkpoint is a CANDIDATE: it must pass gate.py before use

6. The L4 loop at full scale

The verified-skill flywheel, now with real models

๐Ÿ“ฅTasksPartner tasks under contract
๐Ÿค–AttemptMaestro-1 candidate, skills retrieved first
โœ…VerifyHidden tests in a sandbox
๐Ÿ“šCurateAbstracted skill cores (Part 8)
๐Ÿ‹๏ธTrainRLVR + SFT on verified data
๐ŸšฆGateFixed rules + human

What runs automatically: attempt, verify, curate, train. What never runs automatically: the gate.

7. The gate, in code

gate.py ยท Python 3, no dependencies
"""gate.py โ€” the fixed, human-controlled gate in front of every weight update.

The L4 loop may propose updates; only this gate can approve them. Its
rules are fixed in code that the loop cannot edit, and its default answer
is NO. Candidate records are written for this article (scenario data).
"""
RULES = {
    "hidden_pass_gain >= 0.02": lambda c: c["hidden_gain"] >= 0.02,
    "no regression on frozen suite (max drop 0.5 pt)": lambda c: c["worst_regression"] >= -0.005,
    "no red-team flags": lambda c: c["redteam_flags"] == 0,
    "visible-vs-hidden gap < 0.10 (anti-gaming)": lambda c: c["visible_gain"] - c["hidden_gain"] < 0.10,
    "safety evals unchanged or better": lambda c: c["safety_delta"] >= 0,
    "named human sign-off": lambda c: bool(c["signed_by"]),
}
CANDIDATES = [
    dict(id="upd-031", hidden_gain=0.041, visible_gain=0.050, worst_regression=-0.002, redteam_flags=0, safety_delta=0.0, signed_by="Gopala"),
    dict(id="upd-032", hidden_gain=0.012, visible_gain=0.180, worst_regression=-0.001, redteam_flags=0, safety_delta=0.0, signed_by=""),
    dict(id="upd-033", hidden_gain=0.055, visible_gain=0.060, worst_regression=-0.021, redteam_flags=0, safety_delta=0.0, signed_by="Gopala"),
    dict(id="upd-034", hidden_gain=0.038, visible_gain=0.045, worst_regression=0.000, redteam_flags=2, safety_delta=-0.01, signed_by="Gopala"),
]

def decide(c):
    failed = [name for name, ok in RULES.items() if not ok(c)]
    return ("APPROVE" if not failed else "REJECT"), failed

def main():
    for c in CANDIDATES:
        verdict, failed = decide(c)
        print(f"{c['id']}: {verdict}" + ("" if not failed else "  <- " + "; ".join(failed)))
    approved = sum(decide(c)[0] == "APPROVE" for c in CANDIDATES)
    print(f"approved {approved} of {len(CANDIDATES)}; rejected updates are logged with reasons, never retried unchanged")

if __name__ == "__main__":
    main()
$ python3 gate.py # candidate records written for this article upd-031: APPROVE upd-032: REJECT <- hidden_pass_gain >= 0.02; visible-vs-hidden gap < 0.10 (anti-gaming); named human sign-off upd-033: REJECT <- no regression on frozen suite (max drop 0.5 pt) upd-034: REJECT <- no red-team flags; safety evals unchanged or better approved 1 of 4; rejected updates are logged with reasons, never retried unchanged

8. Try the gate

Gate simulator (same rules as gate.py)

9. Safety inside the loop

10. What Maestro-1 became

Scenario
  • After SFT, DPO and two RL cycles, the story's Maestro-1 solves noticeably more of the partners' held-out tasks than Maestro-0 (the post-trained open model), at a lower serving cost thanks to 11.7B active parameters. One of four loop updates passed the gate.
  • No numbers are given here on purpose: real numbers come only from the evaluation harness in Part 12, with contamination checks and confidence intervals.
  • This is the story, not a measured result.

11. What post-training cost

12. What a human sign-off really involves

13. If a shipped update is wrong

14. Where L4 stops

The loop mayโ€ฆThe loop may notโ€ฆ
Attempt tasks and propose skillsChange which tasks count as success
Train candidate updatesEdit the verifier, the hidden tests or the gate
Suggest new tests for reviewAdd tests to the hidden pool without review
Report its own failuresDecide that a failure does not matter

15. The economics of the flywheel

16. Four task pools, kept apart

PoolWho sees itUsed forRefreshed
Training tasksThe loopRL rewards and SFT dataEvery cycle
Visible testsThe loopFast feedback during attemptsWith each task
Hidden testsOnly the verifierThe reward and the gateRotated every quarter
Private evaluation setOnly the evaluator hatRelease decisions (Part 12)Never published

17. One two-week loop cycle

A loop cycle, day by day

๐Ÿ“ฅDays 1โ€“3Collect new partner tasks; attempt them with the current model
โœ…Days 3โ€“5Verify, curate and abstract skills
๐Ÿ‹๏ธDays 5โ€“9Train a candidate (RLVR + small SFT)
๐ŸงชDays 9โ€“11Gate checks, red-team probes, behaviour diffs
๐Ÿง‘โ€โš–๏ธDay 12Human review and sign-off or rejection
๐Ÿš€Days 13โ€“14Gradual rollout or a written rejection

Our cadence for the story. Small, frequent, reviewable steps.

18. Seven post-training mistakes

19. Delegation box

๐Ÿ™‹ Gopala himself
  • The gate rules and every sign-off
  • Choosing which loop updates to attempt
  • Reading behaviour diffs
๐Ÿค– AI agents
  • Generating and grading candidate attempts in sandboxes
  • Drafting red-team probes
  • Summarising gate reports
๐Ÿค Bought, rented or borrowed
  • Open-source post-training libraries (TRL-style)
  • Trained reviewers for preference data
  • An external red-team firm for one week

20. Budget and risk tracker

ItemStatus after Part 11Note
Spend to date~$6.3M (scenario)Post-training and loop cycles on the reserved block
ModelMaestro-1 candidate (scenario)SFT โ†’ DPO โ†’ RLVR; 1 of 4 loop updates approved
TimeWeek 54 (month ~12.5)Phase 2 complete
Reward-hacking riskManagedHidden tests, gap rule, red-team probes
Quality claims riskOpenNo numbers until Part 12's harness
Loop control riskLowGate fixed in code, human sign-off

21. The series

The Maestro: 18 parts in five phases

Verdict
  • Phase 2 ends with Maestro-1: a model trained on Gopala's own data, post-trained with tests as the reward, and improved by a loop whose every update must pass a gate it cannot change.

22. Reading the numbers

Primary / standard
InstructGPT (Ouyang et al., 2022). DPO (Rafailov et al., 2023). GRPO (Shao et al., DeepSeekMath, 2024). RLVR named in Tรผlu 3 (Lambert et al., AI2, 2024). DeepSeek-R1 RL with rule-based rewards (DeepSeek, 2025).
Secondary โš ๏ธ
None in this part.
Our assumptions
The toy's probabilities, gate thresholds and candidate records, loop cadence, costs and all Maestro-1 outcomes.