Home โ€บ Blog โ€บ The Maestro, Season 2 ยท Part 8
The Maestro, Season 2 ยท Part 8 ยท Phase 6 ยท ๐Ÿ—๏ธ Research director hat

Environments as the Asset

Part 7 chose the next moat: verified environments at scale. An environment is a real task packaged so a model can practise it and an automatic check can say whether it succeeded. This part shows how the factory works, why half of all environments are thrown away, why the library decays unless someone maintains it, and how it compounds into a lead that is hard to copy.

FL
FrontierAGI Team
๐Ÿ”ฌ Research director๐Ÿ›ก๏ธ Safety๐Ÿ’ป Engineer
โš ๏ธ How to read this (as of 3 Nov 2026) env_flywheel.py and env_quality.py were run (real output below). Build rates, pass rates, drift, the gain constant and the candidate environments are the story's scenario. See also Agent Layer, Part 8 for how RL environments work in industry. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: months 32โ€“36

Week 128 ยท The factory
  • Semi-automatic environment building starts at 250 a month.
Week 130 ยท The probes
  • Three cheating agents run against every new environment (env_quality.py).
Week 144 ยท The library
  • About 1,900 maintained environments; +5 points on held-out tasks (plan).

2. The map: the environment factory

๐Ÿ•ธ๏ธ From a customer task to a compounding asset

or click any node

Cheaters in red, guards in green, data in teal. The loop closes at the bottom: better models win more tasks.

3. Half are thrown away, on purpose

env_quality.py ยท Python 3, no dependencies; scenario candidates
"""env_quality.py โ€” reject environments whose checks can be passed without
doing the task (the 'leaky check' problem from Season 1, Part 11).

Each candidate environment is probed with three cheating agents: one that
does nothing, one that deletes the tests, one that hard-codes expected output.
An environment is kept only if every cheater fails and a reference solution
passes. Candidates and probe results are SCENARIO data.
"""
CANDS = {  # env: (noop passes, delete-tests passes, hardcode passes, reference passes)
    "fix-null-pointer-svc-a": (False, False, False, True),
    "migrate-orm-v3": (False, True, False, True),
    "add-retry-to-client": (False, False, False, True),
    "format-dates-util": (False, False, True, True),
    "refactor-auth-module": (False, False, False, False),
    "bump-dependency-x": (True, False, False, True),
    "paginate-api-endpoint": (False, False, False, True),
    "cache-config-loader": (False, False, False, True),
}

def main():
    kept = 0
    for env, (noop, delete, hard, ref) in CANDS.items():
        why = [n for n, bad in (("noop passes", noop), ("deleting tests passes", delete), ("hard-coding passes", hard), ("reference fails", not ref)) if bad]
        kept += not why
        print(f"{env:26} {'KEEP' if not why else 'REJECT: ' + ', '.join(why)}")
    print(f"kept {kept} of {len(CANDS)} ({kept / len(CANDS):.0%})")

if __name__ == "__main__":
    main()
$ python3 env_quality.py # three cheaters and a reference per environment fix-null-pointer-svc-a KEEP migrate-orm-v3 REJECT: deleting tests passes add-retry-to-client KEEP format-dates-util REJECT: hard-coding passes refactor-auth-module REJECT: reference fails bump-dependency-x REJECT: noop passes paginate-api-endpoint KEEP cache-config-loader KEEP kept 4 of 8 (50%)

Why candidate environments were rejected

Kept4 of 8
Deleting the tests passed1
Hard-coding passed1
Doing nothing passed1
Reference solution failed1

Real output of env_quality.py.

๐Ÿ”Ž Go deeper: why a leaky environment is worse than none
  • A model trained on a check it can game learns to game checks.
    • Season 1, Part 11's toy showed it: with a weak verifier, 98% of "successes" were hacks.
  • So the probes run again whenever the repository behind an environment changes.

    4. The library compounds, if it is maintained

    env_flywheel.py ยท Python 3, no dependencies; scenario constants
    """env_flywheel.py โ€” environments as a compounding asset.
    
    Each month the team builds new environments; a share fails the quality check
    (leaky or trivial checks) and is discarded; kept environments lose 2% a month
    to drift (APIs change, tasks go stale). The model's gain on held-out tasks
    grows with the square root of good environments (diminishing returns).
    Build rates, pass rates and the gain constant are SCENARIO assumptions.
    """
    def run(build_per_month, pass_rate, months=12, decay=0.02, k=0.12):
        good, out = 0.0, []
        for m in range(1, months + 1):
            good = good * (1 - decay) + build_per_month * pass_rate
            out.append((m, good, k * good ** 0.5))
        return out
    
    def main():
        for label, b, q in (("plan: 250/month, 70% pass", 250, 0.70), ("rushed: 400/month, 35% pass", 400, 0.35), ("slow: 120/month, 85% pass", 120, 0.85)):
            r = run(b, q)
            print(f"{label:30} month 6: {r[5][1]:6.0f} envs, +{r[5][2]:.1f} pts   month 12: {r[11][1]:6.0f} envs, +{r[11][2]:.1f} pts")
    
    if __name__ == "__main__":
        main()
    $ python3 env_flywheel.py # three build strategies over 12 months plan: 250/month, 70% pass month 6: 999 envs, +3.8 pts month 12: 1884 envs, +5.2 pts rushed: 400/month, 35% pass month 6: 799 envs, +3.4 pts month 12: 1507 envs, +4.7 pts slow: 120/month, 85% pass month 6: 582 envs, +2.9 pts month 12: 1098 envs, +4.0 pts

    Gain on held-out tasks after 12 months

    Plan: 250/month, 70% pass+5.2 pts ยท 1,884 envs
    Rushed: 400/month, 35% pass+4.7 ยท 1,507
    Slow: 120/month, 85% pass+4.0 ยท 1,098

    Real output. Building more of worse environments loses to building fewer, better ones.

    5. Sandbox: run the factory

    ๐ŸŽฎ Sandbox: run the environment factory

    Gain on held-out tasks (points) over 24 months; same model as env_flywheel.py, scenario constants.

    6. Five mistakes

    7. Delegation box

    ๐Ÿ™‹ Gopala himself
    • The quality bar and data-agreement rules
    ๐Ÿค– AI agents
    • Drafting environments and running probes
    • Drift detection
    ๐Ÿค Bought, rented or borrowed
    • Annotation partners for reference solutions
    • Sandbox infrastructure (Agent Layer, Part 2)

    8. Budget and risk tracker

    ItemStatus after Part 8Note
    Environments~1,900 by month 12 (plan)From env_flywheel.py
    Quality50% rejected in the sampleFrom env_quality.py
    MoatGrowing+5.2 points planned
    MaintenanceWatch2% drift a month

    9. The series

    The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)

    VerdictEnvironments are the next moat because they turn customer work into practice the model can learn from, and because each one must survive three cheaters and constant maintenance. Quantity without quality loses. Next: when research says no.

    10. Reading the numbers

    Primary / standard
    Reward hacking and leaky checks are well documented in RL; see the Agent Layer and Evaluation Layer series.
    Secondary โš ๏ธ
    None in this part.
    Our assumptions
    All rates, constants, candidates and results are the story's scenario.