The Maestro, Season 2 ยท Part 8 ยท Phase 6 ยท ๐๏ธ Research director hat
Environments as the Asset
Part 7 chose the next moat: verified environments at scale. An environment is a real task packaged so a model can practise it and an automatic check can say whether it succeeded. This part shows how the factory works, why half of all environments are thrown away, why the library decays unless someone maintains it, and how it compounds into a lead that is hard to copy.
๐ฌ Research director๐ก๏ธ Safety๐ป Engineer
โ ๏ธ How to read this (as of 3 Nov 2026)
env_flywheel.py and env_quality.py were run (real output below). Build rates, pass rates, drift, the gain constant and the candidate environments are the story's scenario. See also Agent Layer, Part 8 for how RL environments work in industry. Disclosure: Claude, made by Anthropic, helps write this site.
Mission log: months 32โ36
The map: the environment factory
Half are thrown away, on purpose
The library compounds, if it is maintained
Sandbox: run the factory
Five mistakes
Delegation box
Budget and risk tracker
The series
Reading the numbers
1. Mission log: months 32โ36
Week 128 ยท The factory
- Semi-automatic environment building starts at 250 a month.
Week 130 ยท The probes
- Three cheating agents run against every new environment (
env_quality.py).
Week 144 ยท The library
- About 1,900 maintained environments; +5 points on held-out tasks (plan).
2. The map: the environment factory
๐ธ๏ธ From a customer task to a compounding asset
Cheaters in red, guards in green, data in teal. The loop closes at the bottom: better models win more tasks.
- Three things to notice.
- Every environment must beat three cheaters before it is kept.
- Drift erodes the library, so maintenance is part of the moat.
- The loop is the flywheel from Season 1, now feeding the model with practice, not only skills.
3. Half are thrown away, on purpose
env_quality.py ยท Python 3, no dependencies; scenario candidates
"""env_quality.py โ reject environments whose checks can be passed without
doing the task (the 'leaky check' problem from Season 1, Part 11).
Each candidate environment is probed with three cheating agents: one that
does nothing, one that deletes the tests, one that hard-codes expected output.
An environment is kept only if every cheater fails and a reference solution
passes. Candidates and probe results are SCENARIO data.
"""
CANDS = { # env: (noop passes, delete-tests passes, hardcode passes, reference passes)
"fix-null-pointer-svc-a": (False, False, False, True),
"migrate-orm-v3": (False, True, False, True),
"add-retry-to-client": (False, False, False, True),
"format-dates-util": (False, False, True, True),
"refactor-auth-module": (False, False, False, False),
"bump-dependency-x": (True, False, False, True),
"paginate-api-endpoint": (False, False, False, True),
"cache-config-loader": (False, False, False, True),
}
def main():
kept = 0
for env, (noop, delete, hard, ref) in CANDS.items():
why = [n for n, bad in (("noop passes", noop), ("deleting tests passes", delete), ("hard-coding passes", hard), ("reference fails", not ref)) if bad]
kept += not why
print(f"{env:26} {'KEEP' if not why else 'REJECT: ' + ', '.join(why)}")
print(f"kept {kept} of {len(CANDS)} ({kept / len(CANDS):.0%})")
if __name__ == "__main__":
main()$ python3 env_quality.py # three cheaters and a reference per environment
fix-null-pointer-svc-a KEEP
migrate-orm-v3 REJECT: deleting tests passes
add-retry-to-client KEEP
format-dates-util REJECT: hard-coding passes
refactor-auth-module REJECT: reference fails
bump-dependency-x REJECT: noop passes
paginate-api-endpoint KEEP
cache-config-loader KEEP
kept 4 of 8 (50%)
Why candidate environments were rejected
Real output of env_quality.py.
๐ Go deeper: why a leaky environment is worse than none
- A model trained on a check it can game learns to game checks.
- Season 1, Part 11's toy showed it: with a weak verifier, 98% of "successes" were hacks.
- So the probes run again whenever the repository behind an environment changes.
4. The library compounds, if it is maintained
env_flywheel.py ยท Python 3, no dependencies; scenario constants
"""env_flywheel.py โ environments as a compounding asset.
Each month the team builds new environments; a share fails the quality check
(leaky or trivial checks) and is discarded; kept environments lose 2% a month
to drift (APIs change, tasks go stale). The model's gain on held-out tasks
grows with the square root of good environments (diminishing returns).
Build rates, pass rates and the gain constant are SCENARIO assumptions.
"""
def run(build_per_month, pass_rate, months=12, decay=0.02, k=0.12):
good, out = 0.0, []
for m in range(1, months + 1):
good = good * (1 - decay) + build_per_month * pass_rate
out.append((m, good, k * good ** 0.5))
return out
def main():
for label, b, q in (("plan: 250/month, 70% pass", 250, 0.70), ("rushed: 400/month, 35% pass", 400, 0.35), ("slow: 120/month, 85% pass", 120, 0.85)):
r = run(b, q)
print(f"{label:30} month 6: {r[5][1]:6.0f} envs, +{r[5][2]:.1f} pts month 12: {r[11][1]:6.0f} envs, +{r[11][2]:.1f} pts")
if __name__ == "__main__":
main()$ python3 env_flywheel.py # three build strategies over 12 months
plan: 250/month, 70% pass month 6: 999 envs, +3.8 pts month 12: 1884 envs, +5.2 pts
rushed: 400/month, 35% pass month 6: 799 envs, +3.4 pts month 12: 1507 envs, +4.7 pts
slow: 120/month, 85% pass month 6: 582 envs, +2.9 pts month 12: 1098 envs, +4.0 pts
Gain on held-out tasks after 12 months
Real output. Building more of worse environments loses to building fewer, better ones.
- Why it is hard to copy.
- A rival can hire engineers, but not the customer tasks behind the environments or the months of probing and maintenance (Part 7: 18 months to copy).
5. Sandbox: run the factory
๐ฎ Sandbox: run the environment factory
Gain on held-out tasks (points) over 24 months; same model as env_flywheel.py, scenario constants.
- Try this.
- Set drift to 6%: the library levels off near 2,900 environments however long you build.
- Set the pass rate to 20% and build 600 a month: lots of work, little gain.
- Notice the curve keeps rising but flattens: diminishing returns.
6. Five mistakes
- Counting environments instead of good environments.
- Skipping the cheat probes to hit a monthly target.
- Never re-checking environments when code changes.
- Using customer tasks without the data agreement allowing it.
- Training on the same environments used for evaluation.
7. Delegation box
๐ Gopala himself
- The quality bar and data-agreement rules
๐ค AI agents
- Drafting environments and running probes
- Drift detection
๐ค Bought, rented or borrowed
- Annotation partners for reference solutions
- Sandbox infrastructure (Agent Layer, Part 2)
8. Budget and risk tracker
| Item | Status after Part 8 | Note |
|---|---|---|
| Environments | ~1,900 by month 12 (plan) | From env_flywheel.py |
| Quality | 50% rejected in the sample | From env_quality.py |
| Moat | Growing | +5.2 points planned |
| Maintenance | Watch | 2% drift a month |
9. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
VerdictEnvironments are the next moat because they turn customer work into practice the model can learn from, and because each one must survive three cheaters and constant maintenance. Quantity without quality loses. Next: when research says no.
10. Reading the numbers
- Primary / standard
- Reward hacking and leaky checks are well documented in RL; see the Agent Layer and Evaluation Layer series.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- All rates, constants, candidates and results are the story's scenario.