Where Agents Learn: RL Environments as Infrastructure
Everything in this series so far serves agents in production. This last part looks at where agents get good in the first place. Frontier labs now train models to be agents with reinforcement learning (RL): the model attempts a task inside a simulated workplace, a program checks the result, and the model is updated towards whatever worked. The simulated workplaces, called RL environments, have become a product category of their own, with start-ups building them and labs reportedly discussing spending on the scale of a billion dollars. They run on exactly the infrastructure of Parts 2 to 7: sandboxes, tools, durable runs and traces, at a scale of millions of episodes.
1. Case file
2. The training loop
One RL step for an agent
Press play. Our simplified illustration; real systems run thousands of these in parallel.
This is reinforcement learning from verifiable rewards applied to multi-step tasks. Post-training methods are covered in ML Layer Part 5; this part is about the environments and the infrastructure that runs them.
3. What is in an environment
The hard part is not the code but fidelity: a mock website that behaves too simply teaches the agent habits that fail on the real one. That is why environment builders recruit experienced engineers and domain experts; Mechanize reportedly offered software engineers $500,000 salaries to build them โ ๏ธ.
4. Coding environments
Software is the easiest place to build environments, because tests make rewards checkable. SWE-Gym (December 2024) packaged real GitHub issues with executable repositories and tests for training; its authors noted its limits in repositories, task types and languages. SWE-smith scaled the idea by automatically turning any repository into tasks, for example by breaking code so existing tests fail and asking the agent to fix it. A typical grader applies the agent's patch in a clean container and runs two sets of tests: those that should go from failing to passing, and those that must keep passing. Research has also found the reward too sparse when agents rarely succeed early on, so methods add hints or guidance to failed attempts. The same task formats underpin coding benchmarks such as SWE-bench (Evaluation Layer Part 5), which is why contamination between training environments and test sets is a constant worry.
5. The environment business
| Who | What | Status |
|---|---|---|
| Frontier labs | Build most environments in-house and buy specialised ones | Reported: Anthropic leaders discussed spending over $1B on environments in a year โ ๏ธ; OpenAI, Google and Meta also investing (TechCrunch) |
| Mechanize | A small number of robust environments, sold to labs | Reported to be working with Anthropic; neither confirmed โ ๏ธ |
| Prime Intellect | Open environments hub ("Hugging Face for RL environments"), plus compute | Reported $130M+ round; backers include Founders Fund and Menlo โ ๏ธ |
| Data companies | Surge, Mercor and others moving from labelling to environments; Mercor targeting domains such as health and law | Expansion reported โ ๏ธ |
| Open source | SWE-Gym, SWE-smith, browser and OS environments from research groups | Free; widely used in papers |
This is the human-data business of Data Layer Part 5 evolving: instead of labelling answers, experts now build the worlds in which models practise.
6. Reward hacking
When the grader can be fooled
Tap a shortcut an agent might find, and the grader fix that closes it. Our examples of well-known patterns.
RL optimises whatever the grader measures, not what its builder meant. Agents trained hard against weak checks learn the shortcuts above, and evaluators have reported frontier models doing such things in tests โ ๏ธ. Defences: run graders outside the agent's reach, keep hidden tests, limit which files may change, check that tests were not edited, and sample traces for human review (Part 7). Safety evaluations in Evaluation Layer Part 6 look for the same behaviour.
7. The rollout fleet
How much does one RL run need?
Our model: episodes ร steps gives model calls (GPU inference) and sandbox-minutes (CPU). Assumed sizes.
Generating rollouts usually costs more than the weight updates themselves, so RL systems split into a rollout fleet (inference servers plus sandboxes, often asynchronous so slow episodes do not hold up fast ones) and a trainer. Long episodes need durable execution (Part 3) so a crashed sandbox does not waste an hour of work. On our AI factory plan, this is where Hall A (training) and Hall B (inference and agents) meet.
8. Closing the loop
The agent layer ends where it started. Production traces (Part 7) show where agents fail; those failures become new tasks and environments; RL trains the next model on them; the new model is evaluated (Evaluation Layer Part 7) and deployed into the same sandboxes, gateways and identities. Each pass through this loop is one turn of the flywheel that has driven agent progress since 2024.
9. The series
Inside the Agent Layer: eight parts
10. Reading the numbers
- Primary
- SWE-Gym (arXiv 2412.21139). Agent-RLVR (arXiv 2506.11425).
- Reported โ ๏ธ
- Lab spending discussions, the MechanizeโAnthropic work and salaries (TechCrunch, The Information, Sep 2025, unnamed sources). Prime Intellect funding (Intel Capital post, undated). SWE-smith scale (survey, arXiv 2510.12399). Reports of reward hacking by frontier models.
- Our illustrations
- RL loop, reward-hacking examples, rollout calculator.