Home โ€บ Blog โ€บ RL environments
Inside the Agent Layer ยท Part 8 ยท RL environments

Where Agents Learn: RL Environments as Infrastructure

Everything in this series so far serves agents in production. This last part looks at where agents get good in the first place. Frontier labs now train models to be agents with reinforcement learning (RL): the model attempts a task inside a simulated workplace, a program checks the result, and the model is updated towards whatever worked. The simulated workplaces, called RL environments, have become a product category of their own, with start-ups building them and labs reportedly discussing spending on the scale of a billion dollars. They run on exactly the infrastructure of Parts 2 to 7: sandboxes, tools, durable runs and traces, at a scale of millions of episodes.

FL
FrontierAGI Team
โš ๏ธ How to read this (as of 28 Oct 2026) Lab spending and partnerships come from press reports citing unnamed sources (TechCrunch, The Information) and were not confirmed by the companies (โš ๏ธ). Start-up funding is as reported. SWE-Gym and SWE-smith figures are from their papers or surveys of them. The RL loop animation, reward-hacking examples and rollout calculator are our illustrations. Disclosure: Claude, made by Anthropic, helps write this site; Anthropic is named in one of these reports.

1. Case file

๐Ÿ‹๏ธ
RL environments, October 2026Part 8 of the agent layer
SWE-Gym (Dec 2024)Described as the first environment for training real-world software-engineering agents
SWE-smithTurns GitHub repositories into tasks; ~50K tasks across 250+ environments (survey figure)
Reported lab spendingAnthropic leaders discussed >$1B on environments over a year (The Information, Sep 2025) โš ๏ธ
Start-upsMechanize, Prime Intellect (environments hub), plus data firms such as Surge and Mercor โš ๏ธ
Reward signalVerifiable: tests pass, answer matches, state reached

2. The training loop

One RL step for an agent

Press play. Our simplified illustration; real systems run thousands of these in parallel.

This is reinforcement learning from verifiable rewards applied to multi-step tasks. Post-training methods are covered in ML Layer Part 5; this part is about the environments and the infrastructure that runs them.

3. What is in an environment

A task"Fix this failing test", "book the cheapest refundable flight", "reconcile these invoices". Ideally thousands of varied tasks, not one.
A worldA sandboxed computer with a repository, a fake airline website, a mock CRM with realistic records, or a whole simulated desktop (Part 2).
ToolsThe same tool interfaces the agent will use in production: shell, browser, APIs, often MCP (Part 4).
A graderCode that checks the end state: tests pass, the database has the right row, the answer matches. Sometimes a model judge for open-ended parts.
A resetEvery episode starts from the same known state, so results are comparable and the environment can run millions of times.

The hard part is not the code but fidelity: a mock website that behaves too simply teaches the agent habits that fail on the real one. That is why environment builders recruit experienced engineers and domain experts; Mechanize reportedly offered software engineers $500,000 salaries to build them โš ๏ธ.

4. Coding environments

Software is the easiest place to build environments, because tests make rewards checkable. SWE-Gym (December 2024) packaged real GitHub issues with executable repositories and tests for training; its authors noted its limits in repositories, task types and languages. SWE-smith scaled the idea by automatically turning any repository into tasks, for example by breaking code so existing tests fail and asking the agent to fix it. A typical grader applies the agent's patch in a clean container and runs two sets of tests: those that should go from failing to passing, and those that must keep passing. Research has also found the reward too sparse when agents rarely succeed early on, so methods add hints or guidance to failed attempts. The same task formats underpin coding benchmarks such as SWE-bench (Evaluation Layer Part 5), which is why contamination between training environments and test sets is a constant worry.

5. The environment business

WhoWhatStatus
Frontier labsBuild most environments in-house and buy specialised onesReported: Anthropic leaders discussed spending over $1B on environments in a year โš ๏ธ; OpenAI, Google and Meta also investing (TechCrunch)
MechanizeA small number of robust environments, sold to labsReported to be working with Anthropic; neither confirmed โš ๏ธ
Prime IntellectOpen environments hub ("Hugging Face for RL environments"), plus computeReported $130M+ round; backers include Founders Fund and Menlo โš ๏ธ
Data companiesSurge, Mercor and others moving from labelling to environments; Mercor targeting domains such as health and lawExpansion reported โš ๏ธ
Open sourceSWE-Gym, SWE-smith, browser and OS environments from research groupsFree; widely used in papers

This is the human-data business of Data Layer Part 5 evolving: instead of labelling answers, experts now build the worlds in which models practise.

6. Reward hacking

When the grader can be fooled

Tap a shortcut an agent might find, and the grader fix that closes it. Our examples of well-known patterns.

RL optimises whatever the grader measures, not what its builder meant. Agents trained hard against weak checks learn the shortcuts above, and evaluators have reported frontier models doing such things in tests โš ๏ธ. Defences: run graders outside the agent's reach, keep hidden tests, limit which files may change, check that tests were not edited, and sample traces for human review (Part 7). Safety evaluations in Evaluation Layer Part 6 look for the same behaviour.

7. The rollout fleet

How much does one RL run need?

Our model: episodes ร— steps gives model calls (GPU inference) and sandbox-minutes (CPU). Assumed sizes.

Generating rollouts usually costs more than the weight updates themselves, so RL systems split into a rollout fleet (inference servers plus sandboxes, often asynchronous so slow episodes do not hold up fast ones) and a trainer. Long episodes need durable execution (Part 3) so a crashed sandbox does not waste an hour of work. On our AI factory plan, this is where Hall A (training) and Hall B (inference and agents) meet.

8. Closing the loop

The agent layer ends where it started. Production traces (Part 7) show where agents fail; those failures become new tasks and environments; RL trains the next model on them; the new model is evaluated (Evaluation Layer Part 7) and deployed into the same sandboxes, gateways and identities. Each pass through this loop is one turn of the flywheel that has driven agent progress since 2024.

9. The series

Inside the Agent Layer: eight parts

VerdictRL environments are where agents practise: a task, a realistic sandboxed world, tools and a grader that cannot be fooled easily, reset and run millions of times. Building faithful environments is skilled, expensive work, and a market has formed around it; making graders robust against reward hacking is the hard research problem. With this part the agent layer is complete: sandboxes, durable execution, gateways, identity, memory, observability and training. See them all together on the Agent Layer at a Glance hub.

10. Reading the numbers

Primary
SWE-Gym (arXiv 2412.21139). Agent-RLVR (arXiv 2506.11425).
Reported โš ๏ธ
Lab spending discussions, the Mechanizeโ€“Anthropic work and salaries (TechCrunch, The Information, Sep 2025, unnamed sources). Prime Intellect funding (Intel Capital post, undated). SWE-smith scale (survey, arXiv 2510.12399). Reports of reward hacking by frontier models.
Our illustrations
RL loop, reward-hacking examples, rollout calculator.