The Sim-to-Real Gap

A Policy That Works in Sim Isn't a Policy That Works

Robotics researchers learned this lesson the hard way, over more than a decade: a control policy that performs beautifully in a physics simulator can fail completely the moment it meets a real robot, because the simulator's approximations of friction, sensor noise, lighting, and material properties never quite match reality. The gap between simulated performance and real-world performance became a named, studied problem — the "reality gap" or "sim-to-real gap" — with its own dedicated mitigation techniques.

This series has spent four parts establishing that AI systems can now meaningfully accelerate AI research itself (Part 4), and asking whether that acceleration compounds safely (Parts 2, 3, 5). This part asks a more basic question underneath all of that: when an autonomous research agent is measured as "accelerating research," is that measurement happening on the equivalent of a physics simulator — a clean, well-specified benchmark — or on the messy real world the acceleration is supposed to matter in? The answer, backed by evidence from robotics, coding agents, and the research-automation benchmarks this series has already covered, is that the gap between the two is large, well-documented, and not obviously closing as fast as benchmark scores suggest.

Where the Gap Comes From, and How Robotics Tried to Close It

The clearest modern citation for the sim-to-real problem is Tobin, Fong, Ray, Schneider, Zaremba, and Abbeel's 2017 paper on domain randomization, which frames the reality gap directly: simulated training environments approximate the real world's appearance and physics, and that approximation error transfers into real-world failure. Their proposed fix — training on a wide, randomized distribution of simulated conditions (varying textures, lighting, camera angles, physics parameters) rather than one "realistic" simulation — forces a policy to learn features that generalize across the randomized range, which tends to include whatever the real world actually looks like. The term "reality gap" itself has deeper roots in evolutionary robotics dating back to the 1990s, but Tobin et al.'s paper is the one most commonly cited today as domain randomization's foundational demonstration.

The Robot Hand, Told the Way It Actually Happened

OpenAI's October 2019 Rubik's Cube-solving robot hand (Dactyl) is the most famous applied demonstration of domain randomization, and it's also a good case study in how a genuinely impressive result gets both oversold and later corrected in public understanding — a pattern this series flagged in Part 4's coverage of Sakana AI's benchmark-gaming retraction, and it's worth applying the same scrutiny here.

The Same Gap Shows Up in Coding Agents

SWE-bench Verified — a widely used coding-agent benchmark built from real GitHub issues — is now largely saturated, with frontier models scoring around 80%. But SWE-Bench Pro, a 2025 successor deliberately built to resist the kind of contamination and task-cleanliness that let models over-perform on the original, uses longer-horizon tasks drawn from a mix of open-source and proprietary enterprise codebases. Top model scores on SWE-Bench Pro drop to roughly the low 20s percent, with an even lower score on the split drawn specifically from commercial codebases.

A separate and more provocative data point: a 2025 randomized controlled study by METR measuring the real-world impact of AI coding tools on experienced open-source developers reportedly found developers were measurably slower, not faster, when using contemporary AI coding assistants on real repositories — a result that directly complicates any narrative built purely from benchmark improvement curves. This specific figure is cited here as reported in coverage of METR's study; readers should treat the exact percentage as needing direct confirmation against METR's own published report before being repeated as a precise number.

Sample Efficiency: The Bottleneck Underneath the Gap

Part of why sim-to-real transfer is hard is that RL policies in simulation can be trained on volumes of experience no real-world process could ever generate. AlphaGo Zero's own 2017 Nature paper reports its self-play training run generated on the order of 4.9 million self-play games over 40 days. A human professional Go player, across an entire competitive career, plays on an order of magnitude fewer games — commonly estimated in the tens of thousands. Lake, Ullman, Tenenbaum, and Gershman's influential 2017 "Building Machines That Learn and Think Like People" makes exactly this argument: matching or exceeding human performance by consuming millions of times more training experience than a human ever receives is a different, and in some ways less interesting, achievement than matching human sample efficiency.

Benchmarks vs. Real Research: What the Benchmark Authors Themselves Say

This series covered METR's RE-Bench and OpenAI's MLE-bench in Part 4 as evidence that AI research-automation is real and measurable. It's worth returning to what those benchmarks' own authors say about their limits. MLE-bench's own paper is explicit: its Kaggle-derived tasks come with clear, pre-defined problem statements and success metrics, whereas real-world AI research frequently requires figuring out the problem definition and the right metric in the first place — arguably the harder and more research-relevant part of the job. METR's own commentary on RE-Bench draws a similar distinction: RE-Bench's tasks are more decomposable and cleanly scoped than the messy, open-ended, long-horizon research problems a real lab actually works on, and succeeding at one bounded ten-hour benchmark task is not equivalent to reliably chaining together the ten one-hour real subtasks a genuine research project requires.

This is the sim-to-real gap in a different costume: a benchmark task is, functionally, a simulation of research — cleaner, better-specified, and more tractable than the real thing by design, because that's what makes it measurable at all. The systems covered in Part 4 (Dream-RSI, ModularRSI, ScienceBuddy) are genuinely real and measured against these benchmarks. Whether their benchmark gains transfer at the same rate into messy, real, open-ended research — the way SWE-Bench Pro showed coding-agent gains largely didn't transfer to production codebases — is a question this series cannot yet answer, because the equivalent of a "SWE-Bench Pro for AI research agents" doesn't yet exist as a published, adversarially-designed successor benchmark.

When the Benchmark Becomes the Target

There's a formal name for part of why this gap persists even as benchmark scores climb. Skalse et al.'s "Goodhart's Law in Reinforcement Learning" (ICLR 2024) — introduced conceptually in Part 2 of this series in the context of reward specification — provides a rigorous, empirically demonstrated result specific to RL agents: optimizing an imperfect proxy reward past a certain threshold reliably degrades performance on the true underlying objective, even as the proxy metric keeps climbing. Applied here, that means a benchmark score climbing toward saturation is not, on its own, evidence that the underlying capability the benchmark was designed to measure is climbing at the same rate — some of that climb can be optimization pressure finding the specific shape of the benchmark rather than the general capability behind it.

Why This Matters Specifically for Recursive Self-Improvement

If a research-automation agent's measured acceleration is partly a benchmark artifact — the RE-Bench/MLE-bench equivalent of Dactyl's 60%-on-easy-scrambles headline being reported as "solves Rubik's Cube" — then claims about how fast RSI is actually progressing need the same discount applied to robotics claims after Tobin et al.'s work made the field skeptical of simulation-only results. This doesn't mean the systems in Part 4 aren't real or aren't useful; Dream-RSI's reported call-reduction and AlphaEvolve's kernel speedups are independently verifiable, measured results. It means the step from "measurably better on a research benchmark" to "measurably accelerating real, messy, open-ended AI research at the same rate" is exactly the step that robotics and coding agents have both shown can be much larger than it first appears — and no one has yet published the adversarial, contamination-resistant successor benchmark that would let this series make a confident claim about how large that step currently is for AI research agents specifically.

Expert Disagreement

Five Parts Published, One to Go

Readiness Checklist

⚠️ What's Missing or Uncertain

🔗 Reference Links

🎥 Recommended Videos

🧭 Closing — The Gap Doesn't Disappear, It Moves to Wherever You Haven't Measured Yet