AI Automating AI Research: RSI's Testing Ground

From Benchmarks to Real Systems

The RSI primer covered the two standard benchmarks for AI-doing-AI-research (RE-Bench, MLE-bench) and one verified real result (AlphaEvolve's matrix-multiplication improvement). Since then, three more systems have been published, each attacking a different piece of the same problem — and each worth examining not just for what it does, but for what it deliberately does not touch: the base model's weights.

Schulman, O'Neill, and Millidge on Where This Actually Stands

The conversation covers several threads directly relevant to this series. On objective specification, the group reportedly discussed the persistent difficulty of defining the "right" training objective for an AI system, with humans still playing the final role in judging alignment — a theme this series' Part 2 will take up in depth. On distillation and centralization, they explored whether large frontier labs hold a durable structural advantage, noting that while distillation helps smaller models catch up quickly, assembling the complex, realistic prompt distributions needed for frontier-quality training remains a real barrier to entry — directly relevant to this site's DeepSeek lineage article, where distillation from a larger model into six smaller open ones was a central technical story. On training future AI researchers, the participants reportedly debated whether the automation of AI R&D will be a clean, objective-driven optimization process or one still guided by human intuition and iterative testing — a live, unresolved question this piece returns to below. On the sim-to-real gap and sample efficiency, and on how much of recent progress reflects data versus reinforcement learning (with RL framed as increasing the signal-to-noise ratio in training by reinforcing specific correct reasoning paths), the conversation reportedly covered ground this series' Parts 5 and 6 will treat in full depth. The participants' rapid-fire predictions on when a fully autonomous, month-long-horizon AI worker might arrive connects directly to the METR time-horizon discussion already covered in this site's AGI Benchmarks article and this series' Part 1.

Four Real Researchers, One Conversation That Never Happened

Three Real Systems From a Single Week in September 2026

Within days of each other in mid-September 2026, three separate research groups published systems that each improve an AI agent's effectiveness while keeping the underlying language model's weights completely frozen — a genuinely important design choice, since it means these gains come from redesigning the process around the model, not from a bigger or better-trained model itself.

A Near-Ideal Laboratory, and What Happens When It's Gamed

GPU kernel optimization is an unusually good testing ground for automated research because feedback is fast and objective: does the kernel compile, does it produce the correct output, and how fast does it run compared to alternatives. Two real projects have worked in this space, with sharply different outcomes for how much to trust self-reported results.

Beyond Benchmark Score Per Token

As AI systems take on more of the research and engineering pipeline, how they're evaluated starts to matter as much as how capable they are. Vals AI, a real evaluation company founded in 2024 (which raised a $40 million Series A led by a16z), focuses specifically on real-world task reliability in domains like law, finance, and coding rather than raw benchmark leaderboard position. The underlying argument — already familiar from this site's Benchmark Landscape article's discussion of the gap between benchmark performance and production reliability — is that the economically relevant question isn't "what's the highest score," but something closer to useful task completion divided by total system cost, where total cost includes retries, human review of failures, and recovery from mistakes, not just per-token API pricing.

The Question That May Matter More Than Any Benchmark

Ryan Greenblatt, a real researcher at Redwood Research, has written and spoken extensively (including in his own separate interview, distinct from the fabricated joint conversation named above) about a genuine and important distinction: many AI research subtasks — reproducing a bug, implementing a described paper, optimizing an inference pipeline — can be automatically checked for correctness, which makes them trainable via reinforcement learning at scale. The open question Greenblatt raises is whether skill built on thousands of these small, verifiable tasks actually transfers to the higher-level judgment calls that matter most in frontier research: which experiment is worth the compute, which anomaly is a real finding versus a bug, which direction is worth pursuing when the answer isn't yet knowable.

If verifiable-task skill does not transfer well, AI may remain an extraordinarily fast engineering assistant while humans retain strategic control over what actually gets built — a scenario much closer to this series' primer's "automation without strong acceleration" outcome than to a genuine recursive loop.

⚠️ What's Missing or Uncertain

🔗 Reference Links

🎥 Recommended Videos

🧭 Closing — Real Progress, Deliberately Read With Suspicion