Home › Blog › From Automated R&D to Effective RSI
Practical RSI · Frontier & Open Problems · Snapshot: September 2026 📈

From Automated R&D to Effective RSI: Frontier, Evidence and Open Problems

AI already codes experiments, searches literature, generates hypotheses, runs benchmarks, evolves algorithms and distils models. The question is no longer whether AI does R&D, but whether an improvement mechanism can produce successors that are reliably better — generation after generation, under matched resources and independent evaluation. This final part of the self-improvement arc sorts what is already real from what is unproven, explains the difference between structural and effective recursion, reviews the first experiments designed to test it (AIDE², AI4AI-Bench, PAST-Bench), shows with a runnable power analysis why a few generations can't settle the question, and maps the twelve open problems that still block genuine recursive self-improvement.

How to read this: Part 3 after AI Doing R&D and How Self-Improving Agentic Systems Work. The August–September 2026 papers discussed are arXiv preprints, not yet peer-reviewed; numbers are as their authors report and are linked in References. The detection simulator is an illustrative statistical model.

01 Current Reality: AI Already Doing R&D

Eight core research capabilities are established or partially established in bounded settings. Click a tile for the strongest public example and how mature it is:

▶ Demonstrated capabilities Bar = maturity in bounded settings
Click a capability.

These work today in focused domains and controlled environments. None amounts to full recursive self-improvement: each is a component that a recursive system would need, run under human direction.

02 The Autonomy Ladder and Its Evidence

The R0–R7 ladder (introduced in AI Doing R&D) is useful only with an honest evidence column. Public evidence is strong at R0–R3, limited demonstrations exist at R4–R5, and R6–R7 have no evidence yet. Hover a rung:

▶ R0 → R7 with evidence strength Hover a rung
Hover a rung.

03 The Core Frontier Loop

The frontier setup is an end-to-end AI R&D cycle — goal, literature, hypotheses, experiment design, code and training runs, results, analysis and verification, research memory, replanning — run by many agent teams in parallel, sharing what they learn through a common memory. Watch the teams explore; lines show ideas cross-pollinating via memory:

▶ Many agent teams, one research memory Diverse hypotheses, shared results

Parallelism buys exploration, specialisation and speed; shared memory turns one team's failure into every team's lesson. This is the architecture of OpenAI's ~10,000-agent maths run and Anthropic's automated alignment researchers (with a shared forum). What it does not buy by itself is direction — someone still sets the goal at the top.

04 Where the Evidence Is Strongest

AI R&D progress is most reliable when grounded in external, formal signals. The same five-level ranking from the verification hierarchy, now with the R&D result each level has supported:

GroundingReliabilityExample R&D result it supports
1 · Formal verification / proofsHighestLean-checked proofs (OpenAI's Navier–Stokes claim — though what was proved is still debated)
2 · Executable experiments / tests / benchmarksHighAlphaEvolve's faster kernels; ERA's benchmark-beating methods; AIDE²'s held-out gains
3 · Simulators / controlled environmentsModerate–highSandboxed training runs; AI4AI-Bench's fixed re-run protocol
4 · Learned judges / reward modelsModerateAutomated paper reviewing (The AI Scientist-v2's self-review)
5 · Self-judgmentLowest, easy to gameAn agent reporting its own improvement — not evidence

05 Structural vs Effective Recursion

The key distinction. Structural recursion: the improvement mechanism itself is modified and inherited — Improver 1 builds System 2 and an improved Improver 2, which builds System 3. Many systems already do this. Effective recursion: those inherited mechanisms reliably produce better successors — consistent, measurable gains across multiple generations, evaluated independently, under matched resources, not just overfitting. Toggle and play:

▶ Generation chain Bar = gain produced by each improver

✔ Structural recursion

  • Improves its own algorithms, prompts or architecture
  • Changes persist to successors
  • Mechanism-level improvement

✔ Effective recursion adds

  • Consistent, measurable gains
  • Works across multiple generations
  • Evaluated independently
  • Not just overfitting or benchmark gaming
  • Under matched compute, data and time

06 The First Tests of Effective RSI

In August–September 2026 three preprints tried to measure the recursive link directly rather than assert it. Each tests a different piece:

▶ Three experiments Pick one

Together they show both how far things have come and how far they have to go. AIDE² is the strongest public multi-generation result — seven accepted successive improvements, each gated on held-out data, generalising to external benchmarks — yet its gains are to the research agent, not the model, and it doesn't show that later improvements come faster or larger. AI4AI-Bench shows agents can improve training algorithms, but only modestly (best 0.25 on a scale where the shipped algorithm is 0.1 and the optimum 1.0). PAST-Bench shows persistence helps every model — unevenly, and not always via the intended pathway.

07 How Would We Know?

Suppose a lab runs its self-improving system for a few generations and sees gains. Is that compounding? Statistically, telling "gains accelerate" from "gains are constant" needs more generations than most experiments run — and if later generations also get more compute, fake acceleration appears easily. Move the sliders:

▶ Detection power Same model as the script below (500 simulated experiments per point)
"""How would we know? Detecting *effective* recursive self-improvement.

Effective RSI means each generation's improvement mechanism produces a successor whose gain is
LARGER than the last one's: gains accelerate. We simulate experiments: G generations, S independent
seeded runs per generation. Gain of generation k = 1.0 + accel * k + noise (sd 1.0), in benchmark
points. We regress gain on k and call acceleration "detected" if the slope is positive at the
one-sided 5% level (t-test). Also: if each generation also gets more compute than the last and
gains are not resource-normalised, extra compute adds 0.3 * k points and masquerades as acceleration.
2,000 simulated experiments per cell. Standard library only.
"""
import random, math

T95 = [6.314, 2.920, 2.353, 2.132, 2.015, 1.943, 1.895, 1.860, 1.833, 1.812, 1.796, 1.782, 1.771, 1.761,
       1.753, 1.746, 1.740, 1.734, 1.729, 1.725, 1.721, 1.717, 1.714, 1.711, 1.708, 1.706, 1.703, 1.701,
       1.699, 1.697]                                       # one-sided 95% t critical values, df 1..30

def detect(G, S, accel, compute_drift, rng):
    xs, ys = [], []
    for k in range(G):
        for _ in range(S):
            xs.append(k); ys.append(1.0 + (accel + compute_drift) * k + rng.gauss(0, 1.0))
    n = len(xs); mx = sum(xs) / n; my = sum(ys) / n
    sxx = sum((x - mx) ** 2 for x in xs)
    b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sxx
    resid = sum((y - my - b * (x - mx)) ** 2 for x, y in zip(xs, ys))
    se = math.sqrt(resid / (n - 2) / sxx)
    df = n - 2
    return b / se > (T95[df - 1] if df <= 30 else 1.645)

def power(G, S, accel, drift=0.0, trials=2000):
    rng = random.Random(G * 100 + S * 10 + int(accel * 10) + int(drift * 7))
    return sum(detect(G, S, accel, drift, rng) for _ in range(trials)) / trials

print("Chance of detecting accelerating gains (3 seeded runs per generation)\n")
print(f"   {'generations':>11} {'no accel':>10} {'weak 0.1':>10} {'strong 0.3':>11}")
for G in [2, 3, 5, 7, 10]:
    print(f"   {G:>11} {power(G, 3, 0.0):>10.0%} {power(G, 3, 0.1):>10.0%} {power(G, 3, 0.3):>11.0%}")
print("\n'no accel' is the false-positive rate: it should stay near 5%.")
print(f"\nNo real acceleration, but compute grows each generation (not normalised), 7 generations:"
      f" {power(7, 3, 0.0, 0.3):.0%} 'detect' acceleration")
print("\nFew generations cannot tell compounding from constant gains; unmatched compute fakes it.")

Output:

Chance of detecting accelerating gains (3 seeded runs per generation)

   generations   no accel   weak 0.1  strong 0.3
             2         6%         6%          9%
             3         5%         7%         17%
             5         5%        12%         48%
             7         5%        22%         84%
            10         5%        44%        100%

'no accel' is the false-positive rate: it should stay near 5%.

No real acceleration, but compute grows each generation (not normalised), 7 generations: 85% 'detect' acceleration

Few generations cannot tell compounding from constant gains; unmatched compute fakes it.

With three generations, even strong compounding is detected only 17% of the time; with seven it's 84%; weak compounding needs well over ten. And a design that lets each generation use more compute "detects" acceleration 85% of the time when there is none. This is why the definition of effective RSI insists on multiple generations, matched resources and independent evaluation: without them, the claim can't be distinguished from noise or from spending more.

08 Twelve Open Problems Blocking Genuine RSI

Technical, evaluation and governance challenges remain. Filter by type and click a card for why it matters and who is working on it:

▶ Open problems Filter, then click
Click a problem.

09 What Researchers Should Focus on Now

A practical near-term roadmap toward safe, effective RSI — each step makes the next measurable. Press play:

▶ Four-step roadmap Build trustworthy improvement loops, not claims

10 Bottom Line

✅ Already real (demonstrated today)

  • AI-assisted R&D: search, coding, analysis
  • Automated experiments and benchmarking
  • Harness and workflow auto-improvement
  • Parallel agent research
  • Teacher–student distillation and synthetic data

🟠 Actively emerging (rapid progress)

  • Autonomous research loops in bounded domains
  • Model–harness co-evolution
  • Evaluator-aware improvement
  • Long-horizon research agents and research organisations
  • Multi-generation self-improvement of research agents (AIDE²)

❓ Still unproven (open frontier)

  • Robust effective recursive systems (R7)
  • Autonomous agenda-setting
  • Open-ended recursive improvement
  • Sustained multi-generation capability gains that compound
  • Safe and controllable RSI in the real world
Part 3 in one paragraph
AI already does large parts of R&D in bounded settings, and structural recursion — systems that modify and pass on their own improvement machinery — is routine in research. The 2026 frontier is measurement: AIDE² showed seven successive self-improvements of a research agent, each gated on held-out data and generalising to external benchmarks; AI4AI-Bench showed agents can improve training algorithms only modestly; PAST-Bench showed persistence helps unevenly. Public evidence for strong effective multi-generation RSI — gains that compound, under matched resources and independent evaluation — remains absent, and our power analysis shows why short experiments can't provide it. The blockers are evaluators, research taste, agenda selection, generalisation, safe inheritance, reward hacking, resource-normalised measurement, long-horizon planning, independent verification and human oversight. The most valuable current work is not claiming RSI but building trustworthy improvement loops — solid methods, evaluators and evidence will make future RSI more achievable and safer.

Next: Open Problems in Recursive Self-Improvement: A Research Map →

⚠️ Notes

About the source infographic: (1) "Public evidence for strong effective multi-generation RSI remains unproven" — still accurate, but it should now cite AIDE² (arXiv 2609.26457, 23 Sep 2026): seven successive, held-out-gated improvements of a research agent in an 8-day run, generalising to four external benchmarks, with reward hacking on a held-out task family falling from 55% to 32%. It is multi-generation structural recursion with independent evaluation, but improves the agent, not the model, and doesn't demonstrate compounding; (2) the infographic mentions "resource-normalized progress" and "multi-generation experiments with matched resources" without showing why they matter — section 07 quantifies it; (3) we add the purpose-built benchmarks AI4AI-Bench (arXiv 2608.20318) and PAST-Bench (2608.04003); (4) "Research Lead" (R5) shown as "early / partial" rests mostly on lab self-reports. No numerical claims in the infographic needed correction. The R-ladder evidence ratings and the detection model are our own.

🔗 References