Finding the Moat in a Self-Improving (L4) Model
This series follows one person, Gopala, from an idea to a working AI factory and the products that run on it. He wears every hat in turn: researcher, founder, infrastructure engineer, ML engineer, evaluator, safety lead, operator. It starts here, with the least glamorous and most important part: deciding what to build and why anyone would pay for it instead of using a model from a big lab. In three weeks of reading, sketching and coding, Gopala places his idea on the ladder of self-improving AI, picks level L4, writes hypotheses that could prove him wrong, and runs a first experiment that fails before a second one works.
1. Meet the Maestro
Every hat Gopala wears
One person, six hats, 18 parts. The series follows the order the hats are needed.
- Gopala is a researcher-founder.
- He has read most of this site: the compute stack, the gigawatt case studies, the ML, data, evaluation and agent layers, and the RSI Evolution Map of self-improving systems.
- He has a conviction that the next real advantage in AI will not come from a slightly bigger model but from a model that gets better at its job by doing its job, safely, in a way that compounds for its owner.
- Over the series he takes that conviction from a notebook to a running AI factory at start-up scale.
- The phases overlap, because real projects do: racks are ordered while data pipelines are written, evaluations are designed before training starts.
- Mission Control, the series hub, shows all of it on one timeline.
2. Mission log: weeks 1–3
- Day 1–2: pulls every L3–L5 paper from the RSI map's catalog for 2024–2026 into a reading list; asks a research agent to summarise each in a fixed template (claim, evidence, compute, what is frozen, what changes).
- Day 3–4: reads the dozen that matter in full, not the summaries: STaR, Reflexion, Voyager, Self-Rewarding Language Models, the Darwin Gödel Machine, AlphaEvolve, DreamCoder.
- Day 5: writes a one-page "what everyone freezes" table (section 4). The pattern jumps out: almost every system keeps the foundation model frozen and improves something around it.
- Places five candidate ideas on the L1–L5 ladder; rejects two L5 ideas as too hard to make safe and too hard to sell; rejects one L2 idea as easy to copy.
- Chooses an L4 idea: a model whose deployment produces verified skills that are kept, and periodically folded back into the weights, under a fixed, human-approved gate.
- Writes the moat argument and asks two friends who run start-ups to attack it (section 6).
- Writes falsifiable hypotheses and kill criteria (section 7).
- Codes a toy version of the loop in an afternoon. It makes things worse (section 8). Spends two days understanding why.
- Second version works (section 9). Writes up the result honestly, including the failure, and moves on to proving it with real models (Part 2).
3. The self-improvement ladder
- The RSI Evolution Map sorts self-improving AI work into five levels, using the definitions of the Awesome-RSI catalog it draws on.
- The level matters because it decides what can change, who controls the change, and how hard it is to keep safe.
Where does the idea sit?
Tap a level. Definitions from the catalog behind our RSI map; the start-up notes are our reading.
- Gopala's choice is L4, deployment adaptation: the system keeps memory, skills or components that change its later behaviour, but inside an improvement process that stays fixed.
- That last clause is the important one.
- At L4 the system gets better, but the rules for what counts as "better" and what is allowed to be kept are written by people and do not rewrite themselves.
- That is what makes it explainable to a client, testable by an evaluator and acceptable to a safety reviewer.
4. Reading the field
- The question Gopala asks of every paper is the same: what does it change, what does it freeze, and how does it decide a change is good?
| Work | What improves | What stays frozen | How "better" is decided | Lesson for the idea |
|---|---|---|---|---|
| STaR (Zelikman et al., 2022) | Model weights, by fine-tuning on its own rationales that reached correct answers | The training recipe | Ground-truth answers | Verified self-generated data can improve weights |
| Reflexion (Shinn et al., 2023) | A memory of written self-reflections | Model weights | Task feedback | Memory alone helps, but it is per-task and fragile |
| Voyager (Wang et al., 2023) | A library of executable code skills in Minecraft | Model weights | The skill runs and succeeds in the game | Code skills are verifiable and reusable: the closest ancestor |
| DreamCoder (Ellis et al., 2021) | A library of abstractions for program synthesis | The learning algorithm | Solving tasks more compactly | Store the reusable core, not whole solutions (we relearn this in section 8) |
| Self-Rewarding LMs (Yuan et al., 2024) | Model weights, via preferences the model judges itself | The procedure | The model's own judgements | Powerful but risky: the judge and the student are the same model |
| Darwin Gödel Machine (Zhang et al., 2025) | The agent's own code, evolved as an archive | The foundation models | Benchmark scores (SWE-bench 20.0% → 50.0%, Polyglot 14.2% → 30.7%, reported) | Empirical validation replaces proofs; run in sandboxes with human oversight |
| AlphaEvolve (Google DeepMind, 2025) | Algorithms and code, via evolutionary search with LLMs | The models and evaluators | Automatic evaluators | Value comes where results can be checked by machine; DeepMind reports a scheduling heuristic that recovers about 0.7% of Google's fleet-wide compute ⚠️ |
- Two patterns decide the idea.
- First, the systems that improve reliably are the ones whose output can be checked by a machine: code that runs, answers that match, tests that pass.
- Second, almost everyone freezes the foundation model and improves what surrounds it, because changing weights is expensive and changing them with the model's own judgement is risky.
- The gap Gopala sees: a start-up can own the whole loop, from verified skills in deployment back into its own weights, if the checking is strict and the gate is human-controlled.
- The site's longer reviews of this work are self-improving agentic systems, the self-improving agents handbook and AI doing R&D.
5. The idea: a verified-skill flywheel
The verified-skill flywheel
The loop closes back to Attempt. The gate is fixed and owned by a human; the model cannot change it.
The loop Gopala wants to own
Press play. Each lap, verified work becomes skills; at fixed intervals the best skills are folded into the next model, but only through the gate. Our diagram.
- In plain terms: the model works on client tasks where success can be checked by a machine.
- Gopala starts with software and data engineering (writing and fixing code, building data pipelines, migrating schemas), because tests decide success there.
- Every task runs in a sandbox.
- When a solution passes its tests, including tests the model never saw, the reusable core of the solution is stored as a skill: code plus the tests that prove it plus where it applies.
- Next time, the model retrieves relevant skills before it starts.
- Every few weeks, the best-verified skills become training data for a new version of the model, and that version only ships if it passes a release gate that Gopala and, later, an outside reviewer control.
- The model gets better at the client's work by doing the client's work.
- It keeps what it learned in two places: a skill library it can inspect, and its weights.
- And nothing changes the rules of the loop except people.
6. Is it a moat?
- A moat is something a competitor with more money cannot simply buy or copy.
- Gopala scores the idea honestly on the forces that usually decide it.
- Move the sliders to see how the conclusion changes with your own view.
Moat scorecard
Our framework. 1 = weak, 5 = strong.
- The honest conclusion: the technique is not the moat; the verified data and the gate are.
- That changes the plan.
- It means the first product must get into real client workflows early, the skill format and gate must be excellent, and the decision to pre-train a model from scratch has to be justified in Part 3 against the cheaper option of post-training an open model.
7. Hypotheses and kill criteria
- A research plan is only useful if it can fail.
- Gopala writes three hypotheses and, for each, the result that would make him stop or change course.
- H1 can be tested on a laptop; H2 needs real models (Part 2); H3 runs through every phase and becomes central in Phase 3.
8. Experiment 1: it fails
- Before renting a single GPU, Gopala tests the shape of H1 with a toy.
- A "solver" searches for tiny programs that turn inputs into outputs.
- Tasks come in families that share hidden sub-routines, just as client tasks share patterns.
- Each task shows four examples and hides four more.
- A verified solution can be stored as a skill, and the solver gets the same search budget per task whatever its library holds, so any gain must come from the skills, not from more compute.
- His first version does the obvious thing: hoard.
- Every verified solution is added to the search vocabulary.
- The result: the more it remembers, the worse it gets.
The full script is in the next section; this is the line that compares the variants, with a fixed random seed:
- Without skills ("off") the solver averages about 14 of 40 tasks per round.
- Hoarding drops it to about 11, with wild swings, even as the library grows to 40 skills.
Two days of digging find two reasons, both of which matter for the real system:
9. Experiment 2: it works
Experiment results: average tasks solved (of 40, hidden tests)
Real output of l4_toy.py, averaged over six rounds. Hoarding hurts; curating helps.
- The second version, curate, changes three things, each borrowed from the papers in section 4: it stores only the abstracted core of a solution (DreamCoder's lesson), it retrieves the most useful skills first and tries them with small wrappers before falling back to blind search (Voyager's lesson), and every new skill must pass the gate: hidden tests, no duplicates, no drop in a validation score.
- Same budget per task.
"""l4_toy.py — a toy L4 (deployment-adaptation) loop you can run on a laptop.
A solver searches for small programs that match a task's examples. Verified
solutions become reusable skills, kept only if a fixed, human-written gate
approves them. Two versions are compared at the SAME search budget per task:
v1 "hoard": every verified solution is added to the search vocabulary
v2 "curate": abstract the reusable core, retrieve the top skills, try them first
Pure Python 3, no dependencies. Run: python3 l4_toy.py
"""
import random
PRIMS = {
"add1": lambda x: x + 1, "add3": lambda x: x + 3, "sub2": lambda x: x - 2,
"dbl": lambda x: 2 * x, "tri": lambda x: 3 * x, "sq": lambda x: x * x,
"neg": lambda x: -x, "half": lambda x: x // 2, "mod7": lambda x: x % 7,
}
CORES = [["dbl", "add3"], ["sq", "sub2"], ["tri", "mod7"], ["neg", "half"], ["add1", "sq", "dbl"]]
PROBE = list(range(-9, 10))
def run(prog, x):
for op in prog:
x = PRIMS[op](x)
if abs(x) > 10**9:
raise OverflowError
return x
def passes(prog, pairs):
try:
return all(run(prog, x) == y for x, y in pairs)
except OverflowError:
return False
def make_task(rng):
"""Tasks share hidden sub-routines: a random op, a shared core, a random op."""
core = rng.choice(CORES)
pre, post = rng.sample(list(PRIMS), 2)
target = [pre] + core + [post]
xs = rng.sample(PROBE, 8)
pairs = [(x, run(target, x)) for x in xs]
return {"visible": pairs[:4], "hidden": pairs[4:]}
class Agent:
def __init__(self, mode, seed):
self.mode, self.rng, self.skills, self.uses = mode, random.Random(seed), [], {}
# ── the solver: a fixed budget of candidate programs per task ────────────
def solve(self, task, budget=3000):
tried = 0
if self.mode == "curate" and self.skills: # retrieval: best skills first
top = sorted(self.skills, key=lambda s: -self.uses.get(tuple(s), 0))[:8]
wraps = [[]] + [[p] for p in PRIMS]
for sk in top:
for pre in wraps:
for post in wraps:
prog = pre + sk + post
tried += 1
if passes(prog, task["visible"]):
self.uses[tuple(sk)] = self.uses.get(tuple(sk), 0) + 1
return prog
if tried >= budget:
return None
vocab = [[p] for p in PRIMS] + (self.skills if self.mode == "hoard" else [])
while tried < budget: # blind search for the rest
prog = [op for _ in range(self.rng.randint(1, 5)) for op in self.rng.choice(vocab)]
tried += 1
if passes(prog, task["visible"]):
return prog
return None
# ── the gate: a fixed improvement process that never rewrites itself ─────
def gate(self, prog, task, validation):
if not passes(prog, task["hidden"]):
return False # overfit to visible examples
cand = prog[1:-1] if (self.mode == "curate" and len(prog) >= 4) else prog
sig = tuple(run(cand, x) for x in PROBE) if len(cand) >= 2 else None
if sig is None or any(tuple(run(s, x) for x in PROBE) == sig for s in self.skills):
return False # trivial or a duplicate
before = self.score(validation)
self.skills.append(cand)
if self.score(validation) < before: # must not make things worse
self.skills.pop()
return False
return True
def score(self, validation):
state = self.rng.getstate()
n = sum(self.solve(v, budget=800) is not None for v in validation)
self.rng.setstate(state)
return n
def experiment(mode, rounds=6, per_round=40, seed=1):
data = random.Random(seed)
validation = [make_task(data) for _ in range(12)]
agent, rows = Agent(mode, seed), []
for r in range(rounds):
tasks = [make_task(data) for _ in range(per_round)]
solved = 0
for t in tasks:
prog = agent.solve(t)
if prog and passes(prog, t["hidden"]):
solved += 1
if prog and mode != "off":
agent.gate(prog, t, validation)
rows.append((r, solved, len(agent.skills)))
return rows
if __name__ == "__main__":
for mode in ("off", "hoard", "curate"):
rows = experiment(mode)
print(f"{mode:7s}", " ".join(f"r{r}:{s:2d}/40({k:2d})" for r, s, k in rows))
print("format: round:solved-on-hidden-tests(skills in library)")Tasks solved per round, same budget
The real output above, charted. Each round is 40 new tasks; solved means correct on hidden tests.
- Curated skills lift the solve rate from about 14 to about 21 of 40 (roughly 50% more) at the same budget, with a smaller library (23 skills instead of 40).
- That is H1's shape, in miniature.
- It is a toy: one seed, a tiny made-up language, random search instead of a model.
- Its value is not the number but the two design rules it forces, which carry straight into the real system: store abstractions, not transcripts, and retrieve before you search.
10. The gate is the product
What the gate checks before any weight update
If any check says no, the update does not happen. Default answer: no.
- Look at the
gatefunction in the script again.- It is short, and it never changes itself.
- In the real system it grows into the most important piece of software Gopala writes, because it is what makes the loop L4 instead of L5, what a client audits, and what a safety reviewer signs off.
Its checks, in the order they will run:
| Check | Toy version | Real version (Phases 2–4) |
|---|---|---|
| Verified | Passes hidden tests | Passes hidden tests, static analysis and a sandboxed run; reviewed by an LLM judge for intent, with human spot checks (Evaluation Layer Part 3) |
| Reusable | Abstracted core, not a duplicate | Generalised, documented, deduplicated by behaviour; applicability conditions stored |
| No regression | Validation score does not drop | Full evaluation suite per skill batch and per model version; contamination checks (Part 2) |
| Safe | (not modelled) | No secrets, no network calls outside allowlists, no destructive operations; red-team probes for reward hacking (Agent Layer Part 8) |
| Approved | Automatic | Skills: automatic within policy. Weight updates: human sign-off, logged, reversible |
11. Safety from day one
- Self-improving systems are the part of AI that safety researchers worry about most, and with reason: a loop that rewards itself can drift, game its checks or learn things nobody intended.
Gopala writes his safety principles before writing the product, so every later design choice has to satisfy them:
- These become the safety case in Part 13 and the approval file in Part 14.
- The safety evaluations article explains what the frontier labs test for; Gopala will borrow their structure.
12. The 90-day research plan
The 90-day research plan
From Part 1. Each step ends with a decision, not a report.
| Weeks | Goal | Output | Decision at the end |
|---|---|---|---|
| 1–3 | Choose and frame the idea (this part) | Hypotheses, toy result, safety principles | Proceed to real models? Yes |
| 4–8 | Prove it small with real models (Part 2) | H1 and early H2 on 100M–1B-parameter models and an open 7–8B model; scaling-law fits | Does the effect survive with real models and grow with scale? |
| 6–10 | Plan and fund (Part 3) | Compute budget, build-versus-rent, from-scratch versus post-train decision, investor memo | Raise and commit to a training run, or pivot to post-training an open model |
| 8–12 | Design partners | Two to three companies providing real, verifiable tasks under contract | Is there demand at a price that pays for compute? |
13. Delegation box
- Reading the key papers in full
- Choosing the level and the domain
- Writing hypotheses, kill criteria and safety principles
- Writing the gate
- First-pass summaries of ~150 papers in a fixed template
- Boilerplate code and plots for the toy
- Red-teaming the moat argument
- Two founder friends as critics
- API credits for the research agent
- Nothing else yet: Phase 0 runs on a laptop
14. Budget and risk tracker
| Item | Status after Part 1 | Note |
|---|---|---|
| Spend to date | ~$300 (our estimate) | API credits for literature summaries; laptop only |
| Time | 3 of ~78 weeks | Plan horizon ~18 months to first products at scale |
| Technical risk: does L4 work? | Toy says the shape is right | Unproven with real models (Part 2) |
| Market risk: will clients share tasks? | Untested | Design partners in weeks 8–12 |
| Competition risk: big labs | High | Moat must be data and gate, not technique |
| Safety risk: loop drift | Principles written | Gate and tripwires to be built and red-teamed |
| Capital risk: from scratch vs post-train | Open question | Decided with numbers in Part 3 |
15. The series
The Maestro: 18 parts in five phases
- Gopala's idea is an L4 system: a model that keeps machine-verified skills from real work and folds the best of them back into its weights, under a fixed gate that people control.
- The technique is not the moat; the verified data and the gate are.
- A laptop toy shows both the trap (hoarding memories makes things worse) and the fix (abstract, retrieve, gate).
- The next step is to see whether the effect survives with real models and grows with scale: Part 2, proving it small.
16. Reading the numbers
- Primary (papers)
- STaR (arXiv 2203.14465); Reflexion (2303.11366); Voyager (2305.16291); Self-Rewarding Language Models (2401.10020); Darwin Gödel Machine (2505.22954): SWE-bench 20.0% → 50.0%, Polyglot 14.2% → 30.7%; DreamCoder (PLDI 2021). L1–L5 definitions from the Awesome-RSI catalog (CC0) used by our RSI map.
- Company-reported ⚠️
- AlphaEvolve's ~0.7% fleet-wide compute recovery (Google DeepMind).
- Our work
- The scenario, plan, budget, moat scorecard and flywheel diagram. The toy experiment and its output are real and reproducible with the script above (one seed; treat as illustration, not evidence about real models).
- Sources
- STaR · Reflexion · Voyager · Self-Rewarding Language Models
- Darwin Gödel Machine · Sakana AI: DGM
- Google DeepMind: AlphaEvolve · DreamCoder
- Our RSI Evolution Map (levels from the CC0 Awesome-RSI catalog)