Home โ€บ Blog โ€บ The Maestro, Season 2 ยท Part 6
The Maestro, Season 2 ยท Part 6 ยท Phase 6 ยท ๐Ÿงฎ Research director hat

Many Experiments, Cheaply

A research org that runs few, expensive experiments learns slowly. Gopala wants many small bets, most of which die early and cheaply. This part shows how: score proposals by value per GPU-hour, pick the best set that fits the budget, write the kill bars down first, and make every experiment climb an evidence ladder from toy to full scale.

FL
FrontierAGI Team
๐Ÿ’ผ CEO๐Ÿ”ฌ Research director๐Ÿ“Š Analyst
โš ๏ธ How to read this (as of 3 Nov 2026) experiment_portfolio.py and kill_ladder.py were run (real output below). Every experiment, probability, value, cost and result is the story's scenario. Explore the graph first; open the ๐Ÿ”Ž drawers for depth. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: months 28โ€“31

Week 112 ยท Proposals
  • Eight proposals ask for 210k GPU-hours; the quarter has 120k.
Week 113 ยท The pick
  • Five chosen by experiment_portfolio.py; kill bars written for each.
Week 124 ยท The ladder
  • Three experiments killed early; 31% of compute freed (kill_ladder.py).

2. The map: from proposal to result

๐Ÿ•ธ๏ธ How an experiment lives or dies

or click any node

Ideas in purple, rungs in amber, guards in green, kills in red. All numbers from the two scripts below.

3. Pick by value per GPU-hour

experiment_portfolio.py ยท Python 3, no dependencies; scenario proposals
"""experiment_portfolio.py โ€” pick this quarter's experiments under a fixed
compute budget, by expected value per GPU-hour (a 0/1 knapsack).

Every experiment is the story's SCENARIO proposal: p = chance it gives a
usable result, value = what a usable result is worth to the roadmap (0-10),
cost = GPU-hours (thousands). Budget: 120k GPU-hours of research compute.
"""
from itertools import combinations

EXPS = [  # id, track, p, value, cost (k GPU-h)
    ("X1", "L4 skill retrieval v2", 0.6, 8, 30),
    ("X2", "L4 skill pruning", 0.7, 5, 10),
    ("X3", "L3 failure curriculum", 0.4, 7, 25),
    ("X4", "L2 harness search", 0.5, 6, 20),
    ("X5", "L1 eval sampling", 0.9, 2, 5),
    ("X6", "L5 sandbox: self-edited search", 0.15, 9, 40),
    ("X7", "env generator (Part 8)", 0.5, 8, 35),
    ("X8", "long-context skills", 0.3, 6, 45),
]
BUDGET = 120

def main():
    best, bv = (), 0.0
    for k in range(1, len(EXPS) + 1):
        for c in combinations(EXPS, k):
            cost = sum(e[4] for e in c)
            ev = sum(e[2] * e[3] for e in c)
            if cost <= BUDGET and ev > bv:
                best, bv = c, ev
    print(f"{'id':3} {'experiment':30} {'EV':>5} {'k GPU-h':>8} {'EV/k':>6}")
    for e in sorted(EXPS, key=lambda e: -e[2] * e[3] / e[4]):
        mark = "*" if e in best else " "
        print(f"{e[0]:3} {e[1]:30} {e[2] * e[3]:5.2f} {e[4]:8} {e[2] * e[3] / e[4]:6.3f} {mark}")
    print(f"chosen (*): {', '.join(e[0] for e in best)}  cost {sum(e[4] for e in best)}k of {BUDGET}k  EV {bv:.2f}")

if __name__ == "__main__":
    main()
$ python3 experiment_portfolio.py # exact search over all 255 subsets id experiment EV k GPU-h EV/k X5 L1 eval sampling 1.80 5 0.360 X2 L4 skill pruning 3.50 10 0.350 * X1 L4 skill retrieval v2 4.80 30 0.160 * X4 L2 harness search 3.00 20 0.150 * X7 env generator (Part 8) 4.00 35 0.114 * X3 L3 failure curriculum 2.80 25 0.112 * X8 long-context skills 1.80 45 0.040 X6 L5 sandbox: self-edited search 1.35 40 0.034 chosen (*): X1, X2, X3, X4, X7 cost 120k of 120k EV 18.10

Expected value per thousand GPU-hours

X5 eval sampling0.36
X2 skill pruning0.35 โœ“
X1 skill retrieval v20.16 โœ“
X4 harness search0.15 โœ“
X7 env generator0.11 โœ“
X3 failure curriculum0.11 โœ“
X8 long-context skills0.04
X6 L5 self-edited search0.03

Real output; โœ“ = chosen. The L5 experiment loses on value per GPU-hour, not on a rule.

๐Ÿ”Ž Go deeper: why the best ratio (X5) was left out
  • Ratios are a guide, not the answer.
    • The exact search fills the budget to 120k with five experiments worth 18.10; adding X5 would force out a bigger, better experiment.
  • It still ran.
    • When X7 died at the toy rung and freed compute, X5 joined (see the ladder).

4. The evidence ladder: die early, die cheap

Four rungs, each with a bar written in advance

๐ŸงธToy ยท 2%bar 0.5 pts
๐Ÿ”ฌSmall ยท 10%bar 1.0
๐ŸงชMedium ยท 30%bar 1.5
๐ŸFull ยท 100%bar 2.0

Season 1, Part 2's method at team scale.

kill_ladder.py ยท Python 3, no dependencies; scenario results
"""kill_ladder.py โ€” run experiments up an evidence ladder and stop early.

Rungs (Season 1, Part 2's method at team scale): toy (2%) -> small (10%) ->
medium (30%) -> full (100%) of the experiment's compute. Each rung has a
pre-registered bar; an experiment that misses a bar is killed there.
Results are SCENARIO numbers: the measured gain (points) at each rung.
The chosen five ran; X5 joined after X7 was killed and freed compute.
"""
RUNGS = [("toy", 0.02, 0.5), ("small", 0.10, 1.0), ("medium", 0.30, 1.5), ("full", 1.00, 2.0)]
RESULTS = {  # experiment: (k GPU-h if run to full, gains at each rung)
    "X1": (30, [1.1, 1.6, 2.2, 2.6]), "X2": (10, [0.8, 1.2, 1.4, 1.5]),
    "X3": (25, [0.9, 0.6, None, None]), "X4": (20, [0.7, 1.3, 1.9, 2.3]),
    "X5": (5, [0.6, 1.1, 1.6, 2.1]), "X7": (35, [0.3, None, None, None]),
}

def main():
    spent = full = 0.0
    for x, (cost, gains) in RESULTS.items():
        full += cost
        for (name, frac, bar), g in zip(RUNGS, gains):
            spent += cost * frac
            if g is None or g < bar:
                print(f"{x}: killed at {name:6} (gain {g if g is not None else 0:.1f} < bar {bar}) after {cost * frac:5.1f}k GPU-h")
                break
        else:
            print(f"{x}: passed all rungs, {cost}k GPU-h")
    print(f"compute used {spent:.1f}k of {full:.0f}k if everything ran to full ({1 - spent / full:.0%} saved)")

if __name__ == "__main__":
    main()
$ python3 kill_ladder.py # six experiments that ran X1: passed all rungs, 30k GPU-h X2: killed at medium (gain 1.4 < bar 1.5) after 3.0k GPU-h X3: killed at small (gain 0.6 < bar 1.0) after 2.5k GPU-h X4: passed all rungs, 20k GPU-h X5: passed all rungs, 5k GPU-h X7: killed at toy (gain 0.3 < bar 0.5) after 0.7k GPU-h compute used 86.0k of 125k if everything ran to full (31% saved)

What the ladder did this quarter

3 of 6killed before full scale
0.7kGPU-hours to kill X7
31%compute saved
3results sent to the gate

Real output of kill_ladder.py.

5. Sandbox: you pick the experiments

๐ŸŽฎ Sandbox: pick this quarter's experiments

Same numbers as experiment_portfolio.py; scenario estimates.

6. Week 115: the environment generator dies (scenario)

Scenario
  • X7, an automatic generator of task environments, gains only 0.3 points at the toy rung against a bar of 0.5. It is killed after 0.7k GPU-hours.

7. Five mistakes

8. Delegation box

๐Ÿ™‹ Gopala himself
  • The budget split and kill-bar policy
  • Final say on contested kills
๐Ÿค– AI agents
  • Running the portfolio and ladder scripts every week
  • Drafting experiment reports for researchers to check
๐Ÿค Bought, rented or borrowed
  • External reviewers for disputed results

9. Budget and risk tracker

ItemStatus after Part 6Note
Experiments5 chosen, 6 runFrom the scripts
Compute86k of 125k used31% saved by early kills
Research speedBetterMore ideas per quarter
Environment planWatchGenerator v1 killed

10. The series

The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)

VerdictMany cheap experiments beat a few expensive ones: score by value per GPU-hour, fill the budget with an exact search, set the bars first, and let the ladder kill most ideas before they cost much. Next: which of the surviving directions could become the next moat.

11. Reading the numbers

Primary / standard
Method from Season 1, Parts 1โ€“2; knapsack and staged testing are standard techniques.
Secondary โš ๏ธ
None in this part.
Our assumptions
All experiments, probabilities, values, costs, bars and results are the story's scenario.