The Maestro, Season 2 ยท Part 6 ยท Phase 6 ยท ๐งฎ Research director hat
Many Experiments, Cheaply
A research org that runs few, expensive experiments learns slowly. Gopala wants many small bets, most of which die early and cheaply. This part shows how: score proposals by value per GPU-hour, pick the best set that fits the budget, write the kill bars down first, and make every experiment climb an evidence ladder from toy to full scale.
๐ผ CEO๐ฌ Research director๐ Analyst
โ ๏ธ How to read this (as of 3 Nov 2026)
experiment_portfolio.py and kill_ladder.py were run (real output below). Every experiment, probability, value, cost and result is the story's scenario. Explore the graph first; open the ๐ drawers for depth. Disclosure: Claude, made by Anthropic, helps write this site.
Mission log: months 28โ31
The map: from proposal to result
Pick by value per GPU-hour
The evidence ladder: die early, die cheap
Sandbox: you pick the experiments
Week 115: the environment generator dies (scenario)
Five mistakes
Delegation box
Budget and risk tracker
The series
Reading the numbers
1. Mission log: months 28โ31
Week 112 ยท Proposals
- Eight proposals ask for 210k GPU-hours; the quarter has 120k.
Week 113 ยท The pick
- Five chosen by
experiment_portfolio.py; kill bars written for each.
Week 124 ยท The ladder
- Three experiments killed early; 31% of compute freed (
kill_ladder.py).
2. The map: from proposal to result
๐ธ๏ธ How an experiment lives or dies
Ideas in purple, rungs in amber, guards in green, kills in red. All numbers from the two scripts below.
- Three things to notice.
- The budget, not enthusiasm, decides what runs.
- Kill bars are set before the first run.
- Most of the savings come from the bottom rungs.
3. Pick by value per GPU-hour
experiment_portfolio.py ยท Python 3, no dependencies; scenario proposals
"""experiment_portfolio.py โ pick this quarter's experiments under a fixed
compute budget, by expected value per GPU-hour (a 0/1 knapsack).
Every experiment is the story's SCENARIO proposal: p = chance it gives a
usable result, value = what a usable result is worth to the roadmap (0-10),
cost = GPU-hours (thousands). Budget: 120k GPU-hours of research compute.
"""
from itertools import combinations
EXPS = [ # id, track, p, value, cost (k GPU-h)
("X1", "L4 skill retrieval v2", 0.6, 8, 30),
("X2", "L4 skill pruning", 0.7, 5, 10),
("X3", "L3 failure curriculum", 0.4, 7, 25),
("X4", "L2 harness search", 0.5, 6, 20),
("X5", "L1 eval sampling", 0.9, 2, 5),
("X6", "L5 sandbox: self-edited search", 0.15, 9, 40),
("X7", "env generator (Part 8)", 0.5, 8, 35),
("X8", "long-context skills", 0.3, 6, 45),
]
BUDGET = 120
def main():
best, bv = (), 0.0
for k in range(1, len(EXPS) + 1):
for c in combinations(EXPS, k):
cost = sum(e[4] for e in c)
ev = sum(e[2] * e[3] for e in c)
if cost <= BUDGET and ev > bv:
best, bv = c, ev
print(f"{'id':3} {'experiment':30} {'EV':>5} {'k GPU-h':>8} {'EV/k':>6}")
for e in sorted(EXPS, key=lambda e: -e[2] * e[3] / e[4]):
mark = "*" if e in best else " "
print(f"{e[0]:3} {e[1]:30} {e[2] * e[3]:5.2f} {e[4]:8} {e[2] * e[3] / e[4]:6.3f} {mark}")
print(f"chosen (*): {', '.join(e[0] for e in best)} cost {sum(e[4] for e in best)}k of {BUDGET}k EV {bv:.2f}")
if __name__ == "__main__":
main()$ python3 experiment_portfolio.py # exact search over all 255 subsets
id experiment EV k GPU-h EV/k
X5 L1 eval sampling 1.80 5 0.360
X2 L4 skill pruning 3.50 10 0.350 *
X1 L4 skill retrieval v2 4.80 30 0.160 *
X4 L2 harness search 3.00 20 0.150 *
X7 env generator (Part 8) 4.00 35 0.114 *
X3 L3 failure curriculum 2.80 25 0.112 *
X8 long-context skills 1.80 45 0.040
X6 L5 sandbox: self-edited search 1.35 40 0.034
chosen (*): X1, X2, X3, X4, X7 cost 120k of 120k EV 18.10
Expected value per thousand GPU-hours
Real output; โ = chosen. The L5 experiment loses on value per GPU-hour, not on a rule.
๐ Go deeper: why the best ratio (X5) was left out
- Ratios are a guide, not the answer.
- The exact search fills the budget to 120k with five experiments worth 18.10; adding X5 would force out a bigger, better experiment.
- It still ran.
- When X7 died at the toy rung and freed compute, X5 joined (see the ladder).
4. The evidence ladder: die early, die cheap
Four rungs, each with a bar written in advance
๐งธToy ยท 2%bar 0.5 pts
๐ฌSmall ยท 10%bar 1.0
๐งชMedium ยท 30%bar 1.5
๐Full ยท 100%bar 2.0
Season 1, Part 2's method at team scale.
kill_ladder.py ยท Python 3, no dependencies; scenario results
"""kill_ladder.py โ run experiments up an evidence ladder and stop early.
Rungs (Season 1, Part 2's method at team scale): toy (2%) -> small (10%) ->
medium (30%) -> full (100%) of the experiment's compute. Each rung has a
pre-registered bar; an experiment that misses a bar is killed there.
Results are SCENARIO numbers: the measured gain (points) at each rung.
The chosen five ran; X5 joined after X7 was killed and freed compute.
"""
RUNGS = [("toy", 0.02, 0.5), ("small", 0.10, 1.0), ("medium", 0.30, 1.5), ("full", 1.00, 2.0)]
RESULTS = { # experiment: (k GPU-h if run to full, gains at each rung)
"X1": (30, [1.1, 1.6, 2.2, 2.6]), "X2": (10, [0.8, 1.2, 1.4, 1.5]),
"X3": (25, [0.9, 0.6, None, None]), "X4": (20, [0.7, 1.3, 1.9, 2.3]),
"X5": (5, [0.6, 1.1, 1.6, 2.1]), "X7": (35, [0.3, None, None, None]),
}
def main():
spent = full = 0.0
for x, (cost, gains) in RESULTS.items():
full += cost
for (name, frac, bar), g in zip(RUNGS, gains):
spent += cost * frac
if g is None or g < bar:
print(f"{x}: killed at {name:6} (gain {g if g is not None else 0:.1f} < bar {bar}) after {cost * frac:5.1f}k GPU-h")
break
else:
print(f"{x}: passed all rungs, {cost}k GPU-h")
print(f"compute used {spent:.1f}k of {full:.0f}k if everything ran to full ({1 - spent / full:.0%} saved)")
if __name__ == "__main__":
main()$ python3 kill_ladder.py # six experiments that ran
X1: passed all rungs, 30k GPU-h
X2: killed at medium (gain 1.4 < bar 1.5) after 3.0k GPU-h
X3: killed at small (gain 0.6 < bar 1.0) after 2.5k GPU-h
X4: passed all rungs, 20k GPU-h
X5: passed all rungs, 5k GPU-h
X7: killed at toy (gain 0.3 < bar 0.5) after 0.7k GPU-h
compute used 86.0k of 125k if everything ran to full (31% saved)
What the ladder did this quarter
3 of 6killed before full scale
0.7kGPU-hours to kill X7
31%compute saved
3results sent to the gate
Real output of kill_ladder.py.
5. Sandbox: you pick the experiments
๐ฎ Sandbox: pick this quarter's experiments
Same numbers as experiment_portfolio.py; scenario estimates.
- Try this.
- Tick X6 (the L5 experiment): its expected value per GPU-hour is the lowest of all.
- Drop the budget to 60k: which two experiments would you keep?
- Raise it to 210k: everything fits, and the value barely rises at the end.
6. Week 115: the environment generator dies (scenario)
Scenario
- X7, an automatic generator of task environments, gains only 0.3 points at the toy rung against a bar of 0.5. It is killed after 0.7k GPU-hours.
- Why it matters.
- The idea (environments as a moat) is still strong; this way of building them is not.
- Part 8 builds environments differently: semi-automatically, with every one checked against cheating agents.
7. Five mistakes
- Funding experiments by who argues best.
- Writing kill bars after seeing the first results.
- Skipping the toy rung for "obvious" ideas.
- Treating a passed ladder as permission to ship.
- Keeping a killed experiment alive "for one more week".
8. Delegation box
๐ Gopala himself
- The budget split and kill-bar policy
- Final say on contested kills
๐ค AI agents
- Running the portfolio and ladder scripts every week
- Drafting experiment reports for researchers to check
๐ค Bought, rented or borrowed
- External reviewers for disputed results
9. Budget and risk tracker
| Item | Status after Part 6 | Note |
|---|---|---|
| Experiments | 5 chosen, 6 run | From the scripts |
| Compute | 86k of 125k used | 31% saved by early kills |
| Research speed | Better | More ideas per quarter |
| Environment plan | Watch | Generator v1 killed |
10. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
VerdictMany cheap experiments beat a few expensive ones: score by value per GPU-hour, fill the budget with an exact search, set the bars first, and let the ladder kill most ideas before they cost much. Next: which of the surviving directions could become the next moat.
11. Reading the numbers
- Primary / standard
- Method from Season 1, Parts 1โ2; knapsack and staged testing are standard techniques.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- All experiments, probabilities, values, costs, bars and results are the story's scenario.