Home โ€บ Blog โ€บ The Maestro ยท Part 2
The Maestro ยท Part 2 ยท Phase 0 ยท ๐Ÿ”ฌ Researcher + ๐Ÿง  ML engineer hats

Proving It Small

A toy on a laptop is not evidence about real models. Before he raises money or rents a cluster, Gopala needs two things: a training stack he trusts, with loss that falls predictably as models grow, and a fair test of whether verified skills help a real language model. This part builds both. The trainer and the scaling-law fit are real code with real output from a laptop CPU; the real-model experiments are written as runnable scripts with fixed protocols and decision rules, priced at a few thousand dollars of rented GPU time.

FL
FrontierAGI Team
๐Ÿ”ฌ Researcher๐Ÿง  ML engineer๐Ÿ“ Evaluator
โš ๏ธ What is real and what is scenario Real: train_tiny.py, the four-model ladder, the scaling-law fit and every printed output in sections 3โ€“5 were run for this article on a laptop-class CPU (PyTorch 2.15). Runnable but not run here: skill_eval.py and lora_skills.py need a GPU and a model server; they were syntax-checked, not executed, and their results are not reported as facts. Where the story says what Gopala's real-model runs found, that is the scenario, marked as such. Prices are assumptions. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 4โ€“8

Week 4 ยท The trainer
  • Writes a 150-line GPT trainer from scratch rather than starting from a big framework, so he understands every line he will later scale (section 3).
  • Smoke-tests it on his laptop; finds and fixes an initialisation bug that made the first loss value absurd.
Week 5 ยท The ladder
  • Trains four sizes on the same data, fits a scaling law, and checks the fit predicts the largest model from the three smaller ones (sections 4โ€“5).
  • Rents one GPU for two days to repeat the ladder at 10โ€“100ร— the size (section 6).
Weeks 6โ€“8 ยท The real test (in parallel with Part 3's planning)
  • Builds a 300-task suite from design-partner-style problems, split by repository so test tasks never share code with skill sources.
  • Rents one 8-GPU node in short bursts: serves an open 7โ€“8B model with vLLM, runs the retrieval test (H1) and the LoRA comparison (H2).
  • Applies the decision rules written before the runs (sections 7โ€“9).

2. Two questions

Two questions, two kinds of evidence

โ“H1Do curated skills help a real model at inference time?
๐ŸงชTestPaired comparison on the same tasks, with a confidence interval
โ“H2Do verified skills help when trained into the weights?
๐ŸงชTestLoRA fine-tune vs base on held-out tasks

Phase 0 must answer both cheaply before any large money is spent.

Track A ยท Can he train?Does his own training code work, and does loss fall predictably with model size? Without this, every later decision about model size is a guess.
Track B ยท Does the idea hold?Do verified skills help a real language model on held-out tasks (H1)? Does folding them into weights beat generic data (H2)?

3. A trainer you trust

train_tiny.py ยท PyTorch ยท CPU or one GPU
"""train_tiny.py โ€” a minimal GPT trainer for the Maestro's small-scale proofs.

The same file trains a 0.1M-parameter smoke-test model on a laptop CPU and,
unchanged, a 125Mโ€“1B model on one rented GPU (multi-GPU comes in Part 9).
Features that matter later: cosine schedule with warmup, bf16 autocast, gradient
clipping, periodic eval, and checkpoint/resume so a crashed run loses minutes.

  python3 train_tiny.py --size xs --steps 400          # CPU smoke test
  python3 train_tiny.py --size 125M --steps 20000 --bs 64 --ctx 1024 --lr 6e-4   # one GPU
"""
import argparse, glob, math, os, time
import torch
import torch.nn as nn
import torch.nn.functional as F

SIZES = {  # name: (layers, heads, d_model) โ€” params grow ~12ยทLยทdยฒ
    "xs": (2, 2, 64), "s": (4, 4, 128), "sm": (4, 4, 192), "m": (6, 8, 256),
    "125M": (12, 12, 768), "350M": (24, 16, 1024), "1B": (24, 16, 2048),
}


class Block(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj, self.h = nn.Linear(d, 3 * d), nn.Linear(d, d), h
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x):
        B, T, C = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(C, dim=2)
        q, k, v = (t.view(B, T, self.h, C // self.h).transpose(1, 2) for t in (q, k, v))
        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)   # FlashAttention on GPU
        x = x + self.proj(y.transpose(1, 2).reshape(B, T, C))
        return x + self.mlp(self.ln2(x))


class GPT(nn.Module):
    def __init__(self, vocab, ctx, L, H, D):
        super().__init__()
        self.tok, self.pos = nn.Embedding(vocab, D), nn.Embedding(ctx, D)
        self.blocks = nn.ModuleList(Block(D, H) for _ in range(L))
        self.ln, self.head = nn.LayerNorm(D), nn.Linear(D, vocab, bias=False)
        self.head.weight = self.tok.weight                            # weight tying
        self.apply(self._init)

    @staticmethod
    def _init(m):                                                      # GPT-2 style: small init
        if isinstance(m, (nn.Linear, nn.Embedding)):
            nn.init.normal_(m.weight, std=0.02)
        if isinstance(m, nn.Linear) and m.bias is not None:
            nn.init.zeros_(m.bias)

    def forward(self, idx, targets=None):
        x = self.tok(idx) + self.pos(torch.arange(idx.shape[1], device=idx.device))
        for b in self.blocks:
            x = b(x)
        logits = self.head(self.ln(x))
        loss = None if targets is None else F.cross_entropy(logits.flatten(0, 1), targets.flatten())
        return logits, loss


def load_text():
    """Byte-level data: any local text. Default: the Python standard library source."""
    root = os.path.dirname(os.__file__)
    files = sorted(glob.glob(os.path.join(root, "*.py")))[:120]
    data = b"".join(open(f, "rb").read() for f in files)
    t = torch.tensor(list(data), dtype=torch.long)
    n = int(0.95 * len(t))
    return t[:n], t[n:]


def batch(data, bs, ctx, gen):
    ix = torch.randint(len(data) - ctx - 1, (bs,), generator=gen)
    x = torch.stack([data[i:i + ctx] for i in ix])
    return x, torch.stack([data[i + 1:i + ctx + 1] for i in ix])


def lr_at(step, total, peak, warm):
    if step < warm:
        return peak * (step + 1) / warm
    return peak * 0.5 * (1 + math.cos(math.pi * (step - warm) / max(1, total - warm)))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--size", default="xs"); ap.add_argument("--steps", type=int, default=400)
    ap.add_argument("--bs", type=int, default=16); ap.add_argument("--ctx", type=int, default=128)
    ap.add_argument("--lr", type=float, default=3e-3); ap.add_argument("--ckpt", default="ckpt_tiny.pt")
    ap.add_argument("--seed", type=int, default=0)
    a = ap.parse_args()
    torch.manual_seed(a.seed); gen = torch.Generator().manual_seed(a.seed)
    dev = "cuda" if torch.cuda.is_available() else "cpu"
    train, val = load_text()
    L, H, D = SIZES[a.size]
    model = GPT(256, a.ctx, L, H, D).to(dev)
    n_params = sum(p.numel() for p in model.parameters()) - model.pos.weight.numel()
    opt = torch.optim.AdamW(model.parameters(), lr=a.lr, betas=(0.9, 0.95), weight_decay=0.1)
    start = 0
    if os.path.exists(a.ckpt):                                        # resume after a crash
        s = torch.load(a.ckpt, map_location=dev)
        model.load_state_dict(s["model"]); opt.load_state_dict(s["opt"]); start = s["step"]
    tokens = a.steps * a.bs * a.ctx
    print(f"size={a.size} params={n_params/1e6:.2f}M tokens={tokens/1e6:.2f}M "
          f"computeโ‰ˆ{6*n_params*tokens:.2e} FLOPs device={dev}")
    t0 = time.time()
    for step in range(start, a.steps):
        for g in opt.param_groups:
            g["lr"] = lr_at(step, a.steps, a.lr, warm=max(1, a.steps // 20))
        x, y = batch(train, a.bs, a.ctx, gen)
        with torch.autocast(dev, dtype=torch.bfloat16, enabled=dev == "cuda"):
            _, loss = model(x.to(dev), y.to(dev))
        opt.zero_grad(set_to_none=True); loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
        if step % 100 == 0 or step == a.steps - 1:
            model.eval()
            with torch.no_grad():
                vl = sum(model(*(t.to(dev) for t in batch(val, a.bs, a.ctx, gen)))[1].item() for _ in range(8)) / 8
            model.train()
            print(f"step {step:5d}  train {loss.item():.3f}  val {vl:.3f}  ({time.time()-t0:.0f}s)")
            torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "step": step + 1}, a.ckpt)
    print(f"final_val {vl:.4f} params {n_params}")


if __name__ == "__main__":
    main()

After it, the smoke test behaves:

$ python3 train_tiny.py --size xs --steps 400 size=xs params=0.12M tokens=0.82M computeโ‰ˆ5.73e+11 FLOPs device=cpu step 0 train 5.486 val 5.432 (0s) step 100 train 2.724 val 2.559 (1s) step 200 train 2.479 val 2.304 (3s) step 300 train 2.206 val 2.332 (4s) step 399 train 2.187 val 2.332 (5s) final_val 2.3317 params 116480

4. The model ladder

The model ladder: final validation loss (lower is better)

xs ยท 0.12M params1.810
s ยท 0.83M1.573
sm ยท 1.83M1.494
m ยท 4.80M, lr 3e-31.617 โœ—
m ยท 4.80M, lr 1e-31.499

Real output. The biggest model was worse until its learning rate was lowered: one setting does not fit all sizes.

ladder.sh
#!/bin/sh
# ladder.sh โ€” train four sizes on the same data budget, collect "params loss" for scaling_fit.py
rm -f ladder.txt ckpt_*.pt
for size in xs s sm m; do
  python3 train_tiny.py --size $size --steps 1500 --ckpt ckpt_$size.pt | tee log_$size.txt
  grep final_val log_$size.txt | awk '{print $4, $2}' >> ladder.txt
done
python3 scaling_fit.py ladder.txt
$ sh ladder.sh # same 3.07M tokens and 1,500 steps for every size, lr 3e-3 size=xs params=0.12M ... final_val 1.8097 size=s params=0.83M ... final_val 1.5734 size=sm params=1.83M ... final_val 1.4943 size=m params=4.80M ... final_val 1.6173 โ† the biggest model is WORSE $ python3 train_tiny.py --size m --steps 1500 --lr 1e-3 # rerun with a smaller learning rate size=m params=4.80M tokens=3.07M computeโ‰ˆ8.86e+13 FLOPs device=cpu ... final_val 1.4991

5. Fitting a scaling law

What the scaling-law fit says

1.450loss floor in the fit
โˆ’0.604exponent on parameters
1.410 vs 1.499predicted vs measured for the largest model
0.0147fit error (RMSE) on 4 runs

Real output of scaling_fit.py. Fitting on three points over-predicted the gain: small ladders extrapolate badly.

Gopala's fit uses only numpy:

scaling_fit.py ยท numpy
"""scaling_fit.py โ€” fit L(N) = E + AยทN^(-alpha) to a ladder of small runs.

Input: one line per run, "params final_val_loss" (the last line printed by
train_tiny.py). Output: the fitted law and a prediction for bigger models.
Uses only numpy: a grid over (E, alpha) with least squares for A.

  python3 scaling_fit.py ladder.txt
"""
import sys
import numpy as np

rows = [tuple(map(float, l.split())) for l in open(sys.argv[1]) if l.strip() and not l.startswith("#")]
N, Lv = np.array([r[0] for r in rows]), np.array([r[1] for r in rows])

best = None
for E in np.linspace(0, Lv.min() * 0.98, 200):          # irreducible loss below the best run
    for alpha in np.linspace(0.05, 1.0, 200):
        x = N ** -alpha
        A = np.dot(x, Lv - E) / np.dot(x, x)              # least squares for A given E, alpha
        err = np.mean((E + A * x - Lv) ** 2)
        if A > 0 and (best is None or err < best[0]):
            best = (err, E, A, alpha)

err, E, A, alpha = best
print(f"fit: L(N) = {E:.3f} + {A:.2f} * N^-{alpha:.3f}   (rmse {err**0.5:.4f} on {len(N)} runs)")
for n in N:
    print(f"  N={n:>10,.0f}  measured {Lv[list(N).index(n)]:.3f}  fitted {E + A * n ** -alpha:.3f}")
for n in (1e7, 1e8, 1e9):
    print(f"  predicted at N={n:.0e}: {E + A * n ** -alpha:.3f}  (extrapolation: treat with care)")
$ python3 scaling_fit.py ladder.txt # with the lr 1e-3 rerun for the largest model fit: L(N) = 1.450 + 413.06 * N^-0.604 (rmse 0.0147 on 4 runs) N= 116,480 measured 1.810 fitted 1.810 N= 826,112 measured 1.573 fitted 1.560 N= 1,828,992 measured 1.494 fitted 1.518 N= 4,804,608 measured 1.499 fitted 1.488 predicted at N=1e+07: 1.474 (extrapolation: treat with care) predicted at N=1e+08: 1.456 (extrapolation: treat with care) predicted at N=1e+09: 1.451 (extrapolation: treat with care) # predict-then-check: fit only the three smallest runs, predict the fourth 3-point fit predicts 1.410 at 4.8M parameters; measured 1.499

Measured loss and the fitted law

Real points from the ladder (log scale on size). The dashed line is the fit; beyond the last point it is extrapolation.

6. Moving to a real GPU

What changes is not the code but the discipline:

ConcernLaptopOne rented GPU
DataLocal Python files, bytesTokenized shards of a public corpus, streamed
PrecisionFP32BF16 autocast (already in the script)
Throughput checkSeconds per stepTokens per second and MFU: compare achieved FLOP/s (6ยทNยทtokens per second) with the GPU's peak
Failure handlingRerunCheckpoint every few hundred steps; resume automatically (already in the script)
Loggingprint()An experiment tracker; every run tagged with code version and config (research tooling)

7. H1 on a real model

skill_eval.py ยท needs a model server ยท not executed for this article
"""skill_eval.py โ€” H1 with a real model: does retrieving verified skills help?

Runs the same coding tasks twice against any OpenAI-compatible endpoint (for
example an open 7โ€“8B model served by vLLM on one rented GPU):
  arm "base":   task prompt only
  arm "skills": task prompt + the k most similar verified skills from the library
Each answer is executed against the task's hidden tests in a subprocess with a
timeout. Same model, same sampling settings, same number of attempts per task.

  vllm serve <open-model> --port 8000           # on the rented GPU
  python3 skill_eval.py tasks.jsonl skills.jsonl --base-url http://localhost:8000/v1

tasks.jsonl  : {"id", "prompt", "tests"}           (tests = Python asserts)
skills.jsonl : {"name", "code", "doc", "tests"}    (only skills that passed the gate)
NOTE: this script needs a running model server; it was not executed for this article.
"""
import argparse, json, re, subprocess, sys, tempfile
from collections import Counter
from openai import OpenAI          # pip install openai


def tokens(s):
    return Counter(re.findall(r"[a-z_]{3,}", s.lower()))


def top_k(task, skills, k=3):
    """Cheap lexical retrieval; swap in embeddings once the library is large."""
    q = tokens(task["prompt"])
    score = lambda s: sum((q & tokens(s["doc"] + " " + s["name"])).values())
    return [s for s in sorted(skills, key=score, reverse=True)[:k] if score(s) > 0]


def extract_code(text):
    m = re.search(r"```(?:python)?\n(.*?)```", text, re.S)
    return m.group(1) if m else text


def run_tests(code, tests, timeout=10):
    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
        f.write(code + "\n\n" + tests + "\n")
    try:
        r = subprocess.run([sys.executable, f.name], capture_output=True, timeout=timeout)
        return r.returncode == 0
    except subprocess.TimeoutExpired:
        return False


def ask(client, model, prompt, n):
    out = client.chat.completions.create(model=model, n=n, temperature=0.6, max_tokens=1024,
                                         messages=[{"role": "user", "content": prompt}])
    return [extract_code(c.message.content) for c in out.choices]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("tasks"); ap.add_argument("skills")
    ap.add_argument("--base-url", default="http://localhost:8000/v1"); ap.add_argument("--model", default="default")
    ap.add_argument("--attempts", type=int, default=4)
    a = ap.parse_args()
    client = OpenAI(base_url=a.base_url, api_key="local")
    tasks = [json.loads(l) for l in open(a.tasks)]
    skills = [json.loads(l) for l in open(a.skills)]
    wins = {"base": 0, "skills": 0}
    for t in tasks:
        arms = {"base": t["prompt"]}
        found = top_k(t, skills)
        ctx = "\n\n".join(f"# Verified skill: {s['name']}\n# {s['doc']}\n{s['code']}" for s in found)
        arms["skills"] = (f"These verified helper functions may be useful:\n```python\n{ctx}\n```\n\n" if found else "") + t["prompt"]
        for arm, prompt in arms.items():
            ok = any(run_tests(c, t["tests"]) for c in ask(client, a.model, prompt, a.attempts))
            wins[arm] += ok
    n = len(tasks)
    print(f"pass@{a.attempts}: base {wins['base']}/{n}  skills {wins['skills']}/{n}")


if __name__ == "__main__":
    main()

8. Fair statistics

Fair statistics: the paired test

124 / 300base model solves
140 / 300with skills
+0.053difference
[+0.030, +0.080]95% confidence interval

Simulated self-check of paired_ci.py (not real model results). The interval excludes zero, so the rule says proceed.

paired_ci.py ยท Python 3, no dependencies
"""paired_ci.py โ€” is the difference between two arms real, or luck?

Both arms run the SAME tasks, so compare them task by task (a paired test) and
bootstrap a confidence interval for the difference in success rate. Gopala's
rule: proceed only if the 95% interval for (skills โˆ’ base) is entirely above 0.

  python3 paired_ci.py            # runs a self-check on simulated outcomes
"""
import random


def paired_bootstrap(a, b, iters=10_000, seed=0):
    """a, b: lists of 0/1 outcomes per task for arm A and arm B (same order)."""
    rng, n = random.Random(seed), len(a)
    diffs = []
    for _ in range(iters):
        idx = [rng.randrange(n) for _ in range(n)]
        diffs.append(sum(b[i] - a[i] for i in idx) / n)
    diffs.sort()
    return sum(y - x for x, y in zip(a, b)) / n, diffs[int(0.025 * iters)], diffs[int(0.975 * iters)]


if __name__ == "__main__":
    # Self-check with SIMULATED outcomes (not results): 300 tasks, true rates 40% vs 47%.
    rng = random.Random(1)
    base = [int(rng.random() < 0.40) for _ in range(300)]
    skills = [max(x, int(rng.random() < 0.12)) for x in base]     # skills help some tasks, rarely hurt
    d, lo, hi = paired_bootstrap(base, skills)
    print(f"simulated: base {sum(base)}/300, skills {sum(skills)}/300, diff {d:+.3f}, 95% CI [{lo:+.3f}, {hi:+.3f}]")
    print("decision:", "proceed" if lo > 0 else "not proven")
$ python3 paired_ci.py simulated: base 124/300, skills 140/300, diff +0.053, 95% CI [+0.030, +0.080] decision: proceed
Outcome (written in advance)Decision
Interval entirely above 0, gain โ‰ฅ 5 pointsProceed to H2 and to planning (Part 3)
Interval above 0, gain under 5 pointsProceed cautiously; improve retrieval and skill quality before raising
Interval includes 0Not proven: enlarge the suite or rethink the skill format; do not raise on this
Interval below 0Stop: skills hurt real models, as hoarding hurt the toy

9. H2: into the weights

lora_skills.py ยท Hugging Face TRL + PEFT ยท needs a GPU ยท not executed for this article
"""lora_skills.py โ€” H2 preview: fold verified skills into weights with LoRA.

Two adapters on the same open base model, trained for the SAME number of tokens:
  --data skills   verified skills turned into (task โ†’ solution) chat examples
  --data generic  an equal-size sample of high-quality public code data
Then rerun skill_eval.py WITHOUT retrieval against each adapter: if the skills
adapter wins on held-out tasks, weights absorbed something retrieval provided.

  pip install transformers peft trl datasets
  python3 lora_skills.py --base <open-model> --data skills  --out adapter-skills
  python3 lora_skills.py --base <open-model> --data generic --out adapter-generic
NOTE: needs a GPU (one 80 GB card is enough for a 7โ€“8B model with LoRA);
it was not executed for this article.
"""
import argparse, json
from datasets import Dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer


def skill_examples(path):
    for line in open(path):
        s = json.loads(line)                       # {"name", "code", "doc", "tests", "task"}
        yield {"messages": [
            {"role": "user", "content": s["task"]},
            {"role": "assistant", "content": f"```python\n{s['code']}\n```"},
        ]}


def generic_examples(path, limit):
    for i, line in enumerate(open(path)):          # {"prompt", "completion"} from a public code set
        if i >= limit:
            break
        r = json.loads(line)
        yield {"messages": [{"role": "user", "content": r["prompt"]},
                            {"role": "assistant", "content": r["completion"]}]}


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", required=True); ap.add_argument("--data", choices=["skills", "generic"], required=True)
    ap.add_argument("--skills", default="skills_train.jsonl"); ap.add_argument("--generic", default="generic_code.jsonl")
    ap.add_argument("--out", required=True)
    a = ap.parse_args()
    skills = list(skill_examples(a.skills))
    rows = skills if a.data == "skills" else list(generic_examples(a.generic, len(skills)))
    trainer = SFTTrainer(
        model=a.base,
        train_dataset=Dataset.from_list(rows),
        peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM"),
        args=SFTConfig(output_dir=a.out, num_train_epochs=2, per_device_train_batch_size=4,
                       gradient_accumulation_steps=8, learning_rate=1e-4, lr_scheduler_type="cosine",
                       warmup_ratio=0.03, bf16=True, logging_steps=10, save_strategy="epoch", seed=0),
    )
    trainer.train()
    trainer.save_model(a.out)
    print(f"saved {a.data} adapter to {a.out} ({len(rows)} examples)")


if __name__ == "__main__":
    main()

10. What Gopala found (scenario)

Scenario, not measurement
  • This section continues the story.
    • The outcomes below are what the scenario assumes so the series can proceed; they are not results of running the scripts above.
    • Run them yourself to get real numbers.

11. What it cost

What Phase 0 experiments cost

๐Ÿ’ปLaptop CPUTrainer, ladder and fit: free
๐Ÿ–ฅ๏ธRented GPUsShort runs for H1/H2 (scenario)
๐Ÿ”ŒAPIsModel calls for skill extraction (scenario)

Small proofs are cheap. The expensive part is the time to get them right.

Phase 0 experiment budget

Our model; GPU price is an on-demand assumption.

12. Delegation box

๐Ÿ™‹ Gopala himself
  • The trainer and its debugging
  • Task-suite design and the repository split
  • Decision rules, written before the runs
๐Ÿค– AI agents
  • Drafting tasks and hidden tests (every one reviewed)
  • Boilerplate: data loaders, plotting, logging
  • Running and re-running sweeps overnight
๐Ÿค Bought, rented or borrowed
  • One GPU and one 8-GPU node, on demand
  • Open-weight model, vLLM, TRL/PEFT (open source)
  • Two engineers from design partners reviewing tasks

13. Budget and risk tracker

ItemStatus after Part 2Note
Spend to date~$2โ€“3k (our estimate)Rented GPUs plus Part 1's API credits
Time8 of ~78 weeksOverlaps with Part 3's planning
Can he train?Yes, at small scaleMulti-GPU in Part 9
H1 (skills help)Supported (scenario)Real runs needed before real money
H2 (weights absorb skills)Weakly supported, with a regression (scenario)Gate's no-regression check is essential
Evaluation validityRepository split in placeContamination checks for public data in Phase 3

14. The series

The Maestro: 18 parts in five phases

Verdict
  • Track A works: a 150-line trainer trains, resumes and scales predictably across a four-size ladder, and the fitted law gives a first, cautious handle on how big to go.
    • Track B has runnable, pre-registered tests for both hypotheses, with paired statistics and decision rules written before the data.
    • The scenario's results are good enough to plan around and honest enough to show where the gate matters.
    • Next: turning evidence into a budget and a raise in Part 3, the plan and the money.

15. Reading the numbers

Real (run for this article)
Smoke test, ladder, scaling fit and the paired-CI self-check (simulated inputs, real method). CPU runs; your numbers will differ slightly with hardware and library versions.
Not run
skill_eval.py, lora_skills.py (need GPUs and a model server).
Scenario
Section 10's outcomes.
Standard references
Scaling-law form (Kaplan et al. 2020; Hoffmann et al. 2022); LoRA (Hu et al. 2021); GPT-2 initialisation; FineWeb-Edu (Hugging Face).