Proving It Small
A toy on a laptop is not evidence about real models. Before he raises money or rents a cluster, Gopala needs two things: a training stack he trusts, with loss that falls predictably as models grow, and a fair test of whether verified skills help a real language model. This part builds both. The trainer and the scaling-law fit are real code with real output from a laptop CPU; the real-model experiments are written as runnable scripts with fixed protocols and decision rules, priced at a few thousand dollars of rented GPU time.
train_tiny.py, the four-model ladder, the scaling-law fit and every printed output in sections 3โ5 were run for this article on a laptop-class CPU (PyTorch 2.15). Runnable but not run here: skill_eval.py and lora_skills.py need a GPU and a model server; they were syntax-checked, not executed, and their results are not reported as facts. Where the story says what Gopala's real-model runs found, that is the scenario, marked as such. Prices are assumptions. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 4โ8
- Writes a 150-line GPT trainer from scratch rather than starting from a big framework, so he understands every line he will later scale (section 3).
- Smoke-tests it on his laptop; finds and fixes an initialisation bug that made the first loss value absurd.
- Trains four sizes on the same data, fits a scaling law, and checks the fit predicts the largest model from the three smaller ones (sections 4โ5).
- Rents one GPU for two days to repeat the ladder at 10โ100ร the size (section 6).
- Builds a 300-task suite from design-partner-style problems, split by repository so test tasks never share code with skill sources.
- Rents one 8-GPU node in short bursts: serves an open 7โ8B model with vLLM, runs the retrieval test (H1) and the LoRA comparison (H2).
- Applies the decision rules written before the runs (sections 7โ9).
2. Two questions
Two questions, two kinds of evidence
Phase 0 must answer both cheaply before any large money is spent.
- Both tracks are cheap on purpose.
- The start-up playbook makes the point that experiments usually cost several times the final run
- the cheapest experiments are the ones that kill bad ideas early.
- The start-up playbook makes the point that experiments usually cost several times the final run
3. A trainer you trust
- The trainer is deliberately small: a GPT-style decoder (pre-norm blocks, causal attention through PyTorch's fused
scaled_dot_product_attention, which uses FlashAttention-style kernels on a GPU), byte-level tokens so no tokenizer is needed yet, AdamW, a warmup-then-cosine learning-rate schedule, gradient clipping, bf16 on GPUs, periodic validation and checkpoint-and-resume.- Those last features look like luxuries on a laptop
- on a cluster they are what separates a lost hour from a lost week (running the Llama 3 run).
- Those last features look like luxuries on a laptop
"""train_tiny.py โ a minimal GPT trainer for the Maestro's small-scale proofs.
The same file trains a 0.1M-parameter smoke-test model on a laptop CPU and,
unchanged, a 125Mโ1B model on one rented GPU (multi-GPU comes in Part 9).
Features that matter later: cosine schedule with warmup, bf16 autocast, gradient
clipping, periodic eval, and checkpoint/resume so a crashed run loses minutes.
python3 train_tiny.py --size xs --steps 400 # CPU smoke test
python3 train_tiny.py --size 125M --steps 20000 --bs 64 --ctx 1024 --lr 6e-4 # one GPU
"""
import argparse, glob, math, os, time
import torch
import torch.nn as nn
import torch.nn.functional as F
SIZES = { # name: (layers, heads, d_model) โ params grow ~12ยทLยทdยฒ
"xs": (2, 2, 64), "s": (4, 4, 128), "sm": (4, 4, 192), "m": (6, 8, 256),
"125M": (12, 12, 768), "350M": (24, 16, 1024), "1B": (24, 16, 2048),
}
class Block(nn.Module):
def __init__(self, d, h):
super().__init__()
self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
self.qkv, self.proj, self.h = nn.Linear(d, 3 * d), nn.Linear(d, d), h
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
def forward(self, x):
B, T, C = x.shape
q, k, v = self.qkv(self.ln1(x)).split(C, dim=2)
q, k, v = (t.view(B, T, self.h, C // self.h).transpose(1, 2) for t in (q, k, v))
y = F.scaled_dot_product_attention(q, k, v, is_causal=True) # FlashAttention on GPU
x = x + self.proj(y.transpose(1, 2).reshape(B, T, C))
return x + self.mlp(self.ln2(x))
class GPT(nn.Module):
def __init__(self, vocab, ctx, L, H, D):
super().__init__()
self.tok, self.pos = nn.Embedding(vocab, D), nn.Embedding(ctx, D)
self.blocks = nn.ModuleList(Block(D, H) for _ in range(L))
self.ln, self.head = nn.LayerNorm(D), nn.Linear(D, vocab, bias=False)
self.head.weight = self.tok.weight # weight tying
self.apply(self._init)
@staticmethod
def _init(m): # GPT-2 style: small init
if isinstance(m, (nn.Linear, nn.Embedding)):
nn.init.normal_(m.weight, std=0.02)
if isinstance(m, nn.Linear) and m.bias is not None:
nn.init.zeros_(m.bias)
def forward(self, idx, targets=None):
x = self.tok(idx) + self.pos(torch.arange(idx.shape[1], device=idx.device))
for b in self.blocks:
x = b(x)
logits = self.head(self.ln(x))
loss = None if targets is None else F.cross_entropy(logits.flatten(0, 1), targets.flatten())
return logits, loss
def load_text():
"""Byte-level data: any local text. Default: the Python standard library source."""
root = os.path.dirname(os.__file__)
files = sorted(glob.glob(os.path.join(root, "*.py")))[:120]
data = b"".join(open(f, "rb").read() for f in files)
t = torch.tensor(list(data), dtype=torch.long)
n = int(0.95 * len(t))
return t[:n], t[n:]
def batch(data, bs, ctx, gen):
ix = torch.randint(len(data) - ctx - 1, (bs,), generator=gen)
x = torch.stack([data[i:i + ctx] for i in ix])
return x, torch.stack([data[i + 1:i + ctx + 1] for i in ix])
def lr_at(step, total, peak, warm):
if step < warm:
return peak * (step + 1) / warm
return peak * 0.5 * (1 + math.cos(math.pi * (step - warm) / max(1, total - warm)))
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--size", default="xs"); ap.add_argument("--steps", type=int, default=400)
ap.add_argument("--bs", type=int, default=16); ap.add_argument("--ctx", type=int, default=128)
ap.add_argument("--lr", type=float, default=3e-3); ap.add_argument("--ckpt", default="ckpt_tiny.pt")
ap.add_argument("--seed", type=int, default=0)
a = ap.parse_args()
torch.manual_seed(a.seed); gen = torch.Generator().manual_seed(a.seed)
dev = "cuda" if torch.cuda.is_available() else "cpu"
train, val = load_text()
L, H, D = SIZES[a.size]
model = GPT(256, a.ctx, L, H, D).to(dev)
n_params = sum(p.numel() for p in model.parameters()) - model.pos.weight.numel()
opt = torch.optim.AdamW(model.parameters(), lr=a.lr, betas=(0.9, 0.95), weight_decay=0.1)
start = 0
if os.path.exists(a.ckpt): # resume after a crash
s = torch.load(a.ckpt, map_location=dev)
model.load_state_dict(s["model"]); opt.load_state_dict(s["opt"]); start = s["step"]
tokens = a.steps * a.bs * a.ctx
print(f"size={a.size} params={n_params/1e6:.2f}M tokens={tokens/1e6:.2f}M "
f"computeโ{6*n_params*tokens:.2e} FLOPs device={dev}")
t0 = time.time()
for step in range(start, a.steps):
for g in opt.param_groups:
g["lr"] = lr_at(step, a.steps, a.lr, warm=max(1, a.steps // 20))
x, y = batch(train, a.bs, a.ctx, gen)
with torch.autocast(dev, dtype=torch.bfloat16, enabled=dev == "cuda"):
_, loss = model(x.to(dev), y.to(dev))
opt.zero_grad(set_to_none=True); loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
if step % 100 == 0 or step == a.steps - 1:
model.eval()
with torch.no_grad():
vl = sum(model(*(t.to(dev) for t in batch(val, a.bs, a.ctx, gen)))[1].item() for _ in range(8)) / 8
model.train()
print(f"step {step:5d} train {loss.item():.3f} val {vl:.3f} ({time.time()-t0:.0f}s)")
torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "step": step + 1}, a.ckpt)
print(f"final_val {vl:.4f} params {n_params}")
if __name__ == "__main__":
main()- The first smoke test printed an initial loss of about 33, far above the ~5.5 expected for random guessing over 256 byte values (ln 256 โ 5.55).
- The cause: PyTorch's default embedding initialisation combined with tied input and output weights produced huge logits.
- The fix is the GPT-2-style small initialisation in
_init.
After it, the smoke test behaves:
4. The model ladder
The model ladder: final validation loss (lower is better)
Real output. The biggest model was worse until its learning rate was lowered: one setting does not fit all sizes.
- Next, four sizes from about 0.1M to 4.8M parameters, each trained for 1,500 steps on the same 3.1M tokens of Python source, with the same learning-rate recipe.
- Same data, same steps: the only thing that changes is the model size, so the curve isolates the effect of size.
#!/bin/sh
# ladder.sh โ train four sizes on the same data budget, collect "params loss" for scaling_fit.py
rm -f ladder.txt ckpt_*.pt
for size in xs s sm m; do
python3 train_tiny.py --size $size --steps 1500 --ckpt ckpt_$size.pt | tee log_$size.txt
grep final_val log_$size.txt | awk '{print $4, $2}' >> ladder.txt
done
python3 scaling_fit.py ladder.txt5. Fitting a scaling law
What the scaling-law fit says
Real output of scaling_fit.py. Fitting on three points over-predicted the gain: small ladders extrapolate badly.
- Scaling laws describe how loss falls as a power of model size, approaching a floor: L(N) = E + AยทNโฮฑ.
- Fitting one to small runs is how labs decide how big to go before spending real money (planning the run).
Gopala's fit uses only numpy:
"""scaling_fit.py โ fit L(N) = E + AยทN^(-alpha) to a ladder of small runs.
Input: one line per run, "params final_val_loss" (the last line printed by
train_tiny.py). Output: the fitted law and a prediction for bigger models.
Uses only numpy: a grid over (E, alpha) with least squares for A.
python3 scaling_fit.py ladder.txt
"""
import sys
import numpy as np
rows = [tuple(map(float, l.split())) for l in open(sys.argv[1]) if l.strip() and not l.startswith("#")]
N, Lv = np.array([r[0] for r in rows]), np.array([r[1] for r in rows])
best = None
for E in np.linspace(0, Lv.min() * 0.98, 200): # irreducible loss below the best run
for alpha in np.linspace(0.05, 1.0, 200):
x = N ** -alpha
A = np.dot(x, Lv - E) / np.dot(x, x) # least squares for A given E, alpha
err = np.mean((E + A * x - Lv) ** 2)
if A > 0 and (best is None or err < best[0]):
best = (err, E, A, alpha)
err, E, A, alpha = best
print(f"fit: L(N) = {E:.3f} + {A:.2f} * N^-{alpha:.3f} (rmse {err**0.5:.4f} on {len(N)} runs)")
for n in N:
print(f" N={n:>10,.0f} measured {Lv[list(N).index(n)]:.3f} fitted {E + A * n ** -alpha:.3f}")
for n in (1e7, 1e8, 1e9):
print(f" predicted at N={n:.0e}: {E + A * n ** -alpha:.3f} (extrapolation: treat with care)")Measured loss and the fitted law
Real points from the ladder (log scale on size). The dashed line is the fit; beyond the last point it is extrapolation.
- Two lessons came out of this ladder, both of which repeat at every scale.
- First, one learning rate does not fit all sizes:
- with the same 3e-3 as the small models, the 4.8M model trained worse than the 1.8M one
- at 1e-3 it recovered.
- Larger models need smaller learning rates, which is why real scaling studies tune them per size or use parameterisations that transfer them.
- Second, the predict-then-check test failed usefully: fitted on the three smallest runs, the law predicted 1.41 for the largest model, which measured 1.50.
- The cause is data: every run saw the same 3.1M tokens, so beyond about 2M parameters extra size buys almost nothing, and the fitted floor of ~1.45 is a property of this tiny dataset, not of the models.
- Two cautions matter more than the numbers.
- First, these runs hold data fixed, so bigger models are increasingly data-starved; a real plan varies both size and tokens (the Chinchilla approach).
- Second, extrapolating two orders of magnitude from four points is risky; labs fit on dozens of runs spanning several orders of magnitude.
- The value here is the habit: predict, then check.
- Gopala refits with the three smallest runs and checks how well that predicts the fourth before he believes any extrapolation.
- First, one learning rate does not fit all sizes:
6. Moving to a real GPU
- The same script runs unchanged on one rented GPU at 125Mโ1B parameters (
--size 125M --ctx 1024 --bs 64), with a proper tokenizer and a slice of a public dataset such as FineWeb-Edu replacing the Python source.
What changes is not the code but the discipline:
| Concern | Laptop | One rented GPU |
|---|---|---|
| Data | Local Python files, bytes | Tokenized shards of a public corpus, streamed |
| Precision | FP32 | BF16 autocast (already in the script) |
| Throughput check | Seconds per step | Tokens per second and MFU: compare achieved FLOP/s (6ยทNยทtokens per second) with the GPU's peak |
| Failure handling | Rerun | Checkpoint every few hundred steps; resume automatically (already in the script) |
| Logging | print() | An experiment tracker; every run tagged with code version and config (research tooling) |
- Multi-GPU training (data, tensor and pipeline parallelism) waits for Part 9; for Phase 0, single-GPU runs are enough to validate the stack and the scaling behaviour.
7. H1 on a real model
- Part 1's toy showed the shape of H1.
- The real test uses an open 7โ8B instruction model served with vLLM on one GPU, and 300 engineering tasks written in the style of the design partners' work: fix this failing test, add this feature to this function, write this data transformation.
- Each has hidden tests.
- Tasks are split by repository: skills may only come from training repositories, and test tasks come from repositories the library never saw, so a skill cannot simply be the answer.
"""skill_eval.py โ H1 with a real model: does retrieving verified skills help?
Runs the same coding tasks twice against any OpenAI-compatible endpoint (for
example an open 7โ8B model served by vLLM on one rented GPU):
arm "base": task prompt only
arm "skills": task prompt + the k most similar verified skills from the library
Each answer is executed against the task's hidden tests in a subprocess with a
timeout. Same model, same sampling settings, same number of attempts per task.
vllm serve <open-model> --port 8000 # on the rented GPU
python3 skill_eval.py tasks.jsonl skills.jsonl --base-url http://localhost:8000/v1
tasks.jsonl : {"id", "prompt", "tests"} (tests = Python asserts)
skills.jsonl : {"name", "code", "doc", "tests"} (only skills that passed the gate)
NOTE: this script needs a running model server; it was not executed for this article.
"""
import argparse, json, re, subprocess, sys, tempfile
from collections import Counter
from openai import OpenAI # pip install openai
def tokens(s):
return Counter(re.findall(r"[a-z_]{3,}", s.lower()))
def top_k(task, skills, k=3):
"""Cheap lexical retrieval; swap in embeddings once the library is large."""
q = tokens(task["prompt"])
score = lambda s: sum((q & tokens(s["doc"] + " " + s["name"])).values())
return [s for s in sorted(skills, key=score, reverse=True)[:k] if score(s) > 0]
def extract_code(text):
m = re.search(r"```(?:python)?\n(.*?)```", text, re.S)
return m.group(1) if m else text
def run_tests(code, tests, timeout=10):
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
f.write(code + "\n\n" + tests + "\n")
try:
r = subprocess.run([sys.executable, f.name], capture_output=True, timeout=timeout)
return r.returncode == 0
except subprocess.TimeoutExpired:
return False
def ask(client, model, prompt, n):
out = client.chat.completions.create(model=model, n=n, temperature=0.6, max_tokens=1024,
messages=[{"role": "user", "content": prompt}])
return [extract_code(c.message.content) for c in out.choices]
def main():
ap = argparse.ArgumentParser()
ap.add_argument("tasks"); ap.add_argument("skills")
ap.add_argument("--base-url", default="http://localhost:8000/v1"); ap.add_argument("--model", default="default")
ap.add_argument("--attempts", type=int, default=4)
a = ap.parse_args()
client = OpenAI(base_url=a.base_url, api_key="local")
tasks = [json.loads(l) for l in open(a.tasks)]
skills = [json.loads(l) for l in open(a.skills)]
wins = {"base": 0, "skills": 0}
for t in tasks:
arms = {"base": t["prompt"]}
found = top_k(t, skills)
ctx = "\n\n".join(f"# Verified skill: {s['name']}\n# {s['doc']}\n{s['code']}" for s in found)
arms["skills"] = (f"These verified helper functions may be useful:\n```python\n{ctx}\n```\n\n" if found else "") + t["prompt"]
for arm, prompt in arms.items():
ok = any(run_tests(c, t["tests"]) for c in ask(client, a.model, prompt, a.attempts))
wins[arm] += ok
n = len(tasks)
print(f"pass@{a.attempts}: base {wins['base']}/{n} skills {wins['skills']}/{n}")
if __name__ == "__main__":
main()- Design choices worth copying: the two arms use the same model, the same sampling settings and the same number of attempts; correctness is decided by executing hidden tests, never by a judge model; and retrieval starts deliberately simple (word overlap) so that a positive result cannot be blamed on a clever retriever.
- Part 6 of the Agent Layer series explains why stuffing too many skills into the prompt backfires; the script caps it at three.
8. Fair statistics
Fair statistics: the paired test
Simulated self-check of paired_ci.py (not real model results). The interval excludes zero, so the rule says proceed.
- With 300 tasks, a few points of difference can be luck.
- Because both arms solve the same tasks, the right comparison is paired: task by task.
- Gopala writes his decision rule before running anything: proceed only if the 95% bootstrap interval for (skills โ base) lies entirely above zero.
"""paired_ci.py โ is the difference between two arms real, or luck?
Both arms run the SAME tasks, so compare them task by task (a paired test) and
bootstrap a confidence interval for the difference in success rate. Gopala's
rule: proceed only if the 95% interval for (skills โ base) is entirely above 0.
python3 paired_ci.py # runs a self-check on simulated outcomes
"""
import random
def paired_bootstrap(a, b, iters=10_000, seed=0):
"""a, b: lists of 0/1 outcomes per task for arm A and arm B (same order)."""
rng, n = random.Random(seed), len(a)
diffs = []
for _ in range(iters):
idx = [rng.randrange(n) for _ in range(n)]
diffs.append(sum(b[i] - a[i] for i in idx) / n)
diffs.sort()
return sum(y - x for x, y in zip(a, b)) / n, diffs[int(0.025 * iters)], diffs[int(0.975 * iters)]
if __name__ == "__main__":
# Self-check with SIMULATED outcomes (not results): 300 tasks, true rates 40% vs 47%.
rng = random.Random(1)
base = [int(rng.random() < 0.40) for _ in range(300)]
skills = [max(x, int(rng.random() < 0.12)) for x in base] # skills help some tasks, rarely hurt
d, lo, hi = paired_bootstrap(base, skills)
print(f"simulated: base {sum(base)}/300, skills {sum(skills)}/300, diff {d:+.3f}, 95% CI [{lo:+.3f}, {hi:+.3f}]")
print("decision:", "proceed" if lo > 0 else "not proven")- The self-check uses simulated outcomes to show the method works; it is not a result.
- With real outcomes, the same function decides the experiment.
| Outcome (written in advance) | Decision |
|---|---|
| Interval entirely above 0, gain โฅ 5 points | Proceed to H2 and to planning (Part 3) |
| Interval above 0, gain under 5 points | Proceed cautiously; improve retrieval and skill quality before raising |
| Interval includes 0 | Not proven: enlarge the suite or rethink the skill format; do not raise on this |
| Interval below 0 | Stop: skills hurt real models, as hoarding hurt the toy |
9. H2: into the weights
- H1 shows skills help when they are in the prompt.
- H2 asks whether the model can carry them in its weights.
- Two LoRA adapters on the same base, trained for the same number of examples: one on verified skills turned into task-and-solution pairs, one on an equal amount of high-quality public code.
- Then both are evaluated without retrieval on the held-out tasks.
"""lora_skills.py โ H2 preview: fold verified skills into weights with LoRA.
Two adapters on the same open base model, trained for the SAME number of tokens:
--data skills verified skills turned into (task โ solution) chat examples
--data generic an equal-size sample of high-quality public code data
Then rerun skill_eval.py WITHOUT retrieval against each adapter: if the skills
adapter wins on held-out tasks, weights absorbed something retrieval provided.
pip install transformers peft trl datasets
python3 lora_skills.py --base <open-model> --data skills --out adapter-skills
python3 lora_skills.py --base <open-model> --data generic --out adapter-generic
NOTE: needs a GPU (one 80 GB card is enough for a 7โ8B model with LoRA);
it was not executed for this article.
"""
import argparse, json
from datasets import Dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
def skill_examples(path):
for line in open(path):
s = json.loads(line) # {"name", "code", "doc", "tests", "task"}
yield {"messages": [
{"role": "user", "content": s["task"]},
{"role": "assistant", "content": f"```python\n{s['code']}\n```"},
]}
def generic_examples(path, limit):
for i, line in enumerate(open(path)): # {"prompt", "completion"} from a public code set
if i >= limit:
break
r = json.loads(line)
yield {"messages": [{"role": "user", "content": r["prompt"]},
{"role": "assistant", "content": r["completion"]}]}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--base", required=True); ap.add_argument("--data", choices=["skills", "generic"], required=True)
ap.add_argument("--skills", default="skills_train.jsonl"); ap.add_argument("--generic", default="generic_code.jsonl")
ap.add_argument("--out", required=True)
a = ap.parse_args()
skills = list(skill_examples(a.skills))
rows = skills if a.data == "skills" else list(generic_examples(a.generic, len(skills)))
trainer = SFTTrainer(
model=a.base,
train_dataset=Dataset.from_list(rows),
peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM"),
args=SFTConfig(output_dir=a.out, num_train_epochs=2, per_device_train_batch_size=4,
gradient_accumulation_steps=8, learning_rate=1e-4, lr_scheduler_type="cosine",
warmup_ratio=0.03, bf16=True, logging_steps=10, save_strategy="epoch", seed=0),
)
trainer.train()
trainer.save_model(a.out)
print(f"saved {a.data} adapter to {a.out} ({len(rows)} examples)")
if __name__ == "__main__":
main()- Why LoRA here and not full fine-tuning: it trains a few million parameters instead of billions, fits on one GPU, and makes the comparison cheap to repeat with different seeds.
- The full version of this experiment, folding skills into a model during pre-training and post-training, is Phase 2's job (Parts 10โ11).
10. What Gopala found (scenario)
- This section continues the story.
- The outcomes below are what the scenario assumes so the series can proceed; they are not results of running the scripts above.
- Run them yourself to get real numbers.
- In the scenario, H1 lands in the first row of the decision table: retrieval of verified skills improves pass@4 on held-out repositories, with a paired interval clearly above zero.
- H2 lands in the second row: the skills adapter beats the generic-code adapter on held-out tasks, but by less, and it slightly weakens performance on a general coding benchmark, which is exactly what the gate's no-regression check exists to catch.
- The ladder and scaling fit behave as expected.
- Gopala's conclusion: the effect is real enough to plan around, the gate is not optional, and how skills enter the weights will need care at scale.
11. What it cost
What Phase 0 experiments cost
Small proofs are cheap. The expensive part is the time to get them right.
Phase 0 experiment budget
Our model; GPU price is an on-demand assumption.
- A few thousand dollars buys the evidence that decides whether to raise twenty million.
- That ratio is why Phase 0 exists.
12. Delegation box
- The trainer and its debugging
- Task-suite design and the repository split
- Decision rules, written before the runs
- Drafting tasks and hidden tests (every one reviewed)
- Boilerplate: data loaders, plotting, logging
- Running and re-running sweeps overnight
- One GPU and one 8-GPU node, on demand
- Open-weight model, vLLM, TRL/PEFT (open source)
- Two engineers from design partners reviewing tasks
13. Budget and risk tracker
| Item | Status after Part 2 | Note |
|---|---|---|
| Spend to date | ~$2โ3k (our estimate) | Rented GPUs plus Part 1's API credits |
| Time | 8 of ~78 weeks | Overlaps with Part 3's planning |
| Can he train? | Yes, at small scale | Multi-GPU in Part 9 |
| H1 (skills help) | Supported (scenario) | Real runs needed before real money |
| H2 (weights absorb skills) | Weakly supported, with a regression (scenario) | Gate's no-regression check is essential |
| Evaluation validity | Repository split in place | Contamination checks for public data in Phase 3 |
14. The series
The Maestro: 18 parts in five phases
- Track A works: a 150-line trainer trains, resumes and scales predictably across a four-size ladder, and the fitted law gives a first, cautious handle on how big to go.
- Track B has runnable, pre-registered tests for both hypotheses, with paired statistics and decision rules written before the data.
- The scenario's results are good enough to plan around and honest enough to show where the gate matters.
- Next: turning evidence into a budget and a raise in Part 3, the plan and the money.
15. Reading the numbers
- Real (run for this article)
- Smoke test, ladder, scaling fit and the paired-CI self-check (simulated inputs, real method). CPU runs; your numbers will differ slightly with hardware and library versions.
- Not run
skill_eval.py,lora_skills.py(need GPUs and a model server).- Scenario
- Section 10's outcomes.
- Standard references
- Scaling-law form (Kaplan et al. 2020; Hoffmann et al. 2022); LoRA (Hu et al. 2021); GPT-2 initialisation; FineWeb-Edu (Hugging Face).