The Research Accelerators: Engineers Who Build the Tools for Thousands of Experiments
Behind every frontier model are thousands of smaller experiments: data-mix ablations, scaling-law sweeps, architecture tweaks, and evaluations of every checkpoint. The scarcest resource in a lab is not GPUs but researchers' attention, so a whole layer of the factory exists to make experiments fast to launch, easy to compare and hard to get wrong. This part follows the research infrastructure engineers who build that layer: launchers, experiment tracking, dashboards, evaluation harnesses and leaderboards.
1. Where we are in the stack
The AI factory stack, by role
2. The role: research infrastructure engineer
Thinking Machines Lab's postings describe the job clearly. Its Research Infrastructure Engineer, Research Acceleration (February 2026) builds the libraries and tools that speed up internal research, including evaluation frameworks, reinforcement-learning training systems and experiment tracking, with a listed range of $350,000โ$475,000. Its Software Engineer, Research Tools (August 2026) maintains the platform researchers use to monitor training runs, debug model outputs and compare results on shared leaderboards, listed at $300,000โ$475,000. The pay signals how much labs value making researchers faster.
What the role needs
Our reading of the postings (1โ5). The day timeline is a composite illustration.
3. Why research needs its own tooling
Research is a loop, and every turn of it costs researcher time. Good tooling removes friction at each step. Click a step.
The experiment loop
4. What lives at this layer
The research tooling stack
5. Experiment tracking
Every run logs its configuration, code version, data version, metrics and artefacts, so that results can be compared and reproduced months later. Weights & Biases became the best-known tool for this, with OpenAI and Meta among its reported users. In March 2025 the AI cloud provider CoreWeave agreed to acquire it, a deal The Information reported at about $1.7 billion (terms were not disclosed), completed around May 2025; CoreWeave said the platform would stay compatible with other clouds and frameworks. Open-source MLflow and in-house systems fill the same role elsewhere. The value is less the charts than the discipline: a run nobody can reproduce is a run that taught nothing.
Reading a run dashboard
Illustrative loss curves for four runs. Toggle smoothing and look for problems.
6. Evaluation infrastructure
Every checkpoint of every serious run gets evaluated automatically, on dozens of benchmarks, so researchers see regressions early (see Producing Intelligence Part 7). The tooling has converged on a few open frameworks. EleutherAI's lm-evaluation-harness is widely used for academic benchmarks. The UK AI Security Institute's Inspect, developed with Meridian Labs and released as open source under the MIT licence, targets frontier evaluations including agents and tools; its companion Inspect Evals offers 200+ ready-made evaluations, and AISI reports that Apollo Research dropped its own internal framework after a three-month trial of Inspect. A third-party guide lists METR, Apollo and several government testing bodies as users.
Checkpoint ร benchmark heatmap
Illustrative scores for six checkpoints of one run. A good eval dashboard makes regressions jump out.
7. Case study: Meta's FBLearner Flow and after
Meta has published less about its LLM research tooling than about its hardware, so this case study is brief. Its earlier platform, FBLearner Flow, began as a system for model training and experimentation and, according to an MLOps case study, grew over about four years to cover feature stores, lineage tracking and governance โ ๏ธ secondary. For today's large-model research, the pieces covered in this series fit together: MAST (Part 5) places jobs, PyTorch and TorchTitan (Part 7) run them.
8. Sweeps and the economics of experiments
Choosing a learning rate, batch size or data mix means running many small experiments. How you search matters as much as how much compute you have: grid search wastes runs on bad regions, random search covers important dimensions better, and methods like successive halving stop bad runs early. Every GPU-hour spent on a sweep is one not spent on the main run, which is why research tooling teams work closely with the people who allocate compute (see Compute Economics).
Sweep planner
Illustrative model for tuning two hyperparameters with a fixed GPU budget.
9. When it breaks
Research tooling detective
Pick a symptom.
10. Handoffs and careers
Who research tooling engineers work with
- Typical backgrounds
- Research engineers who built tools for their own team, MLOps/platform engineers, full-stack engineers with ML interest, evaluation researchers.
- Core skills
- Python, distributed systems basics, data and metrics pipelines, web dashboards, evaluation design, empathy for researchers' workflows, ruthless attention to reproducibility.
- Growth path
- Research tools engineer โ owner of the eval or experiment platform โ head of research infrastructure.
- Further reading
- From Idea to Merged PR ยท Is It Ready? ยท Compute Economics
๐๏ธ Case file
- Meta
- FBLearner Flow grew from training platform to end-to-end ML platform โ ๏ธ secondary; LLM research runs on MAST, PyTorch and TorchTitan.
- Industry
- Weights & Biases (CoreWeave, 2025) for tracking; lm-evaluation-harness and UK AISI's Inspect for evaluation.
- Roles
- Thinking Machines Lab: research acceleration and research tools teams, $300โ475K listed.
๐ Next in the series
Part 10: Inference platform โ the inference engineers who turn trained models into fast, cheap, reliable services.
๐ข Reading the numbers
- CoreWeaveโW&B: announcement March 2025, completion reported around May 2025; the ~$1.7B price is The Information's report, not disclosed.
- Inspect facts come from UK AISI's documentation and blog; the user list is partly from a third-party guide.
- FBLearner Flow history comes from a ZenML MLOps case study โ ๏ธ secondary.
- Role descriptions and pay come from Thinking Machines Lab postings.
- The loop diagram, dashboard, heatmap and sweep planner are illustrations.
๐ References
- Tools
- TechCrunch: CoreWeave acquires AI developer platform Weights & Biases
- CoreWeave announcement
- UK AISI: Inspect Evals ยท AISI engineering playbook
- ZenML: Evolving FBLearner Flow (secondary)
- Roles
- Thinking Machines Lab: Research Infrastructure Engineer, Research Acceleration (posting)
- Thinking Machines Lab: Software Engineer, Research Tools (posting)
- Related on this site
- Is It Ready? ยท Part 8: Data platform ยท From Idea to Merged PR