Home โ€บ Blog โ€บ Research Tooling
Who Runs the AI Factory ยท Part 9 ยท Research tooling

The Research Accelerators: Engineers Who Build the Tools for Thousands of Experiments

Behind every frontier model are thousands of smaller experiments: data-mix ablations, scaling-law sweeps, architecture tweaks, and evaluations of every checkpoint. The scarcest resource in a lab is not GPUs but researchers' attention, so a whole layer of the factory exists to make experiments fast to launch, easy to compare and hard to get wrong. This part follows the research infrastructure engineers who build that layer: launchers, experiment tracking, dashboards, evaluation harnesses and leaderboards.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are coverage of CoreWeave's acquisition of Weights & Biases, the UK AI Security Institute's documentation and posts on Inspect, an MLOps case study of Meta's FBLearner Flow (โš ๏ธ secondary) and real job postings from Thinking Machines Lab (no Meta, OpenAI or Anthropic postings for this role were found). Public detail on how frontier labs run research tooling for LLMs is thin, so the Meta section is short. The dashboard, eval heatmap and sweep planner are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The role: research infrastructure engineer

Thinking Machines Lab's postings describe the job clearly. Its Research Infrastructure Engineer, Research Acceleration (February 2026) builds the libraries and tools that speed up internal research, including evaluation frameworks, reinforcement-learning training systems and experiment tracking, with a listed range of $350,000โ€“$475,000. Its Software Engineer, Research Tools (August 2026) maintains the platform researchers use to monitor training runs, debug model outputs and compare results on shared leaderboards, listed at $300,000โ€“$475,000. The pay signals how much labs value making researchers faster.

What the role needs

Our reading of the postings (1โ€“5). The day timeline is a composite illustration.

3. Why research needs its own tooling

Research is a loop, and every turn of it costs researcher time. Good tooling removes friction at each step. Click a step.

The experiment loop

4. What lives at this layer

The research tooling stack

5. Experiment tracking

Every run logs its configuration, code version, data version, metrics and artefacts, so that results can be compared and reproduced months later. Weights & Biases became the best-known tool for this, with OpenAI and Meta among its reported users. In March 2025 the AI cloud provider CoreWeave agreed to acquire it, a deal The Information reported at about $1.7 billion (terms were not disclosed), completed around May 2025; CoreWeave said the platform would stay compatible with other clouds and frameworks. Open-source MLflow and in-house systems fill the same role elsewhere. The value is less the charts than the discipline: a run nobody can reproduce is a run that taught nothing.

Reading a run dashboard

Illustrative loss curves for four runs. Toggle smoothing and look for problems.

6. Evaluation infrastructure

Every checkpoint of every serious run gets evaluated automatically, on dozens of benchmarks, so researchers see regressions early (see Producing Intelligence Part 7). The tooling has converged on a few open frameworks. EleutherAI's lm-evaluation-harness is widely used for academic benchmarks. The UK AI Security Institute's Inspect, developed with Meridian Labs and released as open source under the MIT licence, targets frontier evaluations including agents and tools; its companion Inspect Evals offers 200+ ready-made evaluations, and AISI reports that Apollo Research dropped its own internal framework after a three-month trial of Inspect. A third-party guide lists METR, Apollo and several government testing bodies as users.

Checkpoint ร— benchmark heatmap

Illustrative scores for six checkpoints of one run. A good eval dashboard makes regressions jump out.

7. Case study: Meta's FBLearner Flow and after

Meta has published less about its LLM research tooling than about its hardware, so this case study is brief. Its earlier platform, FBLearner Flow, began as a system for model training and experimentation and, according to an MLOps case study, grew over about four years to cover feature stores, lineage tracking and governance โš ๏ธ secondary. For today's large-model research, the pieces covered in this series fit together: MAST (Part 5) places jobs, PyTorch and TorchTitan (Part 7) run them.

8. Sweeps and the economics of experiments

Choosing a learning rate, batch size or data mix means running many small experiments. How you search matters as much as how much compute you have: grid search wastes runs on bad regions, random search covers important dimensions better, and methods like successive halving stop bad runs early. Every GPU-hour spent on a sweep is one not spent on the main run, which is why research tooling teams work closely with the people who allocate compute (see Compute Economics).

Sweep planner

Illustrative model for tuning two hyperparameters with a fixed GPU budget.

9. When it breaks

Research tooling detective

Pick a symptom.

10. Handoffs and careers

Who research tooling engineers work with

๐Ÿงญ Getting into the role
Typical backgrounds
Research engineers who built tools for their own team, MLOps/platform engineers, full-stack engineers with ML interest, evaluation researchers.
Core skills
Python, distributed systems basics, data and metrics pipelines, web dashboards, evaluation design, empathy for researchers' workflows, ruthless attention to reproducibility.
Growth path
Research tools engineer โ†’ owner of the eval or experiment platform โ†’ head of research infrastructure.
Further reading
From Idea to Merged PR ยท Is It Ready? ยท Compute Economics

๐Ÿ—‚๏ธ Case file

Meta
FBLearner Flow grew from training platform to end-to-end ML platform โš ๏ธ secondary; LLM research runs on MAST, PyTorch and TorchTitan.
Industry
Weights & Biases (CoreWeave, 2025) for tracking; lm-evaluation-harness and UK AISI's Inspect for evaluation.
Roles
Thinking Machines Lab: research acceleration and research tools teams, $300โ€“475K listed.
Verdict Research tooling multiplies everything else in the factory: a lab with great GPUs and poor tooling learns slowly. The best teams make experiments cheap to launch, impossible to lose and easy to compare, and they automate evaluation so that every checkpoint tells researchers something. As AI agents start running experiments themselves, this layer is becoming the interface between automated research and human judgement.

๐Ÿ‘€ Next in the series

Part 10: Inference platform โ€” the inference engineers who turn trained models into fast, cheap, reliable services.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References