Home โ€บ Blog โ€บ Training Frameworks
Who Runs the AI Factory ยท Part 7 ยท Training frameworks

The Code That Trains the Model: ML Systems Engineers and the Frameworks Behind Frontier Training

A researcher writes a model in a few hundred lines of Python. Running it on 16,000 GPUs takes a training framework: software that splits the model and its state across machines, overlaps communication with computation, applies mixed precision, saves checkpoints and keeps going when hardware fails. This part follows the ML systems engineers who build and maintain that software, with Meta's TorchTitan, torchft and Monarch as the case study. How Llama 3 was actually split across GPUs is in Producing Intelligence Part 4.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are the TorchTitan paper (arXiv 2410.06511, ICLR 2025), the PyTorch blog posts on torchft and Monarch, framework documentation and real job postings (no Meta, OpenAI or Anthropic postings for this role were found). Framework comparisons describe documented features, not head-to-head benchmarks. The optimisation stack, fault-tolerance simulator and controller diagram are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The role: ML systems engineer

Job postings describe engineers who sit between research and infrastructure. Canva's Senior ML Systems Engineer (Training Optimization) designs distributed training with Megatron-LM, NeMo, FSDP and Triton and asks for deep knowledge of PyTorch or JAX and libraries such as Megatron-LM, NeMo or DeepSpeed. Cohere's Senior ML Systems Engineer (Frameworks & Tooling) owns the training framework for large-scale LLM training and debugs performance across CUDA, NCCL, networking, I/O and data pipelines. Metamorphic's ML Research Engineer, Distributed Training lists FSDP, DeepSpeed ZeRO, Megatron and TorchTitan plus NVLink/NVSwitch topology; Together AI's Research Engineer, Large-Scale Training asks for data, tensor, pipeline and expert parallelism and mixed precision; Periodic Labs lists $250,000โ€“$350,000 for an ML systems role.

What the role needs

Our reading of the postings (1โ€“5). The day timeline is a composite illustration.

3. What a training framework does

A training framework turns "train this model" into thousands of coordinated processes. Click a layer to see what it handles.

The training framework stack

4. The framework landscape

Four families dominate large-scale training. They overlap heavily and borrow ideas from each other; the right choice depends on hardware, team and how much you value composability versus tuned performance.

Training frameworks compared

Documented features and positioning, not benchmark results.

5. Case study: TorchTitan

Meta built TorchTitan as a PyTorch-native, production-ready reference for LLM pre-training (paper with Harvard co-authors, ICLR 2025). Its idea is composability: pieces of PyTorch that can be stacked in any combination.

Against optimised baselines on H100 GPUs, the paper reports speed-ups of 65.08% for Llama 3.1 8B on 128 GPUs (1D parallelism), 12.59% for Llama 3.1 70B on 256 GPUs (2D) and 30% for Llama 3.1 405B on 512 GPUs (3D). For long context, its 405B run on 512 GPUs (tensor and pipeline parallel degree 8 each) fell from 76 tokens per second per GPU at a 32,768-token sequence length to 16 at 262,144, which is why context parallelism exists.

Stacking optimisations

Illustrative: how each technique adds throughput on a large model. The bars are not the paper's measurements; the paper's totals are quoted above.

6. Fault tolerance: training through failures

In a synchronised job, one failure normally stops everyone, and the job restarts from its last checkpoint (Producing Intelligence Part 5 and Part 9). Meta's torchft, an alpha project, takes a different approach for hybrid sharded data parallelism (HSDP): the model is replicated across several groups, and a fault-tolerant all-reduce lets a failed group drop out while the others keep training, then rejoin. A central "Lighthouse" tracks worker health through heartbeats, with a manager per replica group. In a PyTorch blog demonstration with Crusoe, a 1B-parameter Llama 3 model on L40S GPUs kept training through about 2,000 synthetic failures, one roughly every 15 seconds, with no checkpoints (the post's chart shows 1,200 of them).

Restart vs keep going

Illustrative model of a large job. Restart = every failure stops the job and reloads the last checkpoint; fault-tolerant replicas = the failed group drops out and the rest continue at slightly lower throughput.

7. The next programming model: Monarch

Traditional distributed PyTorch follows the HPC pattern: the same script is launched on every machine, and each copy works out its own role (multiple controllers). In October 2025 the PyTorch team at Meta introduced Monarch, an experimental framework that flips this: a single controller program, written in Python, directs the whole cluster as if it were one machine, with a Rust back end. The aim is to make complex workflows (reinforcement learning with many interacting parts, fault recovery, debugging) easier to express.

Many controllers vs one

8. When it breaks

Framework detective

Pick a symptom. For a full walkthrough see Debugging a Distributed Job.

9. Handoffs and careers

Who ML systems engineers work with

๐Ÿงญ Getting into the role
Typical backgrounds
ML engineers who scaled their own training, HPC/MPI programmers, distributed-systems engineers, PyTorch or JAX contributors.
Core skills
PyTorch internals, FSDP/ZeRO and the parallelism dimensions, NCCL and profiling, mixed precision and numerics, checkpointing, reading large codebases, debugging at scale.
Growth path
ML systems engineer โ†’ owner of a parallelism or fault-tolerance subsystem โ†’ tech lead for a lab's training stack.
Further reading
Reading and Writing Production Training Code ยท Splitting a Giant ยท Scaling Laws, GPUs and Distributed Training

๐Ÿ—‚๏ธ Case file: Meta

PyTorch
Created at Meta; FSDP2 and distributed checkpointing are part of it.
TorchTitan
Composable 4D parallelism, torch.compile, Float8; +65% / +12.6% / +30% at 8B / 70B / 405B scales vs optimised baselines.
torchft
Fault-tolerant HSDP (alpha); ~2,000 synthetic failures survived without checkpoints in a demo.
Monarch
Single-controller distributed programming, Python front end and Rust back end (experimental, Oct 2025).
Verdict Training frameworks are where every other layer of the factory meets: kernels, collectives, storage and scheduling all show up as framework features or framework bugs. The trend is clear: composable, open frameworks that make 4D parallelism a configuration choice, and a shift from "restart after failure" to "keep training through failure". The engineers who understand both the model and the machine are the bridge between research and infrastructure.

๐Ÿ‘€ Next in the series

Part 8: Data platform โ€” the data infrastructure engineers who crawl, deduplicate, filter and version trillions of tokens at scale.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References