The Code That Trains the Model: ML Systems Engineers and the Frameworks Behind Frontier Training
A researcher writes a model in a few hundred lines of Python. Running it on 16,000 GPUs takes a training framework: software that splits the model and its state across machines, overlaps communication with computation, applies mixed precision, saves checkpoints and keeps going when hardware fails. This part follows the ML systems engineers who build and maintain that software, with Meta's TorchTitan, torchft and Monarch as the case study. How Llama 3 was actually split across GPUs is in Producing Intelligence Part 4.
1. Where we are in the stack
The AI factory stack, by role
2. The role: ML systems engineer
Job postings describe engineers who sit between research and infrastructure. Canva's Senior ML Systems Engineer (Training Optimization) designs distributed training with Megatron-LM, NeMo, FSDP and Triton and asks for deep knowledge of PyTorch or JAX and libraries such as Megatron-LM, NeMo or DeepSpeed. Cohere's Senior ML Systems Engineer (Frameworks & Tooling) owns the training framework for large-scale LLM training and debugs performance across CUDA, NCCL, networking, I/O and data pipelines. Metamorphic's ML Research Engineer, Distributed Training lists FSDP, DeepSpeed ZeRO, Megatron and TorchTitan plus NVLink/NVSwitch topology; Together AI's Research Engineer, Large-Scale Training asks for data, tensor, pipeline and expert parallelism and mixed precision; Periodic Labs lists $250,000โ$350,000 for an ML systems role.
What the role needs
Our reading of the postings (1โ5). The day timeline is a composite illustration.
3. What a training framework does
A training framework turns "train this model" into thousands of coordinated processes. Click a layer to see what it handles.
The training framework stack
4. The framework landscape
Four families dominate large-scale training. They overlap heavily and borrow ideas from each other; the right choice depends on hardware, team and how much you value composability versus tuned performance.
Training frameworks compared
Documented features and positioning, not benchmark results.
5. Case study: TorchTitan
Meta built TorchTitan as a PyTorch-native, production-ready reference for LLM pre-training (paper with Harvard co-authors, ICLR 2025). Its idea is composability: pieces of PyTorch that can be stacked in any combination.
- 4D parallelism: FSDP2 (sharded data parallel), tensor, pipeline and context parallelism, the same four dimensions Llama 3 used.
- Compilation and precision: torch.compile for fused kernels (Part 6) and Float8 training.
- Communication tricks: asynchronous tensor parallelism that overlaps communication with compute.
- Checkpointing: asynchronous distributed checkpointing (Part 4).
Against optimised baselines on H100 GPUs, the paper reports speed-ups of 65.08% for Llama 3.1 8B on 128 GPUs (1D parallelism), 12.59% for Llama 3.1 70B on 256 GPUs (2D) and 30% for Llama 3.1 405B on 512 GPUs (3D). For long context, its 405B run on 512 GPUs (tensor and pipeline parallel degree 8 each) fell from 76 tokens per second per GPU at a 32,768-token sequence length to 16 at 262,144, which is why context parallelism exists.
Stacking optimisations
Illustrative: how each technique adds throughput on a large model. The bars are not the paper's measurements; the paper's totals are quoted above.
6. Fault tolerance: training through failures
In a synchronised job, one failure normally stops everyone, and the job restarts from its last checkpoint (Producing Intelligence Part 5 and Part 9). Meta's torchft, an alpha project, takes a different approach for hybrid sharded data parallelism (HSDP): the model is replicated across several groups, and a fault-tolerant all-reduce lets a failed group drop out while the others keep training, then rejoin. A central "Lighthouse" tracks worker health through heartbeats, with a manager per replica group. In a PyTorch blog demonstration with Crusoe, a 1B-parameter Llama 3 model on L40S GPUs kept training through about 2,000 synthetic failures, one roughly every 15 seconds, with no checkpoints (the post's chart shows 1,200 of them).
Restart vs keep going
Illustrative model of a large job. Restart = every failure stops the job and reloads the last checkpoint; fault-tolerant replicas = the failed group drops out and the rest continue at slightly lower throughput.
7. The next programming model: Monarch
Traditional distributed PyTorch follows the HPC pattern: the same script is launched on every machine, and each copy works out its own role (multiple controllers). In October 2025 the PyTorch team at Meta introduced Monarch, an experimental framework that flips this: a single controller program, written in Python, directs the whole cluster as if it were one machine, with a Rust back end. The aim is to make complex workflows (reinforcement learning with many interacting parts, fault recovery, debugging) easier to express.
Many controllers vs one
8. When it breaks
Framework detective
Pick a symptom. For a full walkthrough see Debugging a Distributed Job.
9. Handoffs and careers
Who ML systems engineers work with
- Typical backgrounds
- ML engineers who scaled their own training, HPC/MPI programmers, distributed-systems engineers, PyTorch or JAX contributors.
- Core skills
- PyTorch internals, FSDP/ZeRO and the parallelism dimensions, NCCL and profiling, mixed precision and numerics, checkpointing, reading large codebases, debugging at scale.
- Growth path
- ML systems engineer โ owner of a parallelism or fault-tolerance subsystem โ tech lead for a lab's training stack.
- Further reading
- Reading and Writing Production Training Code ยท Splitting a Giant ยท Scaling Laws, GPUs and Distributed Training
๐๏ธ Case file: Meta
- PyTorch
- Created at Meta; FSDP2 and distributed checkpointing are part of it.
- TorchTitan
- Composable 4D parallelism, torch.compile, Float8; +65% / +12.6% / +30% at 8B / 70B / 405B scales vs optimised baselines.
- torchft
- Fault-tolerant HSDP (alpha); ~2,000 synthetic failures survived without checkpoints in a demo.
- Monarch
- Single-controller distributed programming, Python front end and Rust back end (experimental, Oct 2025).
๐ Next in the series
Part 8: Data platform โ the data infrastructure engineers who crawl, deduplicate, filter and version trillions of tokens at scale.
๐ข Reading the numbers
- TorchTitan speed-ups and long-context throughput come from arXiv 2410.06511 (v3 abstract; v1 framed the 70B and 405B gains as additional).
- torchft facts come from the PyTorch blog and the project README (which lists it as alpha).
- Monarch facts come from the PyTorch blog and InfoWorld coverage; we omit recovery-time figures found only in a secondary blog.
- Role descriptions come from Canva, Cohere, Metamorphic, Together AI and Periodic Labs postings.
- Framework comparison reflects documented features; the optimisation bars, fault-tolerance simulator and controller diagram are illustrations.
๐ References
- Meta / PyTorch
- Liang et al.: "TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training" (ICLR 2025)
- PyTorch blog: Fault-tolerant Llama training with 2,000 synthetic failures (torchft, with Crusoe)
- torchft on GitHub
- PyTorch blog: Introducing PyTorch Monarch ยท InfoWorld coverage
- Roles
- Canva: Senior ML Systems Engineer, Training Optimization (posting)
- Cohere: Senior ML Systems Engineer, Frameworks & Tooling (posting)
- Together AI: Research Engineer, Large-Scale Training (posting)
- Related on this site
- Splitting a Giant ยท Part 6: Kernels & compilers ยท Production Training Code