The Software Stack of an AI Factory: From CUDA to PyTorch to a 16,000-GPU Training Run
Meta trained Llama 3 405B on up to 16,384 Nvidia H100 GPUs, yet most of its researchers wrote Python. Between their code and the silicon sit about ten layers of software, written in at least five languages, that turn thousands of separate chips into one machine, move data between them, schedule the job, store petabytes and recover from a failure every few hours. This first part of the series maps that stack layer by layer, using Llama 3 as a running case study.
๐งโ๐ป One script, 16,384 GPUs
The AI Compute Stack series followed the hardware: power, factories, chips, networks, memory. This series follows what happens inside the factory, end to end, as a frontier model is produced. We use one model throughout: Meta's Llama 3 405B, released in July 2024, because Meta published unusually detailed information about how it was built. OpenAI, Anthropic and Google have not published comparable detail about their own runs.
A researcher at Meta might express the model in a few thousand lines of Python. Running it meant coordinating 16,384 GPUs in 2,048 servers, each GPU running thousands of tiny programs (kernels) per second, exchanging terabytes of data across a 24,000-GPU network, writing checkpoints to a 240-petabyte storage system, and surviving 466 job interruptions in a single 54-day stretch. All of that is software.
๐งฑ The stack, layer by layer
Read from the bottom up: each layer hides the complexity of the one below. Click a layer to see what it does, the main tools, the language it is written in, and what Meta used for Llama 3.
๐๏ธ The AI factory software stack
Bottom = closest to the hardware. Click a layer.
๐ค Who writes what: the languages of an AI factory
"AI is written in Python" is true only at the top. Python is the glue and the research language; the speed comes from C++ and GPU code underneath, and the infrastructure around it uses yet other languages.
๐งฉ Languages by layer
โ main language ยท โ also used. Typical, not exhaustive.
Our summary of the main open-source projects (PyTorch, CUDA libraries, NCCL, Triton, Megatron, Kubernetes, Slurm).
- Python
- Model definitions, training loops, data pipelines, experiment configs, evaluation. What most researchers write.
- C++
- The core of PyTorch, CUDA runtime libraries, NCCL, inference engines. Where performance-critical systems live.
- CUDA C++ / PTX
- Hand-written GPU kernels: matrix multiplies, attention (FlashAttention), communication kernels. A small group of specialists writes these (see Writing Custom Kernels).
- Triton
- A Python-embedded language from OpenAI for writing GPU kernels without CUDA; PyTorch's compiler generates Triton automatically.
- Rust, Go, C
- Infrastructure: Kubernetes and many cloud tools (Go), Slurm and firmware (C), and a growing share of data and serving tooling (Rust).
๐ฐ The CUDA moat and the portability fight
Nvidia's CUDA (launched 2007, now at version 13) is the layer most AI software was built on, and the main reason switching chips is hard (see Custom Silicon vs Nvidia). AMD's answer is ROCm: version 7 shipped in September 2025 with MI350 support, and AMD's newer "TheRock" stream is set to become its production baseline from mid-2026.
The moat is eroding from the top. PyTorch 2's compiler generates Triton kernels rather than CUDA, so models written in PyTorch increasingly run on AMD with little change, and OpenAI has been optimising GPT-class workloads on AMD hardware through Triton on ROCm. The hardest layer to port is communication: distributed training depends on NCCL (Nvidia's library, version 2.29 in 2026), and even teams that move their model code to ROCm can remain tied to NCCL-specific behaviour at the networking layer. Projects such as HetCCL (January 2026) aim to abstract it.
๐งฐ Training frameworks compared
PyTorch provides the building blocks; a training framework assembles them into a system that can split a model across thousands of GPUs (Part 4 of this series covers the splitting itself).
๐ The main options in 2026
| Framework | From | Built on | Strengths | Used for |
|---|---|---|---|---|
| Megatron-Core (Megatron-LM, NeMo) | Nvidia | PyTorch | The reference implementation of tensor, pipeline, sequence, context and expert parallelism; highly tuned for Nvidia GPUs | Many frontier and national-scale training runs |
| TorchTitan + FSDP2 | Meta / PyTorch team | PyTorch | PyTorch-native 4D parallelism with Float8; recipes for Llama 3.1 8Bโ405B; authors report ~30% faster on 405B at 512 GPUs than optimised baselines | Reference and experimentation platform; FSDP2 replaces deprecated FSDP1 |
| DeepSpeed | Microsoft, now LF AI & Data / PyTorch Foundation | PyTorch | ZeRO memory sharding, offloading; easy to adopt | Fine-tuning and mid-scale training |
| MaxText / JAX | JAX + XLA | Compiler-driven sharding; native to TPUs | Google and TPU users | |
| In-house | Frontier labs | Usually PyTorch or JAX | Tailored to their clusters and models | Meta's Llama 3 ("custom training libraries"), OpenAI, Anthropic, DeepSeek |
TorchTitan paper (ICLR 2025); framework documentation and 2026 overviews (youngju.dev, Spheron, Baseten). Version and adoption details change quickly.
๐ฆ Llama 3 case file: Meta's factory and stack
Here is what the Llama 3 paper says Meta used, layer by layer.
๐บ๏ธ How 24,000 GPUs are wired
Meta's RoCE cluster for Llama 3 405B, a three-layer Clos network. Click a level.
Llama 3 paper, via secondary summaries. Meta later reported reducing cross-zone oversubscription from 1:7 to 1:2.8 ("Collective Communication for 100k+ GPUs", 2025).
๐ง When software meets 16,000 GPUs
At this scale, failure is routine. In a 54-day snapshot of Llama 3 pre-training, Meta recorded 466 job interruptions: 47 planned (firmware upgrades, maintenance) and 419 unexpected, about one every three hours. Around 78% of the unexpected ones were traced to confirmed hardware problems, and GPU issues (including HBM3 memory) were the largest category at 58.7%. Yet automation kept effective training time above 90%, and engineers had to intervene manually only three times.
๐๏ธ Failure simulator: how much training time survives?
Simplified model: each failure loses the work since the last checkpoint (on average half an interval) plus restart time; each checkpoint pauses training briefly. Defaults approximate Llama 3's failure rate.
Effective time โ 1 โ (checkpoint pause รท interval) โ (failures per hour ร (interval รท 2 + restart)). Real systems also lose time to slow nodes and hangs, and use in-memory checkpoints to shrink the interval.
๐ The life of a training job
What actually happens when an engineer launches a large training run? Step through it.
๐๏ธ Slurm vs Kubernetes
๐ต Slurm
The workhorse of supercomputing (see AI Factories vs Supercomputers). Researchers submit batch jobs to queues; Slurm allocates whole nodes, launches processes on all of them together and tracks fair-share usage. Simple and battle-tested for large synchronous training runs.๐ข Kubernetes
The cloud-native standard for running containers and services. Better for inference fleets, data pipelines and mixed workloads; needs add-ons (gang scheduling, topology awareness) for big training jobs. Hybrids such as CoreWeave's SUNK run Slurm on Kubernetes, and Ray is widely used for reinforcement-learning and data jobs.Large labs often build their own: Meta's MAST schedules training across its global fleet, matching jobs to GPUs, data and network topology.
๐ฐ๏ธ Timeline
๐ญ What to watch
- Portable communication: whether libraries like HetCCL loosen NCCL's lock-in.
- Compilers: how much hand-written CUDA is replaced by torch.compile, Triton and XLA.
- Lower precision: FP8 and MXFP8 training becoming standard in frameworks (Part 5).
- Reliability tooling: faster in-memory checkpoints and automatic fault isolation as clusters pass 100,000 GPUs.
- Next in this series: Part 2 follows the data: how Llama 3's ~15 trillion training tokens were collected, cleaned and mixed.
๐ข Reading the numbers
- Llama 3 figures (16,384 GPUs, 240 PB, 466 interruptions, 58.7% GPU-related, >90% effective time, three manual interventions) come from Meta's paper as quoted in search results and press coverage.
- Meta's training code is not public; only its architecture is described.
- TorchTitan's 30% gain is its authors' claim, measured on Llama 3.1 405B at 512 GPUs.
- 2026 versions (CUDA 13.x, ROCm 7.x, NCCL 2.29, DeepSpeed 0.19) come from vendor pages and blogs and change frequently.
- The language matrix and simulator are simplified illustrations.
๐ References and papers
- Llama 3 and Meta's infrastructure
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024)
- "Scaling Llama 3 Training with Efficient Parallelism Strategies" (ISCA 2025)
- Meta: "Collective Communication for 100k+ GPUs" (2025)
- Meta Engineering: Building Meta's GenAI infrastructure (24K-GPU clusters)
- Pan et al.: "Facebook's Tectonic Filesystem: Efficiency from Exascale" (FAST '21)
- Tom's Hardware: Faulty H100s and HBM3 caused half of Llama 3 training failures
- DCD: Hundreds of GPU and HBM3-related interruptions
- Glenn Lockwood: Meta's H100 clusters
- Frameworks and compilers
- Liang et al.: "TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training" (ICLR 2025)
- Shoeybi et al.: "Megatron-LM" (2019)
- Rajbhandari et al.: "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (2020)
- Ansel et al.: "PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation" (ASPLOS 2024)
- Tillet et al.: "Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations" (2019)
- Dao et al.: "FlashAttention" (2022)
- 2026 ecosystem
- A map of the LLM training stack in 2026
- Spheron: Distributed training with FSDP, DeepSpeed and Megatron
- Baseten: How the open-source training stack fits together
- Spheron: CUDA, ROCm and AI frameworks in 2026
- AMD: ROCm Core SDK 7.13 release notes
- SoftwareSeni: CUDA lock-in unpacked
- Nvidia: Optimized frameworks support matrix
- Related on this site
- Scaling Laws, GPUs and Distributed Training ยท Reading and Writing Production Training Code ยท The Interconnect Wars ยท AI Compute Stack at a Glance