Home โ€บ Blog โ€บ The Software Stack of an AI Factory
Producing Intelligence ยท Part 1 ยท Software

The Software Stack of an AI Factory: From CUDA to PyTorch to a 16,000-GPU Training Run

Meta trained Llama 3 405B on up to 16,384 Nvidia H100 GPUs, yet most of its researchers wrote Python. Between their code and the silicon sit about ten layers of software, written in at least five languages, that turn thousands of separate chips into one machine, move data between them, schedule the job, store petabytes and recover from a failure every few hours. This first part of the series maps that stack layer by layer, using Llama 3 as a running case study.

FL
FrontierAGI Team
โš ๏ธ How this was researched The Llama 3 case-file facts come from Meta's paper "The Llama 3 Herd of Models" (2024), as quoted in search results and press coverage (we could not open the full text from our research environment), plus Meta's engineering posts. 2026 software versions (CUDA 13, ROCm 7, NCCL 2.29) come from vendor documentation and secondary blogs. Meta's internal training code is not public; the paper describes it only as custom PyTorch-based libraries. The failure simulator is a simplified model. See Reading the numbers.

๐Ÿง‘โ€๐Ÿ’ป One script, 16,384 GPUs

The AI Compute Stack series followed the hardware: power, factories, chips, networks, memory. This series follows what happens inside the factory, end to end, as a frontier model is produced. We use one model throughout: Meta's Llama 3 405B, released in July 2024, because Meta published unusually detailed information about how it was built. OpenAI, Anthropic and Google have not published comparable detail about their own runs.

A researcher at Meta might express the model in a few thousand lines of Python. Running it meant coordinating 16,384 GPUs in 2,048 servers, each GPU running thousands of tiny programs (kernels) per second, exchanging terabytes of data across a 24,000-GPU network, writing checkpoints to a 240-petabyte storage system, and surviving 466 job interruptions in a single 54-day stretch. All of that is software.

16,384H100 GPUs used to train Llama 3 405B (80 GB HBM3, 700 W each)
240 PBTectonic storage for training, on 7,500 SSD servers
466job interruptions in a 54-day period (419 unexpected)
>90%effective training time despite them

๐Ÿงฑ The stack, layer by layer

Read from the bottom up: each layer hides the complexity of the one below. Click a layer to see what it does, the main tools, the language it is written in, and what Meta used for Llama 3.

๐Ÿ—‚๏ธ The AI factory software stack

Bottom = closest to the hardware. Click a layer.

๐Ÿ”ค Who writes what: the languages of an AI factory

"AI is written in Python" is true only at the top. Python is the glue and the research language; the speed comes from C++ and GPU code underneath, and the infrastructure around it uses yet other languages.

๐Ÿงฉ Languages by layer

โ— main language ยท โ—‹ also used. Typical, not exhaustive.

Our summary of the main open-source projects (PyTorch, CUDA libraries, NCCL, Triton, Megatron, Kubernetes, Slurm).

Python
Model definitions, training loops, data pipelines, experiment configs, evaluation. What most researchers write.
C++
The core of PyTorch, CUDA runtime libraries, NCCL, inference engines. Where performance-critical systems live.
CUDA C++ / PTX
Hand-written GPU kernels: matrix multiplies, attention (FlashAttention), communication kernels. A small group of specialists writes these (see Writing Custom Kernels).
Triton
A Python-embedded language from OpenAI for writing GPU kernels without CUDA; PyTorch's compiler generates Triton automatically.
Rust, Go, C
Infrastructure: Kubernetes and many cloud tools (Go), Slurm and firmware (C), and a growing share of data and serving tooling (Rust).

๐Ÿฐ The CUDA moat and the portability fight

Nvidia's CUDA (launched 2007, now at version 13) is the layer most AI software was built on, and the main reason switching chips is hard (see Custom Silicon vs Nvidia). AMD's answer is ROCm: version 7 shipped in September 2025 with MI350 support, and AMD's newer "TheRock" stream is set to become its production baseline from mid-2026.

The moat is eroding from the top. PyTorch 2's compiler generates Triton kernels rather than CUDA, so models written in PyTorch increasingly run on AMD with little change, and OpenAI has been optimising GPT-class workloads on AMD hardware through Triton on ROCm. The hardest layer to port is communication: distributed training depends on NCCL (Nvidia's library, version 2.29 in 2026), and even teams that move their model code to ROCm can remain tied to NCCL-specific behaviour at the networking layer. Projects such as HetCCL (January 2026) aim to abstract it.

๐Ÿ’ก Why the stack is shaped like this Each layer exists so that the one above can ignore a hard problem: drivers hide the hardware, CUDA hides the GPU's instruction set, NCCL hides the network, PyTorch hides the kernels, Megatron or TorchTitan hide the parallelism, and the scheduler hides the cluster. The cost is that a failure anywhere can surface as a mysterious hang at the top, which is why debugging distributed jobs is a specialist skill (see Debugging a Distributed Job).

๐Ÿงฐ Training frameworks compared

PyTorch provides the building blocks; a training framework assembles them into a system that can split a model across thousands of GPUs (Part 4 of this series covers the splitting itself).

๐Ÿ“‹ The main options in 2026

FrameworkFromBuilt onStrengthsUsed for
Megatron-Core (Megatron-LM, NeMo)NvidiaPyTorchThe reference implementation of tensor, pipeline, sequence, context and expert parallelism; highly tuned for Nvidia GPUsMany frontier and national-scale training runs
TorchTitan + FSDP2Meta / PyTorch teamPyTorchPyTorch-native 4D parallelism with Float8; recipes for Llama 3.1 8Bโ€“405B; authors report ~30% faster on 405B at 512 GPUs than optimised baselinesReference and experimentation platform; FSDP2 replaces deprecated FSDP1
DeepSpeedMicrosoft, now LF AI & Data / PyTorch FoundationPyTorchZeRO memory sharding, offloading; easy to adoptFine-tuning and mid-scale training
MaxText / JAXGoogleJAX + XLACompiler-driven sharding; native to TPUsGoogle and TPU users
In-houseFrontier labsUsually PyTorch or JAXTailored to their clusters and modelsMeta's Llama 3 ("custom training libraries"), OpenAI, Anthropic, DeepSeek

TorchTitan paper (ICLR 2025); framework documentation and 2026 overviews (youngju.dev, Spheron, Baseten). Version and adoption details change quickly.

๐Ÿฆ™ Llama 3 case file: Meta's factory and stack

Here is what the Llama 3 paper says Meta used, layer by layer.

๐Ÿ–ฅ๏ธ ServersGrand Teton, Meta's Open Compute AI server: 8 H100 GPUs and 2 CPUs per server; GPUs at 700 W TDP with 80 GB HBM3. Up to 16,384 GPUs for the 405B run.
๐ŸŒ NetworkFor 405B, RDMA over Converged Ethernet (RoCE) at 400 Gbps, built on Arista 7800 and Meta Minipack2 switches; smaller Llama 3 models trained on an Nvidia Quantum-2 InfiniBand cluster.
๐Ÿ“ก CommunicationNCCLX, Meta's fork of Nvidia's NCCL, tuned for its network and for faster failure detection; PyTorch's NCCL "flight recorder" was used heavily to diagnose hangs.
๐Ÿ—“๏ธ SchedulingMAST, Meta's global-scale training scheduler.
๐Ÿ’พ StorageTectonic, Meta's distributed file system, used to build a 240 PB storage fabric on 7,500 SSD-equipped servers; the main challenge was bursts of checkpoint writes that briefly saturate it.
๐Ÿง  Training codeCustom PyTorch-based libraries with "4D parallelism" (tensor, context, pipeline and data), running on Meta's Research SuperCluster and production clusters.

๐Ÿ—บ๏ธ How 24,000 GPUs are wired

Meta's RoCE cluster for Llama 3 405B, a three-layer Clos network. Click a level.

Llama 3 paper, via secondary summaries. Meta later reported reducing cross-zone oversubscription from 1:7 to 1:2.8 ("Collective Communication for 100k+ GPUs", 2025).

๐Ÿ”ง When software meets 16,000 GPUs

At this scale, failure is routine. In a 54-day snapshot of Llama 3 pre-training, Meta recorded 466 job interruptions: 47 planned (firmware upgrades, maintenance) and 419 unexpected, about one every three hours. Around 78% of the unexpected ones were traced to confirmed hardware problems, and GPU issues (including HBM3 memory) were the largest category at 58.7%. Yet automation kept effective training time above 90%, and engineers had to intervene manually only three times.

๐ŸŽ›๏ธ Failure simulator: how much training time survives?

Simplified model: each failure loses the work since the last checkpoint (on average half an interval) plus restart time; each checkpoint pauses training briefly. Defaults approximate Llama 3's failure rate.

Llama 3: 419 unexpected failures รท 54 days รท 16.4 thousand GPUs โ‰ˆ 0.47

Effective time โ‰ˆ 1 โˆ’ (checkpoint pause รท interval) โˆ’ (failures per hour ร— (interval รท 2 + restart)). Real systems also lose time to slow nodes and hangs, and use in-memory checkpoints to shrink the interval.

๐Ÿš€ The life of a training job

What actually happens when an engineer launches a large training run? Step through it.

๐Ÿ—“๏ธ Slurm vs Kubernetes

๐Ÿ”ต Slurm

The workhorse of supercomputing (see AI Factories vs Supercomputers). Researchers submit batch jobs to queues; Slurm allocates whole nodes, launches processes on all of them together and tracks fair-share usage. Simple and battle-tested for large synchronous training runs.

๐ŸŸข Kubernetes

The cloud-native standard for running containers and services. Better for inference fleets, data pipelines and mixed workloads; needs add-ons (gang scheduling, topology awareness) for big training jobs. Hybrids such as CoreWeave's SUNK run Slurm on Kubernetes, and Ray is widely used for reinforcement-learning and data jobs.

Large labs often build their own: Meta's MAST schedules training across its global fleet, matching jobs to GPUs, data and network topology.

๐Ÿ•ฐ๏ธ Timeline

2007 ยท Nvidia releases CUDA.
2014 ยท cuDNN, Nvidia's deep-learning kernel library.
2015 ยท TensorFlow released; NCCL follows for multi-GPU communication.
2016โ€“17 ยท PyTorch released by Facebook AI Research.
2018 ยท Google releases JAX.
2019 ยท Nvidia's Megatron-LM introduces efficient tensor parallelism.
2020 ยท Microsoft's DeepSpeed and ZeRO memory sharding.
2021 ยท OpenAI open-sources Triton; Meta describes Tectonic (FAST '21).
2022 ยท FlashAttention; PyTorch FSDP.
2023 ยท PyTorch 2.0 introduces torch.compile, generating Triton kernels.
2024 ยท Meta trains Llama 3 405B on 16,384 H100s; TorchTitan released.
2025 ยท ROCm 7 with MI350 support; DeepSpeed moves to the PyTorch Foundation.
2026 ยท CUDA 13.x, NCCL 2.29; FSDP1 deprecated in favour of FSDP2; Nvidia Dynamo 1.0 for inference.

๐Ÿ”ญ What to watch

๐ŸŽฏ The bottom line An AI factory runs on a deep stack of software: drivers and CUDA at the bottom, kernel and communication libraries above, PyTorch and its compilers in the middle, training frameworks that split models across thousands of GPUs, and schedulers, storage and monitoring around it all. Researchers write Python; the performance comes from C++, CUDA and Triton; the infrastructure from Go, C and Rust. Llama 3 shows what that stack must achieve at frontier scale: keeping 16,384 GPUs productive more than 90% of the time despite a failure every few hours.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References and papers