Home โ€บ Blog โ€บ Storage and Data Movement
Who Runs the AI Factory ยท Part 4 ยท Storage

Feeding the Beast: The Engineers Who Move Petabytes In and Terabytes Out of an AI Factory

GPUs are the most expensive thing in an AI factory, and they spend their lives waiting on two kinds of data: training examples streaming in, and checkpoints of the model streaming out. If either is slow, every GPU in a synchronised job waits. This part follows the storage engineers who design the file systems, object stores, caches and checkpoint paths that keep tens of thousands of GPUs fed, with Meta's move from the Llama 3 storage design to its 2026 "AI storage blueprint" as the case study.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are Meta's engineering posts (March 2024 on its 24K-GPU clusters; July 2026 "Meta's AI Storage Blueprint at Scale"), coverage by DCD and Blocks & Files, PyTorch/TorchTitan publications on checkpointing, and real job postings. Some figures come only from secondary summaries and are marked โš ๏ธ secondary. The 405B checkpoint size is our own arithmetic. The simulators are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The role: storage engineer for AI

Real postings show a consistent profile. MBZUAI's Institute of Foundation Models wants a Senior Storage Systems Engineer to design storage for GPU clusters running training and inference, architecting distributed and parallel file systems such as Lustre, DDN and VAST, integrated with Slurm and Kubernetes. Together AI's Staff Engineer, Distributed Storage evaluates VAST, Weka, Ceph and Lustre and builds Kubernetes-native storage operators. NVIDIA's Distinguished Engineer, Storage (AI Cloud) asks for experience with Lustre, GPFS/Spectrum Scale, WEKA, VAST, BeeGFS or DAOS at multi-petabyte scale and platforms at exabyte scale. Some operators hire storage engineers specifically for incident response: root-causing GPU training outages that turn out to be storage problems.

What the role needs

Our reading of the postings (1โ€“5). The day timeline is a composite illustration.

3. Three very different workloads

An AI factory's storage serves three customers whose demands barely overlap, and a design that is good for one can be bad for another. Toggle each workload.

Traffic patterns over a training day

Illustrative shapes, not measurements.

4. What lives at this layer

The storage software stack

5. Case study: Meta, from Llama 3 to Prometheus

The Llama 3 era. Meta's 24,000-GPU clusters stored training data and checkpoints on Tectonic, its distributed file system, in a deployment of roughly 240 PB built from 7,500 servers โš ๏ธ secondary, designed so that thousands of GPUs could save and load checkpoints at once. Interactive development used a separate Hammerspace-based NFS service, keeping researchers' notebooks away from the training firehose.

The 2026 blueprint. In July 2026 Meta described a rebuild of its AI storage, starting from one observation: in multi-GPU data loading, if one fetch is slow, the whole step slows down for every GPU. The old path ran each read through several stateful metadata lookups and a data proxy, adding hundreds of milliseconds. The new design:

Coverage reported average data ingestion times falling from 150 minutes to 10. By October 2026, coverage of the Prometheus cluster described Meta using object storage for its massive training jobs.

Old path vs new path

Illustrative latency per read. Hops and timings are representative, not Meta's measurements.

6. Checkpoints: the write storm

A checkpoint saves everything needed to resume training: weights, optimiser state, data-loader position and random seeds. For a 405B model trained with mixed-precision Adam, our arithmetic gives about 0.8 TB of BF16 weights plus about 4.9 TB of FP32 master weights and optimiser moments (roughly 12 bytes per parameter), so around 5.7 TB per full checkpoint, written by thousands of GPUs at once. Checkpoint too rarely and every failure costs hours of recomputation; too often and the pauses add up (see Producing Intelligence Part 5).

The main software trick is asynchronous checkpointing. PyTorch's Distributed Checkpoint (DCP) can copy state to CPU memory and finish writing on background threads while training continues. TorchTitan reports this cut checkpoint overhead 5โ€“15ร— compared with synchronous saves for Llama 3.1 8B; the PyTorch blog reports the training pause falling to under 30 seconds, often about 6. Async saving does not make the write itself faster; it hides it.

Checkpoint cost calculator

Simplified model: synchronous checkpoints pause training for the whole write; asynchronous ones pause only for a copy to host memory.

7. Data loading: the stall nobody sees

Training data arrives as billions of samples, often in many small files or shards. Each GPU's data loader must deliver its batch on time, every step. Because ranks synchronise, the step waits for the slowest loader, so the tail of the latency distribution matters more than the average. The usual remedies are prefetching, caching close to the GPU, packing small samples into large shards, and removing slow hops from the read path, which is exactly what Meta's blueprint targeted.

The slowest fetch wins

Illustrative. Each GPU fetches its next batch; most reads are fast, a few are slow. The step waits for the slowest.

8. When it breaks

Storage detective

Pick a symptom. Simplified from operator practice.

9. Handoffs and careers

Who the storage team works with

๐Ÿงญ Getting into the role
Typical backgrounds
HPC storage administrators, distributed-systems engineers from cloud object stores and databases, storage vendor engineers.
Core skills
Parallel file systems (Lustre, GPFS, WEKA, VAST, BeeGFS), object storage, metadata design, caching, NVMe and flash, Linux I/O, Kubernetes/Slurm integration, performance analysis.
Growth path
Storage engineer โ†’ owner of the training data path or checkpoint service โ†’ storage architect across regions.
Further reading
Part 3: Networking ยท The Data Refinery ยท Running the Run

๐Ÿ—‚๏ธ Case file: Meta

Llama 3 era
Tectonic, ~240 PB on 7,500 servers โš ๏ธ secondary, for data and checkpoints; Hammerspace NFS for development.
2026 blueprint
Flat ZippyDB metadata, fat-client SDK, no proxy, regional stacks by the GPUs, memory/flash/HDD tiers.
Results
Ingestion 150 โ†’ 10 minutes (reported); ~80% cache hits โš ๏ธ secondary.
Prometheus
Object storage for massive training jobs (October 2026).
Verdict Storage in an AI factory is judged by one thing: whether GPUs ever wait. That means designing for the tail, not the average, keeping the read path short, putting caches next to the GPUs and hiding checkpoint writes behind training. Meta's 2026 rebuild is a clear example of storage moving from generic infrastructure to something designed around the shape of AI workloads.

๐Ÿ‘€ Next in the series

Part 5: Scheduling and orchestration โ€” the cluster platform engineers who decide which job gets which GPUs, when, and where on the network.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References