Home โ€บ Blog โ€บ Data infrastructure
Inside the Data Layer ยท Part 7 ยท Infrastructure

Feeding the Furnace: Storage, Tokenisation and the Data Loaders Behind Every Training Run

A training run is a furnace that must never go cold: tens of thousands of GPUs each wanting the next batch of tokens every few seconds, for months, in exactly reproducible order. Behind that sits unglamorous infrastructure: petabyte storage, distributed processing jobs, tokenisers that turn text into numbers, packed binary shards, data loaders that resume after failures, and lineage records that say where every token came from. This part explains the plumbing, with calculators for tokens, bandwidth and packing.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are Meta's Llama 3 paper (tokeniser and storage), NVIDIA's Megatron-Core data-loading guide, the FineWeb paper and library documentation (datatrove, MosaicML Streaming, Hugging Face Datasets). Figures not re-checked against the primary source are marked โš ๏ธ. The tokeniser demo is a simplified toy, and the bandwidth and packing calculators are our own models. This part focuses on data; for the storage hardware and the teams who run it, see Who Runs the AI Factory: storage and the data platform.

1. Case file

๐Ÿ—„๏ธ
Data infrastructure, October 2026Stage 8 ("tokenise and load") of the data supply chain
Llama 3 tokeniser128K vocabulary (100K from tiktoken + 28K multilingual)
Llama 3 vs Llama 2 tokeniser3.17 โ†’ 3.94 characters per token (English)
Llama 3 storageTectonic, ~240 PB on ~7,500 servers โš ๏ธ
FineWeb on disk~44 TB for ~15T tokens
Megatron data format.bin token file + .idx index, memory-mapped
Main bottleneck at scale (Megatron guide)Index building and synchronisation, not raw bandwidth

2. The data platform

From raw crawl to GPU memory

Tap a layer. The upper layers run as batch jobs weeks before training; the lower ones run live during it.

3. Processing at scale

Everything in Parts 2 and 3 (extraction, filtering, classifiers, deduplication) runs as large distributed batch jobs, usually on CPU clusters with Spark, Ray, Slurm or purpose-built tools such as Hugging Face's datatrove, and increasingly on GPUs for classifiers and fuzzy deduplication (NVIDIA's NeMo Curator). Data is stored in columnar formats such as Parquet so that each stage can read only the fields it needs, and each stage writes a new versioned copy so that mistakes can be traced and rerun. A single pass of a quality classifier over tens of trillions of tokens is itself a substantial compute job.

4. Tokenisation

Models do not read text; they read token IDs. A tokeniser, usually byte-pair encoding (BPE), is trained once on a sample of the data: it starts from bytes and repeatedly merges the most frequent adjacent pair into a new token, until the vocabulary reaches its target size. Larger vocabularies compress text into fewer tokens (cheaper training and inference per word) at the cost of bigger embedding tables. Meta moved from Llama 2's 32K vocabulary to Llama 3's 128K, raising English compression from 3.17 to 3.94 characters per token, and added 28K tokens for other languages, which are often split into many more tokens per word than English.

Watch BPE merge

A toy BPE trained on your text, in your browser. Each merge joins the most frequent adjacent pair. Real tokenisers train on gigabytes and work on bytes.

Tokens per language

Approximate tokens needed for the same short sentence with an English-centric vs a multilingual vocabulary. Illustrative ratios, in the direction published tokeniser comparisons report.

5. Packing and shards

Training sequences have a fixed length (for example 8,192 tokens), but documents do not. Pipelines pack several documents into each sequence, separated by end-of-document tokens, and usually mask attention so one document cannot see another. The packed token IDs are written as large binary shards with an index of where each sequence starts. Megatron-Core, for example, uses a pair of files per shard: a .bin file of token IDs and an .idx file of offsets, read with memory mapping so any sequence can be fetched without loading the whole file.

Packing simulator

Random document lengths packed into fixed-length sequences. Compare padding each document with greedy packing. Our simulation.

6. How fast must data flow?

Text is small compared with model weights and checkpoints: a token ID takes 2โ€“4 bytes. Even so, a large cluster consumes data continuously, and loaders must keep up while also reading checkpoints, writing logs and surviving failures. The real difficulty is less bandwidth than coordination: every GPU must get a different, deterministic slice, and resume exactly where it stopped after a crash. Megatron's guide notes that at scale the main bottlenecks are index building and synchronisation between processes, not raw bandwidth.

Data bandwidth calculator

Our model: tokens per second consumed by training, converted to bytes. Compare with checkpoint writes, which are far larger.

7. The data loader

DeterministicSame seed, same order, every time, so runs can be reproduced and debugged.
Sharded across ranksEach data-parallel group reads a different slice (see ML Layer Part 3); tensor- and pipeline-parallel peers share the same batch.
ResumableAfter a failure the loader restarts from the exact sample where the last checkpoint was taken, skipping nothing and repeating nothing.
Mixture-awareDraws from several datasets at the weights and schedule set in Part 6, including the annealing switch.
PrefetchingReads ahead on CPU threads so the GPU never waits; a stalled loader idles the whole cluster.

Resume after a crash

A toy deterministic loader: 4 data-parallel ranks, shuffled with a fixed seed. Crash it, then resume from the checkpoint.

8. Lineage and governance

Every token in a training run should be traceable to a source, a licence, the filters it passed and the version of the pipeline that produced it. Lineage answers questions that arrive months later: which model saw this website after it opted out? Which benchmark leaked? Can we remove one author's books and retrain? Good platforms keep immutable dataset versions, manifests of sources and licences, opt-out lists applied at build time, and hashes linking shards back to documents. Regulation is pushing in the same direction: the EU AI Act requires providers of general-purpose models to publish a summary of their training content (Part 8).

9. Tools

LayerCommon tools
Object storage & filesystemsS3 and equivalents, Lustre, Weka, VAST, in-house systems (Meta Tectonic โš ๏ธ, Google Colossus)
FormatsWARC (raw crawl), Parquet / JSONL (processed text), binary token shards (.bin/.idx), MDS (Mosaic)
ProcessingSpark, Ray, Slurm, datatrove, NeMo Curator, Dolma toolkit
Tokeniserstiktoken, Hugging Face tokenizers, SentencePiece
LoadersMegatron-Core datasets, MosaicML Streaming, Hugging Face Datasets (streaming), PyTorch DataLoader, Grain (JAX)
Versioning & lineageDataset manifests, Delta/Iceberg tables, DVC, in-house catalogues

10. Choosing

What does your data stack need?

An editorial guide.

VerdictData infrastructure is rarely why a model is good, but often why a run fails: a tokeniser that wastes a fifth of the compute on other languages, a loader that stalls thousands of GPUs, a resume bug that silently repeats data, or no record of where a disputed document came from. The winning patterns are boring and strict: immutable versions, pre-tokenised packed shards, deterministic and resumable loaders, and lineage from every token back to its source.

Inside the Data Layer

11. Reading the numbers

Primary
Llama 3 tokeniser: 128K tokens (100K tiktoken + 28K), English compression 3.17 โ†’ 3.94 characters per token (Meta paper). Megatron-Core: .bin/.idx pairs, memory-mapped; bottlenecks are index building and barrier synchronisation (NVIDIA guide). FineWeb ~44 TB for ~15T tokens (Hugging Face).
From the papers, not re-checked โš ๏ธ
Llama 3 storage: Tectonic, ~240 PB on ~7,500 servers. Some tool and format details.
Our illustrations
BPE demo (toy, character-level), tokens-per-language ratios, packing simulator, bandwidth calculator and resumable-loader demo.