How LLM Pretraining Works: From Raw Text to a Foundation Model

The Pipeline at a Glance

Pretraining has seventeen distinct stages, but they fall into four phases. Most of the engineering effort — and, in practice, most of what separates a good model from a mediocre one of the same size — goes into the first two, before a single gradient is computed.

Phase 1: Data

Sources and collection

The raw material is text: web pages (overwhelmingly from Common Crawl, a free public archive of billions of web pages, with new snapshots added regularly), books, code from public repositories, academic papers, forums and reference works. None of it is labeled — the text itself will supply the training signal. Scale is the point. Hugging Face's open FineWeb dataset, for example, is 15 trillion tokens distilled from 96 Common Crawl snapshots spanning 2013 to 2024; Meta trained Llama 3's 405-billion-parameter model on over 15 trillion tokens.

Cleaning and filtering

Raw web text is mostly not worth training on: navigation menus, cookie banners, spam, machine-generated pages, text in unwanted languages, and fragments too short to contain a thought. A typical filtering stack runs cheapest-first:

Deduplication

The web repeats itself: mirrors, reposts, templated pages, the same license text on a million sites. Lee et al. showed why this matters. Standard datasets contained near-duplicate documents and one 61-word sentence repeated over 60,000 times in C4. Deduplicating them made models emit memorized training text ten times less often, reach the same accuracy in fewer steps, and fixed train–test overlap that affected over 4% of standard validation sets. Two techniques do most of the work:

• Exact deduplication — hash each document (or each line) and drop repeats. Cheap and exact, but a single changed character defeats it.
• Near-duplicate detection with MinHash — split each document into overlapping word n-grams ("shingles"), and estimate the Jaccard similarity between two documents' shingle sets from a short signature: for each of many hash functions, keep the minimum hash over the document's shingles. The fraction of positions where two signatures agree estimates their similarity, so near-copies can be found without comparing every pair of documents in full. Llama 3's pipeline and FineWeb both use MinHash-style near-duplicate removal.

Here's what that funnel did to our miniature crawl — 200 real pages hidden among 40 exact copies, 30 lightly edited copies, and 15 junk pages:

The data mixture

The final corpus is a deliberate blend, not whatever the crawl happened to contain. Llama 3's final pretraining mix, for example, was roughly 50% general knowledge, 25% mathematics and reasoning, 17% code, and 8% multilingual text — chosen with small-scale experiments that predict how each mix affects the full model. Code and math are over-represented relative to the web because they measurably improve reasoning.

Phase 2: Tokens and Batches

The tokenizer

A tokenizer is trained once, on a sample of the corpus, before model training starts. Byte-pair encoding (BPE) and SentencePiece's variants learn a vocabulary of subword pieces so that common words become one token and rare ones split into several — the algorithm is built from scratch in CS336 Part 2. Vocabulary sizes typically fall between about 32,000 and 200,000; Llama 3's has 128,256 entries. A larger vocabulary compresses text into fewer tokens (faster training and inference per word) at the cost of a bigger embedding table. Our miniature pipeline uses the simplest possible choice: raw bytes, 256 of them, plus one end-of-text token.

Packing into sequences

The model trains on fixed-length sequences of T tokens — 4,096 or 8,192 for Llama 3's main pretraining, 64 in our script. Documents come in every length, so rather than pad short ones (wasting compute on padding), pipelines pack: concatenate tokenized documents end to end with an end-of-text separator between them, then cut the stream into chunks of T + 1 tokens (the extra one lets the last position have a target). Some recipes also mask attention across document boundaries so a token can't read the unrelated document packed before it.

Shuffling, batching, and the held-out split

Sequences are shuffled so each batch mixes sources, then grouped into batches. The global batch is what one optimizer step sees: micro-batch per GPU × gradient-accumulation steps × number of data-parallel GPUs. Batches are measured in tokens and are large. GPT-3 ramped its batch from 32K up to 3.2M tokens; Llama 3 405B started at 4M tokens and doubled twice, to 16M, as training went on. Before any of this, a slice of documents is set aside as a validation set — split by document, not by sequence, so no validation text leaks into training.

The Objective: Next-Token Prediction

There are no labels to collect. The target for every position is simply the next token in the same text — the input shifted left by one:

This is called self-supervised learning: the data provides its own answers. Three details make it efficient. The causal mask means position t can only see tokens up to t, so a single forward pass makes T separate predictions at once — one per position — and all of them count toward the loss. The model always sees the true previous tokens, never its own guesses (called teacher forcing), so every position trains in parallel. And the loss is the cross-entropy averaged over every position of every sequence in the batch:

Inside One Training Step

Every one of the hundreds of thousands of steps in a pretraining run is the same seven operations, with these shapes (B sequences of T tokens, vocabulary V):

1. Take a batch of token IDs(B, T)Plus the same tokens shifted by one as targets.
2. Forward pass(B, T, d)Embeddings (and positional information), N transformer blocks, final norm. The transformer explainer covers every step inside.
3. Logits over the vocabulary(B, T, V)The LM head scores every vocabulary entry at every position.
4. Compare with the shifted targets(B, T)One correct token per position.
5. Cross-entropy lossscalarAveraged over all B·T positions.
6. Backpropagation∇θ LA gradient for every trainable parameter, via automatic differentiation.
7. Optimizer update (AdamW)θ ← θ − …With the current learning rate from the schedule, after gradient clipping.

The Training Recipe

The seven-step loop is simple; getting it to run stably for months is not. Six ingredients show up in essentially every modern recipe.

1. Initialization

Before step one, every weight is set to small random values. The scale matters more than it looks. Our script first used PyTorch's default embedding initialization, which draws from a standard normal. Because the output layer reuses the embedding matrix, that produced huge logits, and the untrained model's validation loss was 63.1 — far worse than random guessing, which scores ln 257 = 5.55. Switching to GPT-2's convention of normal weights with standard deviation 0.02 brought the starting loss to 5.554, exactly where an untrained model should be.

2. The learning-rate schedule

The learning rate starts near zero, rises linearly during a short warmup (large early updates on random weights are destabilizing), then decays — most commonly along a cosine curve — to a small final value. GPT-3 warmed up over its first 375M tokens and decayed with a cosine over 260B tokens. Llama 3 405B used a peak of 8×10⁻⁵, 8,000 warmup steps, and a cosine decay to 8×10⁻⁷ over 1.2 million steps. Our run, compressed to 600 steps:

3. AdamW

The optimizer is almost always AdamW: Adam's per-parameter adaptive steps plus weight decay applied directly to the weights (GPT-3 used a weight decay of 0.1). The update itself is covered in How a Transformer Works. Its practical cost is memory: two extra values per parameter, which is why optimizer state dominates training memory (CS336 Part 3).

4. Gradient clipping

Occasionally a batch produces a huge gradient that would throw the weights far off course. Gradient-norm clipping, introduced for recurrent networks by Pascanu, Mikolov and Bengio, rescales the whole gradient whenever its norm exceeds a threshold, preserving its direction. GPT-3 clipped at 1.0; so does our script, which logged gradient norms between 0.28 and 2.94 — the higher values early in training are exactly when clipping kicks in.

5. Mixed precision

Matrix multiplications run in 16-bit floating point, roughly doubling speed and halving activation memory, while a 32-bit master copy of the weights receives the updates. The original recipe used fp16, which has a narrow exponent range and needs loss scaling to stop small gradients underflowing to zero. The now-standard bfloat16 keeps fp32's exponent range with fewer mantissa bits, so it needs no loss scaling. Our script wraps the forward pass in torch.autocast(..., dtype=torch.bfloat16).

6. Gradient accumulation

If the desired batch doesn't fit in memory, run several smaller micro-batches, add up their gradients, and step once. The update is equivalent to one large batch. Our script accumulates 2 micro-batches of 16 sequences per step.

Scaling Up: Systems

A 405B-parameter model doesn't fit on one GPU — its weights alone need about 810 GB in bf16, and training needs several times that for gradients, optimizer state and activations. Four ways of splitting the work are combined in practice:

Llama 3 405B combined all of these on up to 16,000 H100 GPUs. At that scale hardware failure is routine: during a 54-day stretch of pretraining the job was interrupted 466 times (419 of them unexpected, about half traced to GPUs or their memory), yet automation kept effective training time above 90%. That's only possible because of frequent checkpoints and fast automatic restarts.

How much compute, and how many tokens?

Training compute is roughly 6 × parameters × tokens (derived in CS336 Part 3). For a 405B model on 15T tokens that's about 6 × 4.05×10¹¹ × 1.5×10¹³ ≈ 3.6×10²⁵ FLOPs. How to split a budget between model size and data is the question DeepMind's Chinchilla paper answered: for compute-optimal training, grow parameters and tokens in equal proportion — roughly 20 tokens per parameter. Modern models deliberately train past that point (405B on 15T tokens is about 37 tokens per parameter), because a smaller, longer-trained model is cheaper to serve to millions of users, and inference cost now dominates.

Validation and Checkpoints

Validation loss is the same next-token loss measured on held-out text the model never trains on. It's the main signal that training is working: if training loss keeps falling while validation loss rises, the model is memorizing. It's often reported as perplexity, which is just e^loss — roughly, the number of equally likely tokens the model is choosing between. A model guessing uniformly over our 257-token vocabulary has perplexity 257; ours finished at about 1.1. Real runs also track downstream benchmark scores at checkpoints, since loss doesn't capture everything.

A checkpoint is everything needed to resume exactly: model weights, optimizer state (Adam's m and v for every parameter), the step count, the learning-rate schedule position, the data loader's position and random-number state. Runs save them periodically for three reasons: to survive hardware failures, to roll back past a loss spike (a sudden jump in loss that sometimes doesn't recover), and to keep intermediate models for evaluation.

What Comes Out

The result is a base model, also called a foundation model: a very good next-token predictor. It has absorbed grammar, facts, coding patterns and a surprising amount of reasoning, because predicting text well requires all of them. But it isn't yet an assistant. Ask it a question and it may continue with three more questions, as if completing a list. The later stages — supervised fine-tuning on example conversations, then preference or reinforcement learning — shape that raw capability into something that follows instructions. Pretraining is where the capability comes from; post-training is how you reach it.

Run the Whole Pipeline

Every stage above in one script, scaled down until it runs on a laptop CPU in about a minute and a half: a synthetic "crawl" with page chrome, spam, foreign-language pages and duplicates; heuristic cleaning; exact and MinHash deduplication; byte-level tokenization; document-level train/validation split; packing; a 366K-parameter GPT with GPT-2 initialization and tied embeddings; and a training loop with warmup and cosine decay, AdamW, bf16 autocast, gradient accumulation and clipping, validation, checkpointing, and sampling from the best checkpoint. Needs PyTorch 2.x (pip install torch).

"""A miniature pretraining pipeline: raw text -> clean -> dedup -> tokenize -> pack -> train -> checkpoint."""
import hashlib, itertools, math, os, random, re
import torch
import torch.nn as nn
import torch.nn.functional as F

random.seed(0); torch.manual_seed(0)

# ---------------- 1-2. Raw data: a tiny stand-in for a web crawl ----------------
facts = [
    "The cat sat on the mat and watched the rain fall outside the window.",
    "A dog ran across the park, chasing a red ball thrown by a child.",
    "Water boils at one hundred degrees Celsius at sea level.",
    "The sun rises in the east and sets in the west every day.",
    "Bees collect nectar from flowers and turn it into honey.",
    "The river flows past the old mill and into the quiet lake.",
    "Children read books in the library after school each afternoon.",
    "Trains leave the station every hour for the city center.",
    "The farmer planted wheat in spring and harvested it in autumn.",
    "Birds build nests in tall trees to keep their eggs safe.",
    "The moon pulls on the ocean and causes the tides to rise and fall.",
    "Snow covered the mountain, and the hikers turned back before dark.",
]
combos = list(itertools.combinations(facts, 3))           # 220 distinct 3-sentence "pages"
random.shuffle(combos)
raw_docs = [f"Home | About | Contact\n{' '.join(c)}\nSubscribe to our newsletter! Cookie policy."
            for c in combos[:200]]
raw_docs += raw_docs[:40]                                # 40 exact duplicates (mirrors, reposts)
raw_docs += [d.replace(".\nSubscribe", "!\nSubscribe") for d in raw_docs[40:70]]   # 30 near-duplicates: edited punctuation
raw_docs += ["Click here!!! $$$ WIN BIG $$$ click click click"] * 5          # spam
raw_docs += ["Le chat est assis sur le tapis et regarde la pluie tomber."] * 5  # other language
raw_docs += ["ok"] * 5                                                        # too short
print(f"raw documents:            {len(raw_docs)}")

# ---------------- 3. Cleaning and filtering (heuristics) ----------------
BOILERPLATE = re.compile(r"^(Home \| About \| Contact|Subscribe to our newsletter!.*)$")
ENGLISH = {"the", "and", "a", "of", "in", "to", "is", "at", "on", "every", "each"}
def clean(doc):
    lines = [l for l in doc.split("\n") if not BOILERPLATE.match(l.strip())]
    return " ".join(lines).strip()
def keep(doc):
    words = doc.lower().split()
    if len(words) < 8: return False                                   # too short
    if sum(c.isalpha() or c.isspace() for c in doc) / len(doc) < 0.9: return False   # symbol junk
    if sum(w in ENGLISH for w in words) / len(words) < 0.1: return False            # crude language ID
    return True
docs = [d for d in map(clean, raw_docs) if keep(d)]
print(f"after cleaning/filtering: {len(docs)}")

# ---------------- 4. Deduplication: exact hashes, then MinHash for near-duplicates ----------------
seen, exact = set(), []
for d in docs:
    h = hashlib.sha256(d.encode()).hexdigest()
    if h not in seen:
        seen.add(h); exact.append(d)
print(f"after exact dedup:        {len(exact)}")

def minhash(doc, num_hashes=128, k=5):
    words = doc.lower().split()
    shingles = {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}
    return [min(int.from_bytes(hashlib.blake2b(f"{seed}:{s}".encode(), digest_size=8).digest(), "big")
                for s in shingles) for seed in range(num_hashes)]
kept, signatures = [], []
for d in exact:
    sig = minhash(d)
    if all(sum(a == b for a, b in zip(sig, other)) / len(sig) < 0.8 for other in signatures):
        kept.append(d); signatures.append(sig)                         # estimated Jaccard < 0.8: keep
print(f"after near-dup (MinHash): {len(kept)}")

# ---------------- 5-6. Tokenizer: byte-level (V = 256 bytes + 1 end-of-text token) ----------------
EOT, V = 256, 257
encode = lambda s: list(s.encode("utf-8"))
decode = lambda ids: bytes(i for i in ids if i < 256).decode("utf-8", errors="replace")

# ---------------- 7. Train/val split by document, then pack into fixed-length sequences ----------------
random.shuffle(kept)
n_val = len(kept) // 10
def pack(documents, T):
    stream = []
    for d in documents:
        stream += encode(d) + [EOT]                                    # concatenate with separators
    n = (len(stream) - 1) // T
    return torch.tensor(stream[: n * T + 1])                           # +1 so targets can be shifted
T = 64
train_stream, val_stream = pack(kept[n_val:], T), pack(kept[:n_val], T)
print(f"train tokens: {len(train_stream):,}   val tokens: {len(val_stream):,}   sequence length T = {T}")

# ---------------- 8. Shuffle and batch: random windows, targets shifted by one ----------------
def get_batch(stream, B):
    starts = torch.randint(0, len(stream) - T - 1, (B,))
    x = torch.stack([stream[s:s + T] for s in starts])
    y = torch.stack([stream[s + 1:s + T + 1] for s in starts])        # Y = shift(X, -1)
    return x, y

# ---------------- 9. Initialize a small decoder-only transformer ----------------
d, H, L = 96, 4, 3
class Block(nn.Module):
    def __init__(self):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
    def forward(self, x):
        B_, T_, _ = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=-1)
        q, k, v = (t.view(B_, T_, H, d // H).transpose(1, 2) for t in (q, k, v))
        a = F.scaled_dot_product_attention(q, k, v, is_causal=True)   # masked attention
        x = x + self.proj(a.transpose(1, 2).reshape(B_, T_, d))
        return x + self.mlp(self.ln2(x))
class GPT(nn.Module):
    def __init__(self):
        super().__init__()
        self.tok, self.pos = nn.Embedding(V, d), nn.Embedding(T, d)
        self.blocks = nn.Sequential(*[Block() for _ in range(L)])
        self.ln_f, self.head = nn.LayerNorm(d), nn.Linear(d, V, bias=False)
        self.head.weight = self.tok.weight                            # tied embeddings
    def forward(self, idx):
        x = self.tok(idx) + self.pos(torch.arange(idx.shape[1]))
        return self.head(self.ln_f(self.blocks(x)))
model = GPT()
for m in model.modules():                                              # GPT-2-style init: N(0, 0.02)
    if isinstance(m, (nn.Linear, nn.Embedding)):
        nn.init.normal_(m.weight, std=0.02)
    if isinstance(m, nn.Linear) and m.bias is not None:
        nn.init.zeros_(m.bias)
print(f"parameters: {sum(p.numel() for p in model.parameters()):,}")

# ---------------- 10-14. Training loop ----------------
steps, warmup, peak_lr, micro_B, accum = 600, 60, 3e-3, 16, 2
opt = torch.optim.AdamW(model.parameters(), lr=peak_lr, betas=(0.9, 0.95), weight_decay=0.1)
def lr_at(step):                                                       # linear warmup, cosine decay to 10%
    if step < warmup: return peak_lr * (step + 1) / warmup
    progress = (step - warmup) / (steps - warmup)
    return peak_lr * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * progress)))

@torch.no_grad()
def val_loss(n=20):
    model.eval()
    losses = [F.cross_entropy(model(x).reshape(-1, V), y.reshape(-1)).item()
              for x, y in (get_batch(val_stream, micro_B) for _ in range(n))]
    model.train()
    return sum(losses) / n

best = float("inf")
print(f"step    0  val loss {val_loss():.3f}  (random guessing = ln {V} = {math.log(V):.3f})")
for step in range(steps):
    for g in opt.param_groups: g["lr"] = lr_at(step)
    for _ in range(accum):                                             # gradient accumulation
        x, y = get_batch(train_stream, micro_B)
        with torch.autocast("cpu", dtype=torch.bfloat16):              # mixed precision (bf16)
            logits = model(x)
        loss = F.cross_entropy(logits.float().reshape(-1, V), y.reshape(-1)) / accum
        loss.backward()
    grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # gradient clipping
    opt.step(); opt.zero_grad(set_to_none=True)
    if (step + 1) % 100 == 0:                                          # 16. validate + checkpoint
        vl = val_loss()
        if vl < best:
            best = vl
            torch.save({"step": step + 1, "model": model.state_dict(), "opt": opt.state_dict()}, "ckpt.pt")
        print(f"step {step + 1:>4}  lr {lr_at(step):.5f}  grad norm {grad_norm:.2f}  "
              f"val loss {vl:.3f}  perplexity {math.exp(vl):.1f}")

# ---------------- 17. The "foundation model": reload the best checkpoint and sample ----------------
ckpt = torch.load("ckpt.pt")
model.load_state_dict(ckpt["model"]); model.eval()
idx = torch.tensor([encode("The cat ")])
with torch.no_grad():
    for _ in range(60):
        probs = model(idx[:, -T:])[0, -1].softmax(-1)
        idx = torch.cat([idx, torch.multinomial(probs, 1)[None]], dim=1)
print(f"best checkpoint: step {ckpt['step']}   sample: {decode(idx[0].tolist())!r}")
os.remove("ckpt.pt")

Output (PyTorch 2.14 on CPU; multithreaded CPU math can shift the later decimals between runs):

raw documents:            285
after cleaning/filtering: 270
after exact dedup:        230
after near-dup (MinHash): 200
train tokens: 33,281   val tokens: 3,713   sequence length T = 64
parameters: 366,528
step    0  val loss 5.554  (random guessing = ln 257 = 5.549)
step  100  lr 0.00297  grad norm 2.20  val loss 1.628  perplexity 5.1
step  200  lr 0.00258  grad norm 2.94  val loss 0.305  perplexity 1.4
step  300  lr 0.00189  grad norm 1.01  val loss 0.172  perplexity 1.2
step  400  lr 0.00112  grad norm 0.47  val loss 0.130  perplexity 1.1
step  500  lr 0.00053  grad norm 0.33  val loss 0.117  perplexity 1.1
step  600  lr 0.00030  grad norm 0.28  val loss 0.107  perplexity 1.1
best checkpoint: step 600   sample: 'The cat sat on the mat and watched the rain fall outside the window.'

Read the output against the article. The funnel removes exactly the 15 junk pages, 40 exact copies and 30 near-copies that were planted. The starting validation loss matches random guessing (5.554 vs ln 257 = 5.549). Loss falls fastest during the high-learning-rate phase and flattens as the cosine decays, and the gradient norm settles as training converges. Be honest about the ending, though: a validation perplexity of 1.1 is only possible because our "web" is built from twelve sentences that appear in both the training and validation pages, so the model can memorize them. On real web text, validation loss stays far higher — which is exactly why real pretraining needs trillions of tokens.

⚠️ Notes

🔗 References