The Secret Recipe: How Labs Mix, Weight and Schedule Training Data
Two labs with the same data can train very different models, because what matters is not only what goes in but how much of each source, how often it repeats, and when the model sees it. The data mixture (web, code, maths, books, languages) is one of the most closely guarded secrets in AI. This part explains how mixtures are chosen, why the best data is saved for the end of training, how many times data can safely repeat, and how languages are balanced, with calculators for each.
1. Case file
2. Timeline
Choosing mixtures, 2019 โ 2025
Tap a milestone.
3. Why the mix matters
Every token of code is a token not spent on maths or history. More code improves coding and, many labs report, structured reasoning; more maths lifts problem solving; more multilingual data helps other languages but can cost a little English performance at small scale; more conversational web text helps fluency. Because a training run is fixed in compute, the mixture is a budget allocation, and small changes show up in benchmark scores. That is why GPT-3's paper published its mixture in 2020, and why most frontier labs no longer do.
4. Mix builder
Allocate a 15-trillion-token budget
Move the sliders; shares are normalised to 100%. Skill bars are an illustrative model of typical direction (diminishing returns, cross-domain spillover), not predictions for any real model.
5. Finding the mix
The proxy-model idea
An illustration of RegMix-style search: each dot is a tiny model trained on a random code/web/maths split; a fitted surface predicts the best mix before the expensive run.
6. Curricula and annealing
The mixture also changes over time. Most modern runs end with a short annealing (or "mid-training") phase: as the learning rate decays to zero, the mix shifts toward the highest-quality data (curated maths, code, textbooks, instruction-like text, long documents). Meta's Llama 3 paper describes annealing on upsampled high-quality sources and reports that this lifted maths benchmarks for its 8B model โ ๏ธ, and it used annealing on small samples to judge the value of new datasets cheaply. AI2's OLMo 2 published its mid-training mix ("Dolmino") โ ๏ธ. Long-context training then extends the window with long documents.
A typical training schedule
Tap a phase. Proportions and timings are illustrative.
7. Repetition: how many epochs?
When unique high-quality data runs out, labs repeat it. Muennighoff et al. (NeurIPS 2023, 400+ training runs up to 9B parameters and 900B tokens) found that up to about four epochs of repeated data were nearly as good as fresh data, after which the value of each extra repetition decays toward zero. Their fitted scaling law treats repeated tokens as worth less and less.
Repetition calculator
Effective data from repeating U unique tokens for E epochs, using the paper's functional form Dโฒ = U + UยทR*ยท(1 โ eโ(Eโ1)/R*) with R* โ 15 โ ๏ธ (approximate constant).
8. Balancing languages
Sampling languages in proportion to how much text exists would drown small languages in English; sampling them equally would repeat small languages dozens of times. The standard compromise is temperature sampling: sample each language with probability proportional to its size raised to a power ฮฑ between 0 and 1. mT5 used ฮฑ = 0.3. Lower ฮฑ gives small languages more weight at the cost of repetition.
Temperature sampling
Invented corpus sizes for six languages. Our arithmetic.
9. What labs disclose
| Model | Mixture disclosed |
|---|---|
| GPT-3 (2020) | Yes: weights per source (filtered Common Crawl, WebText2, books, Wikipedia) โ ๏ธ |
| Llama 3 (2024) | Shares by category (~50/25/17/8); annealing approach; not sources |
| OLMo / OLMo 2 (AI2) | Full mixtures and data released, including mid-training |
| DeepSeek-V3 | Total tokens and qualitative emphasis (more maths and code, multilingual) โ ๏ธ |
| GPT-4 onward, Claude, Gemini | Not disclosed |
10. Choosing a recipe
Start here
An editorial guide.
Inside the Data Layer
11. Reading the numbers
- Primary
- Llama 3 mix ~50% general, 25% maths and reasoning, 17% code, 8% multilingual (Meta paper). RegMix: beats human selection; matches or exceeds DoReMi with ~10% of the compute; up to 7B models on 100B tokens (paper). Muennighoff et al.: up to 4 epochs โ unique data; 400+ runs, up to 9B params and 900B tokens (NeurIPS 2023 / JMLR 2025).
- From the papers, not re-checked โ ๏ธ
- DoReMi's 2.6ร speed-up on The Pile; Llama 3 annealing gains; OLMo 2's Dolmino mix; R* โ 15 in the repetition law; mT5's ฮฑ = 0.3; GPT-3 and DeepSeek-V3 mixture details.
- Our illustrations
- Mix-builder skill scores, proxy-model demo, schedule proportions, language sizes and the chooser.