Home โ€บ Blog โ€บ Data mixtures
Inside the Data Layer ยท Part 6 ยท Mixtures

The Secret Recipe: How Labs Mix, Weight and Schedule Training Data

Two labs with the same data can train very different models, because what matters is not only what goes in but how much of each source, how often it repeats, and when the model sees it. The data mixture (web, code, maths, books, languages) is one of the most closely guarded secrets in AI. This part explains how mixtures are chosen, why the best data is saved for the end of training, how many times data can safely repeat, and how languages are balanced, with calculators for each.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are the Llama 3 paper, DoReMi, RegMix, "Scaling Data-Constrained Language Models" (Muennighoff et al.), mT5 and the OLMo reports. Details not re-checked against the primary paper are marked โš ๏ธ. The mix builder's skill scores, the proxy-model demo and schedule diagram are our own illustrations; the repetition and language calculators apply published formulas with approximate constants. See Reading the numbers.

1. Case file

โš–๏ธ
Data mixtures, October 2026Stage 7 ("mix") of the data supply chain
Llama 3 final mix (Meta)~50% general ยท 25% maths & reasoning ยท 17% code ยท 8% multilingual
DoReMi (2023)Learned domain weights: baseline accuracy 2.6ร— faster on The Pile โš ๏ธ
RegMix (2024)Predict the best mix from small proxies; matches DoReMi at ~10% of the compute
Repeated data (Muennighoff et al.)Up to ~4 epochs โ‰ˆ as good as unique data
AnnealingBest data saved for the final phase of training
Language balance (mT5)Temperature sampling, ฮฑ โ‰ˆ 0.3

2. Timeline

Choosing mixtures, 2019 โ†’ 2025

Tap a milestone.

Tap a milestone.

3. Why the mix matters

Every token of code is a token not spent on maths or history. More code improves coding and, many labs report, structured reasoning; more maths lifts problem solving; more multilingual data helps other languages but can cost a little English performance at small scale; more conversational web text helps fluency. Because a training run is fixed in compute, the mixture is a budget allocation, and small changes show up in benchmark scores. That is why GPT-3's paper published its mixture in 2020, and why most frontier labs no longer do.

4. Mix builder

Allocate a 15-trillion-token budget

Move the sliders; shares are normalised to 100%. Skill bars are an illustrative model of typical direction (diminishing returns, cross-domain spillover), not predictions for any real model.

Presets:
Hand-tunedEngineers set weights by intuition and ablations: train small models on a few candidate mixes and compare. Still common, slow and expensive.
DoReMiTrain a small proxy model that up-weights domains where it lags a reference model, then use its learned weights for the big run. On The Pile it reached the baseline's downstream accuracy in far fewer steps โš ๏ธ.
RegMixTrain many tiny models on random mixtures, fit a regression from mixture to performance, and pick the predicted best. Reported to beat human selection and match or exceed DoReMi with about 10% of the compute.
Scaling-law fitsFit how loss changes with each domain's share and extrapolate to the target scale; many labs now treat mixture as another hyperparameter with its own scaling laws.

The proxy-model idea

An illustration of RegMix-style search: each dot is a tiny model trained on a random code/web/maths split; a fitted surface predicts the best mix before the expensive run.

6. Curricula and annealing

The mixture also changes over time. Most modern runs end with a short annealing (or "mid-training") phase: as the learning rate decays to zero, the mix shifts toward the highest-quality data (curated maths, code, textbooks, instruction-like text, long documents). Meta's Llama 3 paper describes annealing on upsampled high-quality sources and reports that this lifted maths benchmarks for its 8B model โš ๏ธ, and it used annealing on small samples to judge the value of new datasets cheaply. AI2's OLMo 2 published its mid-training mix ("Dolmino") โš ๏ธ. Long-context training then extends the window with long documents.

A typical training schedule

Tap a phase. Proportions and timings are illustrative.

7. Repetition: how many epochs?

When unique high-quality data runs out, labs repeat it. Muennighoff et al. (NeurIPS 2023, 400+ training runs up to 9B parameters and 900B tokens) found that up to about four epochs of repeated data were nearly as good as fresh data, after which the value of each extra repetition decays toward zero. Their fitted scaling law treats repeated tokens as worth less and less.

Repetition calculator

Effective data from repeating U unique tokens for E epochs, using the paper's functional form Dโ€ฒ = U + UยทR*ยท(1 โˆ’ eโˆ’(Eโˆ’1)/R*) with R* โ‰ˆ 15 โš ๏ธ (approximate constant).

8. Balancing languages

Sampling languages in proportion to how much text exists would drown small languages in English; sampling them equally would repeat small languages dozens of times. The standard compromise is temperature sampling: sample each language with probability proportional to its size raised to a power ฮฑ between 0 and 1. mT5 used ฮฑ = 0.3. Lower ฮฑ gives small languages more weight at the cost of repetition.

Temperature sampling

Invented corpus sizes for six languages. Our arithmetic.

9. What labs disclose

ModelMixture disclosed
GPT-3 (2020)Yes: weights per source (filtered Common Crawl, WebText2, books, Wikipedia) โš ๏ธ
Llama 3 (2024)Shares by category (~50/25/17/8); annealing approach; not sources
OLMo / OLMo 2 (AI2)Full mixtures and data released, including mid-training
DeepSeek-V3Total tokens and qualitative emphasis (more maths and code, multilingual) โš ๏ธ
GPT-4 onward, Claude, GeminiNot disclosed

10. Choosing a recipe

Start here

An editorial guide.

VerdictThe mixture is where data strategy becomes model behaviour. The open literature has turned it from guesswork into a measurable search (proxy models, regressions, scaling laws), shown that good data can repeat a few times without harm, and made the final annealing phase a lever as strong as the bulk of training. The exact recipes of frontier models remain secret, but the method is no longer: search small, anneal on your best data, repeat carefully and balance languages deliberately.

Inside the Data Layer

11. Reading the numbers

Primary
Llama 3 mix ~50% general, 25% maths and reasoning, 17% code, 8% multilingual (Meta paper). RegMix: beats human selection; matches or exceeds DoReMi with ~10% of the compute; up to 7B models on 100B tokens (paper). Muennighoff et al.: up to 4 epochs โ‰ˆ unique data; 400+ runs, up to 9B params and 900B tokens (NeurIPS 2023 / JMLR 2025).
From the papers, not re-checked โš ๏ธ
DoReMi's 2.6ร— speed-up on The Pile; Llama 3 annealing gains; OLMo 2's Dolmino mix; R* โ‰ˆ 15 in the repetition law; mT5's ฮฑ = 0.3; GPT-3 and DeepSeek-V3 mixture details.
Our illustrations
Mix-builder skill scores, proxy-model demo, schedule proportions, language sizes and the chooser.