AI Research in 2019: The Year Pretraining Became the Default
In 2019 pretraining went from a promising technique to the default way of building AI. GPT-2 showed that next-word prediction alone produces general abilities, and a wave of BERT successors asked how pretraining should work. The same year exposed the next bottlenecks: context length, compute infrastructure, efficiency, robustness, evaluation and planning. Transformer-XL and sparse attention pushed context further, Megatron-LM spread single layers across GPUs, and ALBERT, DistilBERT and EfficientNet made models leaner. StyleGAN, AlphaStar, MuZero and new benchmarks rounded out the year. This guide covers sixteen papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| CVPR 2019 | StyleGAN — Best Paper Honorable Mention |
| NeurIPS 2019 | XLNet · Adversarial Examples Are Not Bugs · SuperGLUE · DistilBERT (EMC² workshop) |
| ACL, ICML, Interspeech 2019 | Transformer-XL (ACL) · EfficientNet (ICML) · wav2vec (Interspeech) |
| Nature | AlphaStar (2019) · MuZero (posted 2019, published 2020) |
| Posted 2019, published 2020 | ALBERT (ICLR 2020) · BART (ACL 2020) |
| Reports and preprints | GPT-2 · RoBERTa · Sparse Transformer · Megatron-LM |
02 Paper Map
Sixteen papers in eight themes. Filter and click to jump:
03 🌐 Large Language Models Start Showing Generality
Specialised NLP model → general pretrained language model.
GPT-2: Language Models are Unsupervised Multitask Learners
Radford et al. (OpenAI) · technical report, Feb 2019GPT-2 was trained on one job only: predict the next word of 8 million web pages. Yet at 1.5 billion parameters it could also answer questions, summarise and translate a little — tasks nobody trained it for. OpenAI released it in stages, worried about misuse.
- 1.5B-parameter decoder-only Transformer trained on WebText (about 40GB of text from outbound Reddit links)
- Zero-shot state of the art on 7 of 8 language-modelling datasets; weaker but non-trivial zero-shot reading comprehension, summarisation, translation and question answering
- Performance improved steadily with model size; staged release 124M → 355M → 774M → 1.5B through 2019
Showed that plain next-token prediction at scale produces general abilities — GPT-3 followed a year later.
04 💥 The Pretraining Explosion
BERT raised the question: how should we pretrain? Permutations, a better recipe, and denoising encoder–decoders.
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang et al. (CMU, Google Brain) · NeurIPS 2019BERT sees words on both sides but must fill in blanks independently; GPT predicts left to right but sees only one side. XLNet predicts words in many random orders, so each word learns from context on both sides while keeping GPT-style prediction.
- Maximises likelihood over permutations of the factorisation order; uses two-stream attention and Transformer-XL ideas
- Outperformed BERT on 20 tasks under comparable settings, often by large margins
Pushed the field to rethink pretraining objectives.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu et al. (Facebook AI, University of Washington) · arXiv Jul 2019Before inventing a new model, check the old one was trained properly. RoBERTa kept BERT's design but trained it longer, on 10× more data, with bigger batches — and matched or beat every newer model. BERT had simply been undertrained.
- Trained longer with larger batches on ~160GB of text (vs BERT's 16GB); dynamic masking; removed next-sentence prediction
- Matched or exceeded all post-BERT methods then published, with state of the art on GLUE, RACE and SQuAD
A lasting lesson: the training recipe can matter as much as the architecture.
BART: Denoising Sequence-to-Sequence Pre-training
Lewis et al. (Facebook AI) · arXiv Oct 2019 · ACL 2020Scramble a sentence — hide words, delete some, shuffle sentences — and train a model to write the original back out. BART pairs a BERT-like reader with a GPT-like writer, so it is good at both understanding and generating text.
- Bidirectional encoder + autoregressive decoder, pretrained to reconstruct corrupted text (text infilling and sentence shuffling worked best)
- Matched RoBERTa on GLUE and SQuAD; state of the art on summarisation, dialogue and abstractive QA with gains of up to 6 ROUGE
The generator inside 2020's RAG, and a template for encoder–decoder pretraining.
05 📜 Transformers Learn Longer Contexts
Fixed dense context → recurrence across segments and sparse attention → today's 100K–1M+ contexts.
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Dai et al. (CMU, Google Brain) · ACL 2019A normal Transformer reads text in separate chunks and forgets the previous chunk. Transformer-XL carries a memory of the last chunk forward, so information can flow across chunk boundaries — like remembering the last page while reading the next.
- Segment-level recurrence (cached hidden states) plus relative positional encoding
- Learned dependencies about 80% longer than RNNs and 450% longer than vanilla Transformers; up to 1,800× faster evaluation
- State of the art on several language-modelling benchmarks at the time
Its memory and relative positions reappear in XLNet and in later long-context work.
Generating Long Sequences with Sparse Transformers
Child, Gray, Radford & Sutskever (OpenAI) · arXiv Apr 2019Full attention compares every word with every other word, so doubling the length quadruples the work. Sparse Transformer lets each position look only at a nearby window plus a few regularly spaced positions, which still connects everything in two hops.
- Factorised sparse attention patterns reduce cost from O(n²) to O(n√n)
- Modelled sequences tens of thousands of steps long — about 30× longer than before — across images, audio and text
An early answer to the quadratic-cost problem behind today's long-context models.
06 🖧 Scaling Becomes a Systems-Engineering Problem
One layer too big for one GPU → split it across many, with communication in between.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi et al. (NVIDIA) · arXiv Sep 2019An 8-billion-parameter model is too big for one GPU. Megatron cuts each layer's big matrices into slices, puts one slice on each GPU, and stitches the results together with a quick exchange of numbers. Simple enough to add to ordinary PyTorch code.
- Intra-layer (tensor) model parallelism for Transformer attention and MLP blocks, needing only a few communication operations
- Trained an 8.3B-parameter GPT-2-style model on 512 V100 GPUs (8-way model parallel × 64-way data parallel), sustaining 15.1 PetaFLOPs at 76% scaling efficiency
- State of the art on WikiText-103 and LAMBADA; also a 3.9B BERT
Tensor parallelism is still a core ingredient of frontier training clusters.
07 🪶 Efficiency Becomes a Research Goal
Scale up (bigger models) vs scale smarter (smaller, faster, balanced models).
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan et al. (Google, Toyota Technological Institute) · ICLR 2020ALBERT shrinks BERT with two tricks: store word meanings in a smaller table and expand them, and let every layer share the same weights. Far fewer parameters, similar skills.
- Factorised embedding parameterisation and cross-layer parameter sharing; sentence-order prediction replaces next-sentence prediction
- ALBERT-large has about 18× fewer parameters than BERT-large; larger ALBERT configurations set state of the art on GLUE, RACE and SQuAD 2.0
Parameter efficiency became a research goal in its own right.
DistilBERT: a distilled version of BERT
Sanh et al. (Hugging Face) · NeurIPS 2019 EMC² workshopA big "teacher" model trains a small "student" to copy not just its answers but its confidence in every option. The student ends up much smaller and faster but almost as capable.
- Knowledge distillation applied during pretraining, with a triple loss (distillation, masked language modelling, cosine embedding)
- 40% smaller and 60% faster than BERT-base while retaining 97% of its language-understanding performance
Made pretrained models practical on phones and small servers; distillation remains standard.
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan & Le (Google) · ICML 2019To grow an image model you can make it deeper, wider, or feed it bigger images. EfficientNet showed the best results come from growing all three together in fixed proportion — compound scaling.
- Scale depth by α^φ, width by β^φ and resolution by γ^φ (α = 1.2, β = 1.1, γ = 1.15 found by a small grid search)
- EfficientNet-B7: 84.4% ImageNet top-1, while 8.4× smaller and 6.1× faster at inference than the best previous ConvNet (as reported at ICML)
Principled, balanced scaling — a vision-side precursor of 2020's scaling laws.
08 🎨 Generative Vision Reaches High Fidelity
Latent code → mapping network → styles at coarse, middle and fine scales → synthetic image.
StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, Laine & Aila (NVIDIA) · CVPR 2019StyleGAN makes photorealistic faces of people who don't exist. Its trick: a random code is turned into "styles" fed in at different sizes — coarse styles set pose and face shape, middle ones set features, fine ones set colour and texture.
- A mapping network turns the latent code into an intermediate style space; styles are injected at each resolution via adaptive instance normalisation; per-layer noise adds stochastic detail
- Better image quality and a more disentangled latent space; introduced the FFHQ face dataset
- CVPR 2019 Best Paper Honorable Mention
The high point of GANs — before diffusion took over from 2020–21.
09 ♟️ Reinforcement Learning in Complex Worlds
Known rules + planning → leagues of self-play agents and learned models + planning.
AlphaStar: Grandmaster Level in StarCraft II using Multi-Agent Reinforcement Learning
Vinyals et al. (DeepMind) · Nature 2019StarCraft II is a fast real-time strategy game with hidden information and thousands of possible actions. AlphaStar first learned by imitating human games, then improved in a league of agents that constantly invented and countered each other's strategies.
- Imitation learning, then reinforcement learning in a league with main agents and "exploiter" agents that target their weaknesses
- Reached Grandmaster for all three races on the official European server, rated above 99.8% of active ranked players, under human-like camera and action-rate limits
Populations of agents improving by competing — an idea close to today's self-improvement research.
MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
Schrittwieser et al. (DeepMind) · arXiv Nov 2019 · Nature 2020AlphaZero was told the rules of chess and Go and planned with them. MuZero is told nothing: it learns its own compact model of just what matters — rewards, how good a position is, which move to try — and plans inside that learned model.
- Learned representation, dynamics and prediction functions; Monte Carlo tree search in the learned latent space
- Matched AlphaZero in chess, shogi and Go without being given the rules; new state of the art on 57 Atari games
A step from planning with known rules toward agents that build their own world models.
10 🔬 Evaluation, Robustness and Self-Supervision Expand
What did models really learn, how robust are they, and do benchmarks still measure progress?
wav2vec: Unsupervised Pre-training for Speech Recognition
Schneider et al. (Facebook AI) · Interspeech 2019Listen to lots of untranscribed audio and learn to tell the real next bit of sound from fakes. The features learned this way make speech recognition work with far fewer transcripts.
- Convolutional encoder and context network trained with a contrastive loss to pick true future audio frames from distractors
- Up to 36% lower word error than a strong baseline when only a few hours of transcribed speech were available; beat Deep Speech 2 on WSJ with two orders of magnitude less labelled data
The first step toward 2020's wav2vec 2.0 and today's speech foundation models.
Adversarial Examples Are Not Bugs, They Are Features
Ilyas et al. (MIT) · NeurIPS 2019Tiny, invisible pixel changes can make a model call a panda a gibbon. This paper argues it isn't a glitch: images contain real but fragile patterns that predict the label, invisible to people. Models use them because they work — so the attack just flips those patterns.
- Constructed datasets that keep only robust or only non-robust features; models trained on non-robust features alone still generalised to the real test set
- Trained on images relabelled to the "wrong" class via adversarial perturbations, models still scored well on the original, correctly labelled test set
Reframed robustness as a property of the data and features models learn — relevant to every later question about what models really learn.
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang et al. (NYU, UW, DeepMind, Facebook AI) · NeurIPS 2019GLUE, the main language-understanding test, was being solved too fast. SuperGLUE kept the idea but used harder tasks, and published a human baseline to aim for.
- Eight harder tasks (e.g. BoolQ, MultiRC, ReCoRD, WiC, Winograd schemas) with a public leaderboard
- At launch BERT scored 69.0 against a human baseline of 89.8; by early 2021 models passed the human baseline
The pattern — new benchmark, rapid progress, saturation, harder benchmark — still runs AI evaluation.
11 Megatron's Split, Computed
Why did an 8.3B-parameter model need to be split, and why does splitting a layer give exactly the same answer? The script works out the training memory, then runs a tiny Transformer MLP on four simulated GPUs:
"""Megatron-LM in a few lines: why split a layer across GPUs, and why it works.
Part 1: memory. Training with Adam in mixed precision needs about 16 bytes
per parameter (fp16 weights 2 + fp16 gradients 2 + fp32 master weights,
momentum and variance 12), before counting activations. A 32GB V100 holds
far less than an 8.3B-parameter model.
Part 2: correctness. Megatron splits a Transformer MLP Y = GeLU(X A) B by
putting column blocks of A and the matching row blocks of B on different GPUs.
Each GPU computes its piece independently; one all-reduce (a sum) gives the
exact full output. Standard library only.
"""
import math, random
BYTES_PER_PARAM, GPU_GB = 16, 32
print("Part 1 - training memory (weights + grads + Adam state, no activations)")
for n in [0.34e9, 1.5e9, 8.3e9]:
gb = n * BYTES_PER_PARAM / 1e9
print(f" {n / 1e9:4.1f}B params -> {gb:6.1f} GB -> at least {math.ceil(gb / GPU_GB)} x 32GB GPUs")
print("\nPart 2 - tensor-parallel MLP on 4 'GPUs' (hidden 8, MLP width 16)")
random.seed(1)
gelu = lambda v: 0.5 * v * (1 + math.tanh(math.sqrt(2 / math.pi) * (v + 0.044715 * v ** 3)))
H, F, G = 8, 16, 4
X = [random.gauss(0, 1) for _ in range(H)]
A = [[random.gauss(0, .3) for _ in range(F)] for _ in range(H)]
B = [[random.gauss(0, .3) for _ in range(H)] for _ in range(F)]
def mlp(cols): # uses only columns `cols` of A and rows `cols` of B
hidden = {j: gelu(sum(X[i] * A[i][j] for i in range(H))) for j in cols}
return [sum(hidden[j] * B[j][k] for j in cols) for k in range(H)]
full = mlp(range(F))
parts = [mlp(range(g * F // G, (g + 1) * F // G)) for g in range(G)]
reduced = [sum(p[k] for p in parts) for k in range(H)] # the all-reduce
for g in range(G):
print(f" GPU {g + 1}: holds {H * F // G + F // G * H} of {2 * H * F} MLP weights, partial output[0] = {parts[g][0]:+.4f}")
print(f" all-reduce sum output[0] = {reduced[0]:+.4f} single-GPU output[0] = {full[0]:+.4f}")
print(f" max difference over all outputs: {max(abs(a - b) for a, b in zip(full, reduced)):.1e}")
print("\nSplitting A by columns keeps GeLU local to each GPU, so only one sum is needed.")Output:
Part 1 - training memory (weights + grads + Adam state, no activations)
0.3B params -> 5.4 GB -> at least 1 x 32GB GPUs
1.5B params -> 24.0 GB -> at least 1 x 32GB GPUs
8.3B params -> 132.8 GB -> at least 5 x 32GB GPUs
Part 2 - tensor-parallel MLP on 4 'GPUs' (hidden 8, MLP width 16)
GPU 1: holds 64 of 256 MLP weights, partial output[0] = -0.0276
GPU 2: holds 64 of 256 MLP weights, partial output[0] = +0.0491
GPU 3: holds 64 of 256 MLP weights, partial output[0] = -0.1799
GPU 4: holds 64 of 256 MLP weights, partial output[0] = +0.6505
all-reduce sum output[0] = +0.4921 single-GPU output[0] = +0.4921
max difference over all outputs: 2.2e-16
Splitting A by columns keeps GeLU local to each GPU, so only one sum is needed.With about 16 bytes per parameter, an 8.3B model needs roughly 133GB for weights, gradients and optimiser state alone, against 32GB on a V100. Activations push that higher, which is why Megatron split each model eight ways. The split is exact, not an approximation: each GPU computes part of the answer, and one sum puts it back together. See it in the Megatron card.
12 The 2017 → 2026 Roadmap
From the Transformer to self-improving systems. Press play:
Next: AI Research in 2018: The Year Pretraining Changed Everything →