Home › Blog › AI Research in 2019
AI Research Progress Through the Years · 2019 🏗️

AI Research in 2019: The Year Pretraining Became the Default

In 2019 pretraining went from a promising technique to the default way of building AI. GPT-2 showed that next-word prediction alone produces general abilities, and a wave of BERT successors asked how pretraining should work. The same year exposed the next bottlenecks: context length, compute infrastructure, efficiency, robustness, evaluation and planning. Transformer-XL and sparse attention pushed context further, Megatron-LM spread single layers across GPUs, and ALBERT, DistilBERT and EfficientNet made models leaner. StyleGAN, AlphaStar, MuZero and new benchmarks rounded out the year. This guide covers sixteen papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the eighth article in AI Research Progress Through the Years. It follows 2018 and is the prequel to 2020, 2021, 2022, 2023, 2024, 2025 and 2026. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.
▶ Two ways forward in 2019 Toggle

01 Venues and Awards

🏅 Best Paper Honorable Mention📄 Accepted paper📰 Nature📝 Preprint / report
VenueCovered here
CVPR 2019StyleGAN — Best Paper Honorable Mention
NeurIPS 2019XLNet · Adversarial Examples Are Not Bugs · SuperGLUE · DistilBERT (EMC² workshop)
ACL, ICML, Interspeech 2019Transformer-XL (ACL) · EfficientNet (ICML) · wav2vec (Interspeech)
NatureAlphaStar (2019) · MuZero (posted 2019, published 2020)
Posted 2019, published 2020ALBERT (ICLR 2020) · BART (ACL 2020)
Reports and preprintsGPT-2 · RoBERTa · Sparse Transformer · Megatron-LM

02 Paper Map

Sixteen papers in eight themes. Filter and click to jump:

▶ 2019 paper map Click to jump

03 🌐 Large Language Models Start Showing Generality

Specialised NLP model → general pretrained language model.

🦄

GPT-2: Language Models are Unsupervised Multitask Learners

Radford et al. (OpenAI) · technical report, Feb 2019
📝 Preprint / report
🧒 In plain English

GPT-2 was trained on one job only: predict the next word of 8 million web pages. Yet at 1.5 billion parameters it could also answer questions, summarise and translate a little — tasks nobody trained it for. OpenAI released it in stages, worried about misuse.

▶ One objective, many tasks Pick a task
🔍 What they found
  • 1.5B-parameter decoder-only Transformer trained on WebText (about 40GB of text from outbound Reddit links)
  • Zero-shot state of the art on 7 of 8 language-modelling datasets; weaker but non-trivial zero-shot reading comprehension, summarisation, translation and question answering
  • Performance improved steadily with model size; staged release 124M → 355M → 774M → 1.5B through 2019
💡 Why it matters

Showed that plain next-token prediction at scale produces general abilities — GPT-3 followed a year later.

04 💥 The Pretraining Explosion

BERT raised the question: how should we pretrain? Permutations, a better recipe, and denoising encoder–decoders.

🔀

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Yang et al. (CMU, Google Brain) · NeurIPS 2019
📄 Accepted paper
🧒 In plain English

BERT sees words on both sides but must fill in blanks independently; GPT predicts left to right but sees only one side. XLNet predicts words in many random orders, so each word learns from context on both sides while keeping GPT-style prediction.

▶ Permutation language modelling Press shuffle
🔍 What they found
  • Maximises likelihood over permutations of the factorisation order; uses two-stream attention and Transformer-XL ideas
  • Outperformed BERT on 20 tasks under comparable settings, often by large margins
💡 Why it matters

Pushed the field to rethink pretraining objectives.

🔗 References
🧪

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu et al. (Facebook AI, University of Washington) · arXiv Jul 2019
📝 Preprint / report
🧒 In plain English

Before inventing a new model, check the old one was trained properly. RoBERTa kept BERT's design but trained it longer, on 10× more data, with bigger batches — and matched or beat every newer model. BERT had simply been undertrained.

▶ Same architecture, better recipe Press train
🔍 What they found
  • Trained longer with larger batches on ~160GB of text (vs BERT's 16GB); dynamic masking; removed next-sentence prediction
  • Matched or exceeded all post-BERT methods then published, with state of the art on GLUE, RACE and SQuAD
💡 Why it matters

A lasting lesson: the training recipe can matter as much as the architecture.

🧩

BART: Denoising Sequence-to-Sequence Pre-training

Lewis et al. (Facebook AI) · arXiv Oct 2019 · ACL 2020
📄 Accepted paper
🧒 In plain English

Scramble a sentence — hide words, delete some, shuffle sentences — and train a model to write the original back out. BART pairs a BERT-like reader with a GPT-like writer, so it is good at both understanding and generating text.

▶ Corrupt, then reconstruct Pick a corruption
🔍 What they found
  • Bidirectional encoder + autoregressive decoder, pretrained to reconstruct corrupted text (text infilling and sentence shuffling worked best)
  • Matched RoBERTa on GLUE and SQuAD; state of the art on summarisation, dialogue and abstractive QA with gains of up to 6 ROUGE
💡 Why it matters

The generator inside 2020's RAG, and a template for encoder–decoder pretraining.

05 📜 Transformers Learn Longer Contexts

Fixed dense context → recurrence across segments and sparse attention → today's 100K–1M+ contexts.

🧵

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Dai et al. (CMU, Google Brain) · ACL 2019
📄 Accepted paper
🧒 In plain English

A normal Transformer reads text in separate chunks and forgets the previous chunk. Transformer-XL carries a memory of the last chunk forward, so information can flow across chunk boundaries — like remembering the last page while reading the next.

▶ Chunks with and without memory Toggle, then read
🔍 What they found
  • Segment-level recurrence (cached hidden states) plus relative positional encoding
  • Learned dependencies about 80% longer than RNNs and 450% longer than vanilla Transformers; up to 1,800× faster evaluation
  • State of the art on several language-modelling benchmarks at the time
💡 Why it matters

Its memory and relative positions reappear in XLNet and in later long-context work.

🕸️

Generating Long Sequences with Sparse Transformers

Child, Gray, Radford & Sutskever (OpenAI) · arXiv Apr 2019
📝 Preprint / report
🧒 In plain English

Full attention compares every word with every other word, so doubling the length quadruples the work. Sparse Transformer lets each position look only at a nearby window plus a few regularly spaced positions, which still connects everything in two hops.

▶ Who attends to whom Toggle
🔍 What they found
  • Factorised sparse attention patterns reduce cost from O(n²) to O(n√n)
  • Modelled sequences tens of thousands of steps long — about 30× longer than before — across images, audio and text
💡 Why it matters

An early answer to the quadratic-cost problem behind today's long-context models.

06 🖧 Scaling Becomes a Systems-Engineering Problem

One layer too big for one GPU → split it across many, with communication in between.

🖥️

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Shoeybi et al. (NVIDIA) · arXiv Sep 2019
📝 Preprint / report
🧒 In plain English

An 8-billion-parameter model is too big for one GPU. Megatron cuts each layer's big matrices into slices, puts one slice on each GPU, and stitches the results together with a quick exchange of numbers. Simple enough to add to ordinary PyTorch code.

▶ Splitting a layer across GPUs Press run
🔍 What they found
  • Intra-layer (tensor) model parallelism for Transformer attention and MLP blocks, needing only a few communication operations
  • Trained an 8.3B-parameter GPT-2-style model on 512 V100 GPUs (8-way model parallel × 64-way data parallel), sustaining 15.1 PetaFLOPs at 76% scaling efficiency
  • State of the art on WikiText-103 and LAMBADA; also a 3.9B BERT
💡 Why it matters

Tensor parallelism is still a core ingredient of frontier training clusters.

07 🪶 Efficiency Becomes a Research Goal

Scale up (bigger models) vs scale smarter (smaller, faster, balanced models).

🪶

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Lan et al. (Google, Toyota Technological Institute) · ICLR 2020
📄 Accepted paper
🧒 In plain English

ALBERT shrinks BERT with two tricks: store word meanings in a smaller table and expand them, and let every layer share the same weights. Far fewer parameters, similar skills.

▶ Where the parameters go Toggle
🔍 What they found
  • Factorised embedding parameterisation and cross-layer parameter sharing; sentence-order prediction replaces next-sentence prediction
  • ALBERT-large has about 18× fewer parameters than BERT-large; larger ALBERT configurations set state of the art on GLUE, RACE and SQuAD 2.0
💡 Why it matters

Parameter efficiency became a research goal in its own right.

🔗 References
⚗️

DistilBERT: a distilled version of BERT

Sanh et al. (Hugging Face) · NeurIPS 2019 EMC² workshop
📄 Accepted paper
🧒 In plain English

A big "teacher" model trains a small "student" to copy not just its answers but its confidence in every option. The student ends up much smaller and faster but almost as capable.

▶ Teacher and student Press distil
🔍 What they found
  • Knowledge distillation applied during pretraining, with a triple loss (distillation, masked language modelling, cosine embedding)
  • 40% smaller and 60% faster than BERT-base while retaining 97% of its language-understanding performance
💡 Why it matters

Made pretrained models practical on phones and small servers; distillation remains standard.

📐

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Tan & Le (Google) · ICML 2019
📄 Accepted paper
🧒 In plain English

To grow an image model you can make it deeper, wider, or feed it bigger images. EfficientNet showed the best results come from growing all three together in fixed proportion — compound scaling.

▶ Compound scaling Drag the scale
🔍 What they found
  • Scale depth by α^φ, width by β^φ and resolution by γ^φ (α = 1.2, β = 1.1, γ = 1.15 found by a small grid search)
  • EfficientNet-B7: 84.4% ImageNet top-1, while 8.4× smaller and 6.1× faster at inference than the best previous ConvNet (as reported at ICML)
💡 Why it matters

Principled, balanced scaling — a vision-side precursor of 2020's scaling laws.

08 🎨 Generative Vision Reaches High Fidelity

Latent code → mapping network → styles at coarse, middle and fine scales → synthetic image.

🎭

StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks

Karras, Laine & Aila (NVIDIA) · CVPR 2019
🏅 Best Paper Honorable Mention
🧒 In plain English

StyleGAN makes photorealistic faces of people who don't exist. Its trick: a random code is turned into "styles" fed in at different sizes — coarse styles set pose and face shape, middle ones set features, fine ones set colour and texture.

▶ Mixing styles by layer Pick which layers take style B
🔍 What they found
  • A mapping network turns the latent code into an intermediate style space; styles are injected at each resolution via adaptive instance normalisation; per-layer noise adds stochastic detail
  • Better image quality and a more disentangled latent space; introduced the FFHQ face dataset
  • CVPR 2019 Best Paper Honorable Mention
💡 Why it matters

The high point of GANs — before diffusion took over from 2020–21.

09 ♟️ Reinforcement Learning in Complex Worlds

Known rules + planning → leagues of self-play agents and learned models + planning.

🛸

AlphaStar: Grandmaster Level in StarCraft II using Multi-Agent Reinforcement Learning

Vinyals et al. (DeepMind) · Nature 2019
📰 Nature
🧒 In plain English

StarCraft II is a fast real-time strategy game with hidden information and thousands of possible actions. AlphaStar first learned by imitating human games, then improved in a league of agents that constantly invented and countered each other's strategies.

▶ The league Press play
🔍 What they found
  • Imitation learning, then reinforcement learning in a league with main agents and "exploiter" agents that target their weaknesses
  • Reached Grandmaster for all three races on the official European server, rated above 99.8% of active ranked players, under human-like camera and action-rate limits
💡 Why it matters

Populations of agents improving by competing — an idea close to today's self-improvement research.

🧭

MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Schrittwieser et al. (DeepMind) · arXiv Nov 2019 · Nature 2020
📰 Nature
🧒 In plain English

AlphaZero was told the rules of chess and Go and planned with them. MuZero is told nothing: it learns its own compact model of just what matters — rewards, how good a position is, which move to try — and plans inside that learned model.

▶ Plan inside a learned model Toggle
🔍 What they found
  • Learned representation, dynamics and prediction functions; Monte Carlo tree search in the learned latent space
  • Matched AlphaZero in chess, shogi and Go without being given the rules; new state of the art on 57 Atari games
💡 Why it matters

A step from planning with known rules toward agents that build their own world models.

10 🔬 Evaluation, Robustness and Self-Supervision Expand

What did models really learn, how robust are they, and do benchmarks still measure progress?

🎙️

wav2vec: Unsupervised Pre-training for Speech Recognition

Schneider et al. (Facebook AI) · Interspeech 2019
📄 Accepted paper
🧒 In plain English

Listen to lots of untranscribed audio and learn to tell the real next bit of sound from fakes. The features learned this way make speech recognition work with far fewer transcripts.

▶ Real future vs distractors Press play
🔍 What they found
  • Convolutional encoder and context network trained with a contrastive loss to pick true future audio frames from distractors
  • Up to 36% lower word error than a strong baseline when only a few hours of transcribed speech were available; beat Deep Speech 2 on WSJ with two orders of magnitude less labelled data
💡 Why it matters

The first step toward 2020's wav2vec 2.0 and today's speech foundation models.

🐼

Adversarial Examples Are Not Bugs, They Are Features

Ilyas et al. (MIT) · NeurIPS 2019
📄 Accepted paper
🧒 In plain English

Tiny, invisible pixel changes can make a model call a panda a gibbon. This paper argues it isn't a glitch: images contain real but fragile patterns that predict the label, invisible to people. Models use them because they work — so the attack just flips those patterns.

▶ Robust vs non-robust features Toggle
🔍 What they found
  • Constructed datasets that keep only robust or only non-robust features; models trained on non-robust features alone still generalised to the real test set
  • Trained on images relabelled to the "wrong" class via adversarial perturbations, models still scored well on the original, correctly labelled test set
💡 Why it matters

Reframed robustness as a property of the data and features models learn — relevant to every later question about what models really learn.

📊

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Wang et al. (NYU, UW, DeepMind, Facebook AI) · NeurIPS 2019
📄 Accepted paper
🧒 In plain English

GLUE, the main language-understanding test, was being solved too fast. SuperGLUE kept the idea but used harder tasks, and published a human baseline to aim for.

▶ The benchmark cycle Press play
🔍 What they found
  • Eight harder tasks (e.g. BoolQ, MultiRC, ReCoRD, WiC, Winograd schemas) with a public leaderboard
  • At launch BERT scored 69.0 against a human baseline of 89.8; by early 2021 models passed the human baseline
💡 Why it matters

The pattern — new benchmark, rapid progress, saturation, harder benchmark — still runs AI evaluation.

11 Megatron's Split, Computed

Why did an 8.3B-parameter model need to be split, and why does splitting a layer give exactly the same answer? The script works out the training memory, then runs a tiny Transformer MLP on four simulated GPUs:

"""Megatron-LM in a few lines: why split a layer across GPUs, and why it works.

Part 1: memory. Training with Adam in mixed precision needs about 16 bytes
per parameter (fp16 weights 2 + fp16 gradients 2 + fp32 master weights,
momentum and variance 12), before counting activations. A 32GB V100 holds
far less than an 8.3B-parameter model.

Part 2: correctness. Megatron splits a Transformer MLP  Y = GeLU(X A) B  by
putting column blocks of A and the matching row blocks of B on different GPUs.
Each GPU computes its piece independently; one all-reduce (a sum) gives the
exact full output. Standard library only.
"""
import math, random

BYTES_PER_PARAM, GPU_GB = 16, 32
print("Part 1 - training memory (weights + grads + Adam state, no activations)")
for n in [0.34e9, 1.5e9, 8.3e9]:
    gb = n * BYTES_PER_PARAM / 1e9
    print(f"   {n / 1e9:4.1f}B params -> {gb:6.1f} GB  -> at least {math.ceil(gb / GPU_GB)} x 32GB GPUs")

print("\nPart 2 - tensor-parallel MLP on 4 'GPUs' (hidden 8, MLP width 16)")
random.seed(1)
gelu = lambda v: 0.5 * v * (1 + math.tanh(math.sqrt(2 / math.pi) * (v + 0.044715 * v ** 3)))
H, F, G = 8, 16, 4
X = [random.gauss(0, 1) for _ in range(H)]
A = [[random.gauss(0, .3) for _ in range(F)] for _ in range(H)]
B = [[random.gauss(0, .3) for _ in range(H)] for _ in range(F)]

def mlp(cols):                     # uses only columns `cols` of A and rows `cols` of B
    hidden = {j: gelu(sum(X[i] * A[i][j] for i in range(H))) for j in cols}
    return [sum(hidden[j] * B[j][k] for j in cols) for k in range(H)]

full = mlp(range(F))
parts = [mlp(range(g * F // G, (g + 1) * F // G)) for g in range(G)]
reduced = [sum(p[k] for p in parts) for k in range(H)]      # the all-reduce
for g in range(G):
    print(f"   GPU {g + 1}: holds {H * F // G + F // G * H} of {2 * H * F} MLP weights, partial output[0] = {parts[g][0]:+.4f}")
print(f"   all-reduce sum output[0] = {reduced[0]:+.4f}   single-GPU output[0] = {full[0]:+.4f}")
print(f"   max difference over all outputs: {max(abs(a - b) for a, b in zip(full, reduced)):.1e}")
print("\nSplitting A by columns keeps GeLU local to each GPU, so only one sum is needed.")

Output:

Part 1 - training memory (weights + grads + Adam state, no activations)
    0.3B params ->    5.4 GB  -> at least 1 x 32GB GPUs
    1.5B params ->   24.0 GB  -> at least 1 x 32GB GPUs
    8.3B params ->  132.8 GB  -> at least 5 x 32GB GPUs

Part 2 - tensor-parallel MLP on 4 'GPUs' (hidden 8, MLP width 16)
   GPU 1: holds 64 of 256 MLP weights, partial output[0] = -0.0276
   GPU 2: holds 64 of 256 MLP weights, partial output[0] = +0.0491
   GPU 3: holds 64 of 256 MLP weights, partial output[0] = -0.1799
   GPU 4: holds 64 of 256 MLP weights, partial output[0] = +0.6505
   all-reduce sum output[0] = +0.4921   single-GPU output[0] = +0.4921
   max difference over all outputs: 2.2e-16

Splitting A by columns keeps GeLU local to each GPU, so only one sum is needed.

With about 16 bytes per parameter, an 8.3B model needs roughly 133GB for weights, gradients and optimiser state alone, against 32GB on a V100. Activations push that higher, which is why Megatron split each model eight ways. The split is exact, not an approximation: each GPU computes part of the answer, and one sum puts it back together. See it in the Megatron card.

12 The 2017 → 2026 Roadmap

From the Transformer to self-improving systems. Press play:

▶ Ten years, twelve stages Click a stage or press play
2019Make pretraining the default
→
2020Discover predictable scaling
→
2021Assemble the foundation-model stack
→
2022Add reasoning, alignment and action
→
2023Programmable platforms
→
2024Systems around models
→
2025Reasoning and agency
→
2026Persistent self-improvement
AI research in 2019, in one paragraph
2019 made pretraining the dominant way to build AI. GPT-2 showed that next-word prediction at scale produces general abilities. XLNet, RoBERTa and BART showed that the objective and the training recipe matter as much as the architecture. Transformer-XL and Sparse Transformer took on context length, and Megatron-LM turned scale into a distributed-systems problem. ALBERT, DistilBERT and EfficientNet argued for scaling smarter, not just bigger. StyleGAN marked the peak of GANs. AlphaStar's leagues and MuZero's learned models pushed reinforcement learning into complex worlds. wav2vec, the adversarial-features paper and SuperGLUE asked what models really learn and how to measure it. Each of these bottlenecks returns in the articles that follow.

Next: AI Research in 2018: The Year Pretraining Changed Everything →

⚠️ Notes

About the source infographic and brief: (1) the GPT-2 chart labels its y-axis "zero-shot accuracy" on a log scale reaching 10¹, which is impossible for an accuracy; it also lists HellaSwag and StoryCloze, which were not among GPT-2's reported evaluations (HellaSwag was published after GPT-2) — the paper used LAMBADA, the Children's Book Test, Winograd schemas, CoQA and others; (2) the panda → gibbon image in the adversarial-examples panel comes from Goodfellow et al. (2014), not from Ilyas et al. (2019); (3) MuZero was posted in November 2019 but published in Nature in December 2020; (4) recognition missing from the infographic is added: StyleGAN received a CVPR 2019 Best Paper Honorable Mention, and AlphaStar was published in Nature. Other figures checked out against the papers (GPT-2's 1.5B parameters and staged release, XLNet's 20 tasks, Transformer-XL's longer dependencies, O(n√n) sparse attention, Megatron's 8.3B parameters on 512 GPUs, ALBERT's 18×, DistilBERT's 40% / 60% / 97%, EfficientNet-B7's 84.4%, AlphaStar's 99.8%). Widget numbers not taken from a paper are marked illustrative.

🔗 All References