AI Research in 2018: The Year Pretraining Changed Everything
2018 changed how AI was built: instead of training one model per task, researchers pretrained a reusable representation once and adapted it everywhere. ELMo made word meanings depend on context, and ULMFiT, GPT-1 and BERT turned pretrain-then-fine-tune into the standard recipe. GLUE became the scoreboard. Beyond language, world models learned to imagine and self-play mastered games. Graph networks reasoned about structure, Neural ODEs made depth continuous and BigGAN produced high-fidelity images. This guide covers fourteen papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| NeurIPS 2018 (Montréal) | Best Paper: Neural ODEs (one of four). Also: World Models (as "Recurrent World Models Facilitate Policy Evolution") |
| NAACL 2018 / 2019 | ELMo — NAACL 2018 Outstanding Paper · BERT (posted Oct 2018) — NAACL 2019 Best Long Paper |
| ACL, ICML 2018 | ULMFiT (ACL) · Graph physics engines (ICML) |
| Science | AlphaZero · Generative Query Network |
| Posted 2018, published 2019 | BigGAN and GLUE (ICLR 2019) |
| Reports and preprints | GPT-1 · OpenAI Five · Dactyl · Graph Networks |
02 Paper Map
Fourteen papers in eight themes. Filter by award and click to jump:
03 🌊 Contextual Language Representations
One fixed vector per word → a representation that depends on the sentence around it.
ELMo: Deep Contextualized Word Representations
Peters et al. (Allen Institute for AI, University of Washington) · NAACL 2018Before ELMo, the word "bank" had one fixed meaning-vector whether it was a river bank or a money bank. ELMo reads the whole sentence first with a pretrained language model, then gives each word a vector that fits its context.
- Word vectors are learned functions of all internal layers of a deep bidirectional LSTM language model pretrained on a large corpus
- Improved the state of the art on six tasks: question answering, textual entailment, semantic role labelling, coreference, named-entity recognition and sentiment
- NAACL 2018 Outstanding Paper
Made contextual, pretrained representations standard — the step before BERT.
04 🔁 Pretrain → Fine-Tune Becomes the New NLP Recipe
Unlabelled text → pretrain once → adapt to many tasks.
ULMFiT: Universal Language Model Fine-tuning for Text Classification
Howard & Ruder (fast.ai, Insight Centre) · ACL 2018Computer vision already reused ImageNet models; ULMFiT brought that to text. Pretrain a language model on Wikipedia, tune it on your kind of text, then on your task — carefully, one layer at a time — and a few hundred labelled examples go a long way.
- Three stages: general LM pretraining (WikiText-103), target-domain LM fine-tuning, classifier fine-tuning
- Techniques: discriminative learning rates, slanted triangular schedules, gradual unfreezing
- 18–24% lower error on most of six text-classification datasets; with 100 labelled examples it matched training from scratch on 100× more data
Showed that transfer learning works for NLP — and in low-data settings.
GPT-1: Improving Language Understanding by Generative Pre-Training
Radford et al. (OpenAI) · technical report, Jun 2018Train a Transformer to predict the next word across 7,000 unpublished books, then fine-tune it briefly for each task, reshaping the inputs so one model fits them all. The first GPT.
- 12-layer Transformer decoder (~117M parameters) pretrained on BooksCorpus, then fine-tuned with minimal task-specific changes via input transformations
- Improved the state of the art on 9 of 12 language-understanding tasks studied
The start of the GPT line — and proof that Transformer pretraining transfers.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin et al. (Google) · arXiv Oct 2018 · NAACL 2019GPT reads left to right. BERT hides 15% of the words in a sentence and learns to fill them in using words on both sides. Add one small output layer, fine-tune for an hour, and it beat specialised systems on eleven tasks.
- Masked language modelling plus next-sentence prediction on BooksCorpus and English Wikipedia; BERT-base 110M and BERT-large 340M parameters
- State of the art on eleven tasks, including GLUE 80.5 (7.7 points absolute improvement) and SQuAD v1.1 F1 93.2
- NAACL 2019 Best Long Paper
Became the default NLP backbone; 2019's XLNet, RoBERTa and ALBERT all start from it.
05 📊 General Language Evaluation Becomes Important
One pretrained model → sentiment, entailment, similarity, QA → one aggregate score.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang et al. (NYU, UW, DeepMind) · BlackboxNLP 2018 · ICLR 2019If one model claims to understand language in general, test it on many different language tasks at once — sentiment, paraphrase, entailment, similarity — and report one combined score. GLUE became the scoreboard of the pretraining race.
- Nine sentence and sentence-pair tasks with a public leaderboard, plus a diagnostic set for linguistic phenomena
- Favoured models that share knowledge across tasks; within about a year models passed its human baseline, prompting SuperGLUE (2019)
Made general transfer, not single-task scores, the target — the start of the benchmark cycle.
06 🌍 World Models and Learned Internal Environments
Observe the world → learn an internal model → imagine consequences → act.
World Models
Ha & Schmidhuber · arXiv Mar 2018 · NeurIPS 2018 (as "Recurrent World Models Facilitate Policy Evolution")An agent watches a video game and learns a compact dream of it: a compressed picture of the screen and a prediction of what comes next. A tiny controller can then practise inside the dream and still play the real game well.
- V: a VAE compresses each frame; M: an MDN-RNN predicts the next latent; C: a tiny linear controller trained with evolution strategies (CMA-ES)
- First reported solution of CarRacing-v0 (average score above 900)
- In VizDoom, a controller trained entirely inside the learned model transferred back to the actual game
The ancestor of Dreamer, MuZero-style learned models and today's world-model research.
Neural Scene Representation and Rendering (Generative Query Network)
Eslami et al. (DeepMind) · Science 2018Show a network a few pictures of a room from different spots. It builds its own internal description of the scene, then draws the room from a new viewpoint it has never seen — without anyone labelling depth or objects.
- A representation network turns a few observations into a scene vector; a generation network renders any queried viewpoint
- Learned from images alone, no labels; captured object identity, position, colour and room layout
An early neural scene representation — a sibling of 2020's NeRF.
07 ♟️ Self-Play and Large-Scale Reinforcement Learning
Self-play or simulation → massive experience → learned strategy → complex or real environments.
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Shogi and Go through Self-Play
Silver et al. (DeepMind) · Science 2018Give one program only the rules of chess, shogi or Go. It plays against itself millions of times, learning which moves and positions are good, and becomes the strongest player of all three games.
- One algorithm: a policy/value network plus Monte Carlo tree search, trained only by self-play from the rules
- Defeated Stockfish (chess), Elmo (shogi) and AlphaGo Zero (Go)
- Preprint Dec 2017; peer-reviewed Science paper Dec 2018
Generalised self-play mastery; MuZero (2019) later removed even the rules.
OpenAI Five
OpenAI · 2018 (paper: Berner et al., arXiv 2019)Dota 2 is a team game with hidden information and long matches. OpenAI Five trained five cooperating agents by playing against themselves — about 180 years of games every day on huge clusters.
- Scaled-up PPO with LSTM policies and self-play; 128,000 CPU cores and 256 GPUs in mid-2018
- In 2018 it beat amateur and semi-professional teams but lost both games against professionals at The International; it beat world champions OG in April 2019
Showed how far scaled-up RL and self-play could go in complex, team-based worlds.
Dactyl: Learning Dexterous In-Hand Manipulation
OpenAI · arXiv Aug 2018Teach a robot hand to turn a block to any face you ask — trained entirely in simulation. The trick: randomise everything in the simulator (friction, weight, colours, lighting) so the real world looks like just another variation.
- Reinforcement learning in heavily randomised simulation, using the same distributed training system as OpenAI Five
- The policy transferred to a physical Shadow Dexterous Hand without real-world training, reaching up to 50 consecutive successful rotations of a block
A landmark in sim-to-real transfer (the Rubik's cube version followed in 2019).
08 🕸️ Relational Reasoning and Graph Networks
Objects as nodes, relations as edges, message passing for structured reasoning.
Relational Inductive Biases, Deep Learning, and Graph Networks
Battaglia et al. (DeepMind, Google Brain, MIT, Edinburgh) · arXiv Jun 2018Many problems are about things and their relationships: atoms in a molecule, bodies in a physical system, people in a network. This paper argued networks should be built around that structure, and proposed a general "graph network" block.
- Unifies many earlier models as graph networks that update edges, then nodes, then a global state
- Argues structured "relational inductive biases" are key to combinatorial generalisation; released the graph_nets library
A foundation for graph learning in chemistry, physics simulation and recommendation — and for AlphaFold-style reasoning over pairs.
Graph Networks as Learnable Physics Engines for Inference and Control
Sanchez-Gonzalez et al. (DeepMind) · ICML 2018Describe a robot or a swimmer as parts (nodes) joined by joints (edges). A graph network learns to predict how it will move — a learned physics engine — and that model can then be used to plan actions.
- Graph-network forward models for simulated bodies (e.g. DeepMind Control Suite) and a real JACO robotic arm
- Accurate multi-step prediction that generalised to new body shapes; used for system identification and model-based control
Led to learned simulators for fluids and materials, and to world models for robots.
09 〰️ Neural Networks Become Continuous
A stack of discrete layers → a continuous transformation defined by a learned differential equation.
Neural Ordinary Differential Equations
Chen, Rubanova, Bettencourt & Duvenaud (University of Toronto, Vector Institute) · NeurIPS 2018A ResNet updates its hidden state in fixed steps, one per layer. A Neural ODE asks instead: what if the state changed smoothly, following a rule learned by a network? An ODE solver then decides how many steps to take.
- Hidden-state dynamics defined by a neural network and integrated with a black-box ODE solver; trained with the adjoint method at constant memory cost
- Adaptive computation (trade precision for speed), continuous normalising flows, and latent ODEs for irregularly sampled time series
- NeurIPS 2018 Best Paper (one of four)
Opened the field of continuous-depth models, neural differential equations and flow-based generative models.
10 🎨 Generative Vision Scales Up
Bigger GANs, bigger batches → high-fidelity images, and a dial between fidelity and diversity.
BigGAN: Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, Donahue & Simonyan (Heriot-Watt, DeepMind) · arXiv Sep 2018 · ICLR 2019Make a GAN much bigger and train it with much bigger batches, and ImageNet pictures suddenly look real. Plus a dial — the "truncation trick" — that trades variety for polish.
- 2–4× more parameters and 8× larger batches than prior work, with orthogonal regularisation for stability
- On ImageNet 128×128: Inception Score 166.5 and FID 7.4, versus the previous best of 52.5 and 18.7
- Truncation trick: sampling the latent from a narrower range raises fidelity at the cost of diversity; documented the instability of training at scale
The peak of class-conditional GANs, followed by StyleGAN (2019) and then diffusion (2020 onwards).
11 A ResNet Is an ODE Solver
The key idea behind Neural ODEs fits in a few lines. A residual block, h + f(h)·dt, is one Euler step of a differential equation. More layers give a finer solver, and an adaptive solver takes only as many steps as the accuracy target needs:
"""Neural ODEs in a few lines: a ResNet is a coarse ODE solver.
A residual block computes h_next = h + f(h) * dt. That is one Euler step of
the differential equation dh/dt = f(h). Chen et al. (2018) replaced the stack
of blocks with the equation itself and let an ODE solver choose the steps.
Here f is fixed (a rotation, so the exact answer is known): the hidden state
should travel a quarter-turn from (1, 0) to (0, 1) between t = 0 and t = 1.
A "ResNet" with more layers = Euler with more, smaller steps. An adaptive
solver takes as many steps as the accuracy target needs. Standard library only.
"""
import math
def f(h): # dh/dt: rotate by 90 degrees per unit time
x, y = h
w = math.pi / 2
return (-w * y, w * x)
def resnet(layers): # h <- h + f(h) * dt, one layer per step
h, dt = (1.0, 0.0), 1.0 / layers
for _ in range(layers):
d = f(h); h = (h[0] + dt * d[0], h[1] + dt * d[1])
return h
def rk4(h, dt):
k1 = f(h); k2 = f((h[0] + dt / 2 * k1[0], h[1] + dt / 2 * k1[1]))
k3 = f((h[0] + dt / 2 * k2[0], h[1] + dt / 2 * k2[1])); k4 = f((h[0] + dt * k3[0], h[1] + dt * k3[1]))
return (h[0] + dt / 6 * (k1[0] + 2 * k2[0] + 2 * k3[0] + k4[0]), h[1] + dt / 6 * (k1[1] + 2 * k2[1] + 2 * k3[1] + k4[1]))
def adaptive(tol): # step doubling: shrink dt until two half-steps agree
t, h, dt, steps = 0.0, (1.0, 0.0), 0.5, 0
while t < 1 - 1e-12:
dt = min(dt, 1 - t)
full = rk4(h, dt); half = rk4(rk4(h, dt / 2), dt / 2)
if math.dist(full, half) < tol:
t, h, steps = t + dt, half, steps + 1; dt *= 1.5
else:
dt /= 2
return h, steps
exact = (0.0, 1.0)
print("ResNet-style (Euler) layers vs error at t = 1")
for n in [2, 4, 8, 16, 64, 256]:
print(f" {n:4d} layers -> error {math.dist(resnet(n), exact):.4f} (stores {n} activations for backprop)")
print("\nAdaptive ODE solver (RK4 with step doubling)")
for tol in [1e-2, 1e-4, 1e-6, 1e-8]:
h, s = adaptive(tol)
print(f" tolerance {tol:.0e} -> {s:3d} steps, error {math.dist(h, exact):.1e}")
print("\nMore layers = a finer solver; an ODE solver picks the depth for the accuracy you ask for,")
print("and the adjoint method trains it with memory that does not grow with the number of steps.")Output:
ResNet-style (Euler) layers vs error at t = 1
2 layers -> error 0.6875 (stores 2 activations for backprop)
4 layers -> error 0.3430 (stores 4 activations for backprop)
8 layers -> error 0.1647 (stores 8 activations for backprop)
16 layers -> error 0.0799 (stores 16 activations for backprop)
64 layers -> error 0.0195 (stores 64 activations for backprop)
256 layers -> error 0.0048 (stores 256 activations for backprop)
Adaptive ODE solver (RK4 with step doubling)
tolerance 1e-02 -> 2 steps, error 3.1e-04
tolerance 1e-04 -> 5 steps, error 1.0e-05
tolerance 1e-06 -> 13 steps, error 2.6e-07
tolerance 1e-08 -> 33 steps, error 6.0e-09
More layers = a finer solver; an ODE solver picks the depth for the accuracy you ask for,
and the adjoint method trains it with memory that does not grow with the number of steps.Doubling the ResNet's depth roughly halves the error, because Euler steps are first-order. The adaptive solver reaches far higher accuracy in a few dozen steps, though each RK4 step evaluates f four times. In a real Neural ODE, f is a trained network; here it is a fixed rotation so that the exact answer is known. See it in the Neural ODE card.
12 The 2017 → 2026 Roadmap
From the Transformer to self-improving systems. Press play:
Next: AI Research in 2017: The Year the Modern AI Architecture Was Born →