Home › Blog › AI Research in 2018
AI Research Progress Through the Years · 2018 🔓

AI Research in 2018: The Year Pretraining Changed Everything

2018 changed how AI was built: instead of training one model per task, researchers pretrained a reusable representation once and adapted it everywhere. ELMo made word meanings depend on context, and ULMFiT, GPT-1 and BERT turned pretrain-then-fine-tune into the standard recipe. GLUE became the scoreboard. Beyond language, world models learned to imagine and self-play mastered games. Graph networks reasoned about structure, Neural ODEs made depth continuous and BigGAN produced high-fidelity images. This guide covers fourteen papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the ninth article in AI Research Progress Through the Years. It follows 2017, and the story continues in 2019, 2020, 2021, 2022, 2023, 2024, 2025 and 2026. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.
▶ The new unit of AI development Toggle

01 Venues and Awards

🏆 Best / Outstanding Paper📄 Accepted paper📰 Science📝 Preprint / report
VenueCovered here
NeurIPS 2018 (Montréal)Best Paper: Neural ODEs (one of four). Also: World Models (as "Recurrent World Models Facilitate Policy Evolution")
NAACL 2018 / 2019ELMo — NAACL 2018 Outstanding Paper · BERT (posted Oct 2018) — NAACL 2019 Best Long Paper
ACL, ICML 2018ULMFiT (ACL) · Graph physics engines (ICML)
ScienceAlphaZero · Generative Query Network
Posted 2018, published 2019BigGAN and GLUE (ICLR 2019)
Reports and preprintsGPT-1 · OpenAI Five · Dactyl · Graph Networks

02 Paper Map

Fourteen papers in eight themes. Filter by award and click to jump:

▶ 2018 paper map Click to jump

03 🌊 Contextual Language Representations

One fixed vector per word → a representation that depends on the sentence around it.

🌊

ELMo: Deep Contextualized Word Representations

Peters et al. (Allen Institute for AI, University of Washington) · NAACL 2018
🏆 Best / Outstanding Paper
🧒 In plain English

Before ELMo, the word "bank" had one fixed meaning-vector whether it was a river bank or a money bank. ELMo reads the whole sentence first with a pretrained language model, then gives each word a vector that fits its context.

▶ Same word, different meaning Pick a sentence
🔍 What they found
  • Word vectors are learned functions of all internal layers of a deep bidirectional LSTM language model pretrained on a large corpus
  • Improved the state of the art on six tasks: question answering, textual entailment, semantic role labelling, coreference, named-entity recognition and sentiment
  • NAACL 2018 Outstanding Paper
💡 Why it matters

Made contextual, pretrained representations standard — the step before BERT.

04 🔁 Pretrain → Fine-Tune Becomes the New NLP Recipe

Unlabelled text → pretrain once → adapt to many tasks.

🎚️

ULMFiT: Universal Language Model Fine-tuning for Text Classification

Howard & Ruder (fast.ai, Insight Centre) · ACL 2018
📄 Accepted paper
🧒 In plain English

Computer vision already reused ImageNet models; ULMFiT brought that to text. Pretrain a language model on Wikipedia, tune it on your kind of text, then on your task — carefully, one layer at a time — and a few hundred labelled examples go a long way.

▶ Three stages Press run
🔍 What they found
  • Three stages: general LM pretraining (WikiText-103), target-domain LM fine-tuning, classifier fine-tuning
  • Techniques: discriminative learning rates, slanted triangular schedules, gradual unfreezing
  • 18–24% lower error on most of six text-classification datasets; with 100 labelled examples it matched training from scratch on 100× more data
💡 Why it matters

Showed that transfer learning works for NLP — and in low-data settings.

📖

GPT-1: Improving Language Understanding by Generative Pre-Training

Radford et al. (OpenAI) · technical report, Jun 2018
📝 Preprint / report
🧒 In plain English

Train a Transformer to predict the next word across 7,000 unpublished books, then fine-tune it briefly for each task, reshaping the inputs so one model fits them all. The first GPT.

▶ One model, reshaped inputs Pick a task
🔍 What they found
  • 12-layer Transformer decoder (~117M parameters) pretrained on BooksCorpus, then fine-tuned with minimal task-specific changes via input transformations
  • Improved the state of the art on 9 of 12 language-understanding tasks studied
💡 Why it matters

The start of the GPT line — and proof that Transformer pretraining transfers.

🧸

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin et al. (Google) · arXiv Oct 2018 · NAACL 2019
🏆 Best / Outstanding Paper
🧒 In plain English

GPT reads left to right. BERT hides 15% of the words in a sentence and learns to fill them in using words on both sides. Add one small output layer, fine-tune for an hour, and it beat specialised systems on eleven tasks.

▶ Fill in the blanks Press predict
🔍 What they found
  • Masked language modelling plus next-sentence prediction on BooksCorpus and English Wikipedia; BERT-base 110M and BERT-large 340M parameters
  • State of the art on eleven tasks, including GLUE 80.5 (7.7 points absolute improvement) and SQuAD v1.1 F1 93.2
  • NAACL 2019 Best Long Paper
💡 Why it matters

Became the default NLP backbone; 2019's XLNet, RoBERTa and ALBERT all start from it.

05 📊 General Language Evaluation Becomes Important

One pretrained model → sentiment, entailment, similarity, QA → one aggregate score.

📊

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Wang et al. (NYU, UW, DeepMind) · BlackboxNLP 2018 · ICLR 2019
📄 Accepted paper
🧒 In plain English

If one model claims to understand language in general, test it on many different language tasks at once — sentiment, paraphrase, entailment, similarity — and report one combined score. GLUE became the scoreboard of the pretraining race.

▶ One score, nine tasks Press evaluate
🔍 What they found
  • Nine sentence and sentence-pair tasks with a public leaderboard, plus a diagnostic set for linguistic phenomena
  • Favoured models that share knowledge across tasks; within about a year models passed its human baseline, prompting SuperGLUE (2019)
💡 Why it matters

Made general transfer, not single-task scores, the target — the start of the benchmark cycle.

06 🌍 World Models and Learned Internal Environments

Observe the world → learn an internal model → imagine consequences → act.

💭

World Models

Ha & Schmidhuber · arXiv Mar 2018 · NeurIPS 2018 (as "Recurrent World Models Facilitate Policy Evolution")
📄 Accepted paper
🧒 In plain English

An agent watches a video game and learns a compact dream of it: a compressed picture of the screen and a prediction of what comes next. A tiny controller can then practise inside the dream and still play the real game well.

▶ Vision, memory, controller Press play
🔍 What they found
  • V: a VAE compresses each frame; M: an MDN-RNN predicts the next latent; C: a tiny linear controller trained with evolution strategies (CMA-ES)
  • First reported solution of CarRacing-v0 (average score above 900)
  • In VizDoom, a controller trained entirely inside the learned model transferred back to the actual game
💡 Why it matters

The ancestor of Dreamer, MuZero-style learned models and today's world-model research.

🧊

Neural Scene Representation and Rendering (Generative Query Network)

Eslami et al. (DeepMind) · Science 2018
📰 Science
🧒 In plain English

Show a network a few pictures of a room from different spots. It builds its own internal description of the scene, then draws the room from a new viewpoint it has never seen — without anyone labelling depth or objects.

▶ Observe, then imagine a new view Press run
🔍 What they found
  • A representation network turns a few observations into a scene vector; a generation network renders any queried viewpoint
  • Learned from images alone, no labels; captured object identity, position, colour and room layout
💡 Why it matters

An early neural scene representation — a sibling of 2020's NeRF.

07 ♟️ Self-Play and Large-Scale Reinforcement Learning

Self-play or simulation → massive experience → learned strategy → complex or real environments.

♞

AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Shogi and Go through Self-Play

Silver et al. (DeepMind) · Science 2018
📰 Science
🧒 In plain English

Give one program only the rules of chess, shogi or Go. It plays against itself millions of times, learning which moves and positions are good, and becomes the strongest player of all three games.

▶ Self-play loop Press play
🔍 What they found
  • One algorithm: a policy/value network plus Monte Carlo tree search, trained only by self-play from the rules
  • Defeated Stockfish (chess), Elmo (shogi) and AlphaGo Zero (Go)
  • Preprint Dec 2017; peer-reviewed Science paper Dec 2018
💡 Why it matters

Generalised self-play mastery; MuZero (2019) later removed even the rules.

🛡️

OpenAI Five

OpenAI · 2018 (paper: Berner et al., arXiv 2019)
📝 Preprint / report
🧒 In plain English

Dota 2 is a team game with hidden information and long matches. OpenAI Five trained five cooperating agents by playing against themselves — about 180 years of games every day on huge clusters.

▶ The 2018 season Press play
🔍 What they found
  • Scaled-up PPO with LSTM policies and self-play; 128,000 CPU cores and 256 GPUs in mid-2018
  • In 2018 it beat amateur and semi-professional teams but lost both games against professionals at The International; it beat world champions OG in April 2019
💡 Why it matters

Showed how far scaled-up RL and self-play could go in complex, team-based worlds.

🖐️

Dactyl: Learning Dexterous In-Hand Manipulation

OpenAI · arXiv Aug 2018
📝 Preprint / report
🧒 In plain English

Teach a robot hand to turn a block to any face you ask — trained entirely in simulation. The trick: randomise everything in the simulator (friction, weight, colours, lighting) so the real world looks like just another variation.

▶ Domain randomisation Press randomise
🔍 What they found
  • Reinforcement learning in heavily randomised simulation, using the same distributed training system as OpenAI Five
  • The policy transferred to a physical Shadow Dexterous Hand without real-world training, reaching up to 50 consecutive successful rotations of a block
💡 Why it matters

A landmark in sim-to-real transfer (the Rubik's cube version followed in 2019).

08 🕸️ Relational Reasoning and Graph Networks

Objects as nodes, relations as edges, message passing for structured reasoning.

🕸️

Relational Inductive Biases, Deep Learning, and Graph Networks

Battaglia et al. (DeepMind, Google Brain, MIT, Edinburgh) · arXiv Jun 2018
📝 Preprint / report
🧒 In plain English

Many problems are about things and their relationships: atoms in a molecule, bodies in a physical system, people in a network. This paper argued networks should be built around that structure, and proposed a general "graph network" block.

▶ Message passing Press step
🔍 What they found
  • Unifies many earlier models as graph networks that update edges, then nodes, then a global state
  • Argues structured "relational inductive biases" are key to combinatorial generalisation; released the graph_nets library
💡 Why it matters

A foundation for graph learning in chemistry, physics simulation and recommendation — and for AlphaFold-style reasoning over pairs.

🪐

Graph Networks as Learnable Physics Engines for Inference and Control

Sanchez-Gonzalez et al. (DeepMind) · ICML 2018
📄 Accepted paper
🧒 In plain English

Describe a robot or a swimmer as parts (nodes) joined by joints (edges). A graph network learns to predict how it will move — a learned physics engine — and that model can then be used to plan actions.

▶ A learned simulator Press simulate
🔍 What they found
  • Graph-network forward models for simulated bodies (e.g. DeepMind Control Suite) and a real JACO robotic arm
  • Accurate multi-step prediction that generalised to new body shapes; used for system identification and model-based control
💡 Why it matters

Led to learned simulators for fluids and materials, and to world models for robots.

09 〰️ Neural Networks Become Continuous

A stack of discrete layers → a continuous transformation defined by a learned differential equation.

〰️

Neural Ordinary Differential Equations

Chen, Rubanova, Bettencourt & Duvenaud (University of Toronto, Vector Institute) · NeurIPS 2018
🏆 Best / Outstanding Paper
🧒 In plain English

A ResNet updates its hidden state in fixed steps, one per layer. A Neural ODE asks instead: what if the state changed smoothly, following a rule learned by a network? An ODE solver then decides how many steps to take.

▶ Layers vs a continuous flow Drag the number of layers
🔍 What they found
  • Hidden-state dynamics defined by a neural network and integrated with a black-box ODE solver; trained with the adjoint method at constant memory cost
  • Adaptive computation (trade precision for speed), continuous normalising flows, and latent ODEs for irregularly sampled time series
  • NeurIPS 2018 Best Paper (one of four)
💡 Why it matters

Opened the field of continuous-depth models, neural differential equations and flow-based generative models.

10 🎨 Generative Vision Scales Up

Bigger GANs, bigger batches → high-fidelity images, and a dial between fidelity and diversity.

🐶

BigGAN: Large Scale GAN Training for High Fidelity Natural Image Synthesis

Brock, Donahue & Simonyan (Heriot-Watt, DeepMind) · arXiv Sep 2018 · ICLR 2019
📄 Accepted paper
🧒 In plain English

Make a GAN much bigger and train it with much bigger batches, and ImageNet pictures suddenly look real. Plus a dial — the "truncation trick" — that trades variety for polish.

▶ The truncation dial Drag truncation
🔍 What they found
  • 2–4× more parameters and 8× larger batches than prior work, with orthogonal regularisation for stability
  • On ImageNet 128×128: Inception Score 166.5 and FID 7.4, versus the previous best of 52.5 and 18.7
  • Truncation trick: sampling the latent from a narrower range raises fidelity at the cost of diversity; documented the instability of training at scale
💡 Why it matters

The peak of class-conditional GANs, followed by StyleGAN (2019) and then diffusion (2020 onwards).

🔗 References

11 A ResNet Is an ODE Solver

The key idea behind Neural ODEs fits in a few lines. A residual block, h + f(h)·dt, is one Euler step of a differential equation. More layers give a finer solver, and an adaptive solver takes only as many steps as the accuracy target needs:

"""Neural ODEs in a few lines: a ResNet is a coarse ODE solver.

A residual block computes  h_next = h + f(h) * dt.  That is one Euler step of
the differential equation  dh/dt = f(h).  Chen et al. (2018) replaced the stack
of blocks with the equation itself and let an ODE solver choose the steps.

Here f is fixed (a rotation, so the exact answer is known): the hidden state
should travel a quarter-turn from (1, 0) to (0, 1) between t = 0 and t = 1.
A "ResNet" with more layers = Euler with more, smaller steps. An adaptive
solver takes as many steps as the accuracy target needs. Standard library only.
"""
import math

def f(h):                                   # dh/dt: rotate by 90 degrees per unit time
    x, y = h
    w = math.pi / 2
    return (-w * y, w * x)

def resnet(layers):                         # h <- h + f(h) * dt, one layer per step
    h, dt = (1.0, 0.0), 1.0 / layers
    for _ in range(layers):
        d = f(h); h = (h[0] + dt * d[0], h[1] + dt * d[1])
    return h

def rk4(h, dt):
    k1 = f(h); k2 = f((h[0] + dt / 2 * k1[0], h[1] + dt / 2 * k1[1]))
    k3 = f((h[0] + dt / 2 * k2[0], h[1] + dt / 2 * k2[1])); k4 = f((h[0] + dt * k3[0], h[1] + dt * k3[1]))
    return (h[0] + dt / 6 * (k1[0] + 2 * k2[0] + 2 * k3[0] + k4[0]), h[1] + dt / 6 * (k1[1] + 2 * k2[1] + 2 * k3[1] + k4[1]))

def adaptive(tol):                          # step doubling: shrink dt until two half-steps agree
    t, h, dt, steps = 0.0, (1.0, 0.0), 0.5, 0
    while t < 1 - 1e-12:
        dt = min(dt, 1 - t)
        full = rk4(h, dt); half = rk4(rk4(h, dt / 2), dt / 2)
        if math.dist(full, half) < tol:
            t, h, steps = t + dt, half, steps + 1; dt *= 1.5
        else:
            dt /= 2
    return h, steps

exact = (0.0, 1.0)
print("ResNet-style (Euler) layers vs error at t = 1")
for n in [2, 4, 8, 16, 64, 256]:
    print(f"   {n:4d} layers -> error {math.dist(resnet(n), exact):.4f}   (stores {n} activations for backprop)")
print("\nAdaptive ODE solver (RK4 with step doubling)")
for tol in [1e-2, 1e-4, 1e-6, 1e-8]:
    h, s = adaptive(tol)
    print(f"   tolerance {tol:.0e} -> {s:3d} steps, error {math.dist(h, exact):.1e}")
print("\nMore layers = a finer solver; an ODE solver picks the depth for the accuracy you ask for,")
print("and the adjoint method trains it with memory that does not grow with the number of steps.")

Output:

ResNet-style (Euler) layers vs error at t = 1
      2 layers -> error 0.6875   (stores 2 activations for backprop)
      4 layers -> error 0.3430   (stores 4 activations for backprop)
      8 layers -> error 0.1647   (stores 8 activations for backprop)
     16 layers -> error 0.0799   (stores 16 activations for backprop)
     64 layers -> error 0.0195   (stores 64 activations for backprop)
    256 layers -> error 0.0048   (stores 256 activations for backprop)

Adaptive ODE solver (RK4 with step doubling)
   tolerance 1e-02 ->   2 steps, error 3.1e-04
   tolerance 1e-04 ->   5 steps, error 1.0e-05
   tolerance 1e-06 ->  13 steps, error 2.6e-07
   tolerance 1e-08 ->  33 steps, error 6.0e-09

More layers = a finer solver; an ODE solver picks the depth for the accuracy you ask for,
and the adjoint method trains it with memory that does not grow with the number of steps.

Doubling the ResNet's depth roughly halves the error, because Euler steps are first-order. The adaptive solver reaches far higher accuracy in a few dozen steps, though each RK4 step evaluates f four times. In a real Neural ODE, f is a trained network; here it is a fixed rotation so that the exact answer is known. See it in the Neural ODE card.

12 The 2017 → 2026 Roadmap

From the Transformer to self-improving systems. Press play:

▶ Ten years, thirteen stages Click a stage or press play
2018One pretrained representation transfers broadly
→
2019Pretraining becomes the default
→
2020Scale it aggressively
→
2021Foundation-model stack
→
2022Reasoning, alignment, action
→
2023+Agentic, then self-improving systems
AI research in 2018, in one paragraph
2018 changed the unit of AI development from "train a model for one task" to "pretrain a reusable representation once, then adapt it everywhere." ELMo made representations contextual. ULMFiT, GPT-1 and BERT made pretrain-then-fine-tune the standard NLP recipe, and GLUE gave it a scoreboard. Alongside, other work pointed to broader intelligence. World Models and GQN learned internal models of environments, and AlphaZero, OpenAI Five and Dactyl showed what self-play and simulation at scale could do. Graph networks brought relational structure, Neural ODEs made depth continuous, and BigGAN showed that scale works for image generation too. Every later year in this series builds on these pieces.

Next: AI Research in 2017: The Year the Modern AI Architecture Was Born →

⚠️ Notes

About the source infographic and brief: (1) the OpenAI Five panel says it "defeated top human players in 5v5 matches"; in 2018 it beat amateur and semi-professional teams but lost both games against professionals at The International 2018; it beat the world champions OG in April 2019; (2) the Dactyl image shows a Rubik's cube; the 2018 Dactyl work manipulated a lettered block, and the Rubik's cube result came in October 2019; (3) the ELMo panel lists "POS" among its six improved tasks; the six benchmark tasks were question answering, textual entailment, semantic role labelling, coreference, named-entity recognition and sentiment; (4) the World Models panel shows a controller acting in the "real world"; in the paper "real environment" means the actual game (CarRacing, VizDoom), not the physical world; (5) awards missing from the infographic are added: Neural ODEs (NeurIPS 2018 Best Paper), ELMo (NAACL 2018 Outstanding Paper) and BERT (NAACL 2019 Best Long Paper). Other figures checked out against the papers (GPT-1's 117M parameters, BERT's eleven tasks, GLUE's nine tasks, AlphaZero from the rules only). The suggested POLO callout is left out to keep the focus on the main themes. Widget numbers not taken from a paper are marked illustrative.

🔗 All References