AI Research in 2017: The Year the Modern AI Architecture Was Born
2017 supplied several of the deepest building blocks of modern AI at once. The Transformer became the architecture. AlphaGo Zero showed self-play as a way to generate experience. PPO gave reinforcement learning a scalable loop. Human preferences became a reward signal. Architecture search and population-based training let AI start designing AI, and sparse experts opened a route to enormous capacity. Generative vision became controllable, and capsules explored a different path. This guide covers eleven papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues
| Venue | Covered here |
|---|---|
| NIPS 2017 (Long Beach) | Attention Is All You Need · Deep RL from Human Preferences · Dynamic Routing Between Capsules |
| ICLR 2017 | Neural Architecture Search · Sparsely-Gated Mixture-of-Experts |
| ICML, CVPR, ICCV 2017 | Wasserstein GAN (ICML) · pix2pix (CVPR) · CycleGAN (ICCV) |
| Nature | AlphaGo Zero |
| Preprints | PPO · Population Based Training |
None of these took a NIPS 2017 Best Paper award; those went to other work, including the subgame-solving paper behind the poker AI Libratus. Most of the papers here were recognised as landmarks only in hindsight.
02 Paper Map
Eleven papers in eight themes. Filter and click to jump:
03 ⚡ Attention Is All You Need: The Transformer
Sequential recurrence → every position attends to every other, in parallel.
Attention Is All You Need
Vaswani et al. (Google Brain, Google Research, University of Toronto) · NIPS 2017Older language models read a sentence one word at a time, like passing a note down a line. The Transformer lets every word look at every other word at once and decide which ones matter to it. That made training far faster on GPUs — and it turned out to work for almost everything.
- Encoder–decoder built only from multi-head self-attention, feed-forward layers, residual connections, layer normalisation and positional encodings
- 28.4 BLEU on WMT 2014 English→German (over 2 BLEU above previous best, including ensembles) and 41.8 BLEU on English→French after 3.5 days on eight GPUs
- Far more parallelisable than recurrent models, with short paths between distant positions
The architecture behind GPT, BERT, T5, ViT and every frontier model since.
04 ♟️ Self-Play Creates Its Own Training Data
A system that generates the experience that trains its next version.
AlphaGo Zero: Mastering the Game of Go without Human Knowledge
Silver et al. (DeepMind) · Nature 2017The first AlphaGo learned from human expert games. AlphaGo Zero started from random play and only played against itself. Each round of games trained a better network, which played better games, which trained an even better network.
- A single residual network outputs both move probabilities and a value; Monte Carlo tree search guided by the network generates self-play games that train it
- No human games or hand-crafted features — only the rules
- Beat the version that defeated Lee Sedol 100–0 after 3 days; beat AlphaGo Master 89–11 after 40 days
The clearest early example of a system generating the data that trains its next version.
05 🧑⚖️ Human Preferences Become a Reward Function
Behaviour A vs behaviour B → human choice → reward model → RL.
Deep Reinforcement Learning from Human Preferences
Christiano, Leike, Brown, Martic, Legg & Amodei (OpenAI, DeepMind) · NIPS 2017Some goals are hard to write as a score ("do a nice backflip"). Instead, show a person two short clips and ask which is better. A reward model learns from those choices, and the agent learns from the reward model.
- Learned a reward model from human comparisons of short trajectory segments, and trained the policy with RL against it — asynchronously, as feedback arrived
- Solved Atari games and simulated robot tasks with human feedback on less than 1% of the agent's interactions
- Learned novel behaviours such as a simulated backflip from about an hour of human feedback
The conceptual core of RLHF — later used in 2020's summarisation work, InstructGPT and ChatGPT.
06 📎 PPO Makes Reinforcement Learning Easier to Scale
Collect experience → estimate advantage → clipped update → repeat.
Proximal Policy Optimization Algorithms (PPO)
Schulman, Wolski, Dhariwal, Radford & Klimov (OpenAI) · arXiv Jul 2017Reinforcement learning can be unstable: one over-eager update can wreck a good policy. PPO limits how far each update can move the policy — if an action already became much more likely, it stops pushing. Simple to implement, and it just works.
- Clipped surrogate objective: the probability ratio between new and old policy is clipped to [1−ε, 1+ε] (ε = 0.2 in the paper)
- Several epochs of minibatch updates per batch of experience
- Better balance of sample efficiency, simplicity and wall-clock time than other online policy-gradient methods on MuJoCo control and Atari
Became the default RL algorithm — used for OpenAI Five, Dactyl and the PPO stage of RLHF.
07 🧬 AI Begins Designing AI
Generate candidate models → train and evaluate → reward, select, copy and mutate.
Neural Architecture Search with Reinforcement Learning
Zoph & Le (Google Brain) · ICLR 2017Let one network design another. A controller writes out a network's blueprint, the network is trained and tested, and its accuracy is the controller's reward. After thousands of tries, the controller proposes architectures as good as expert designs.
- An RNN controller samples architecture descriptions; child networks are trained; validation accuracy is the reward for REINFORCE
- CIFAR-10: 3.65% test error, competitive with the best human-designed networks; Penn Treebank: a new recurrent cell with 62.4 perplexity
- Used 800 GPUs concurrently for the search
An early case of AI searching over the design of AI — ancestor of AutoML and of today's automated research.
Population Based Training of Neural Networks
Jaderberg et al. (DeepMind) · arXiv Nov 2017Train a whole population of models at once. Every so often, weak ones copy a strong one's weights (exploit) and tweak its settings, such as the learning rate (explore). Settings end up changing over time, like an evolving schedule.
- Asynchronous population training that jointly optimises weights and adaptive hyperparameter schedules
- Faster training and better final performance across deep RL (DeepMind Lab, Atari, StarCraft II), machine translation and GANs, with the same compute as a normal hyperparameter search
Selection, inheritance and mutation — the pattern behind evolutionary agents, AlphaEvolve and harness populations.
08 🔀 Sparse Mixture-of-Experts
Store many parameters, use only a few per input.
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer et al. (Google Brain) · ICLR 2017Build a network with thousands of specialist sub-networks, and a small gate that picks just a few of them for each input. The model can hold enormous knowledge while each input costs only a little computation.
- A trainable noisy top-k gate selects a few experts out of up to thousands per example, with load-balancing losses
- Models up to 137 billion parameters, placed between LSTM layers, with only minor losses in computational efficiency
- Better than the state of the art on language modelling and machine translation at lower compute
The start of sparse scaling: GShard (2020), Switch Transformer (2021) and many of today's frontier models.
09 🎨 Generative Vision Becomes Controllable
GANs that translate between image domains, with or without paired examples, and train more stably.
pix2pix: Image-to-Image Translation with Conditional Adversarial Networks
Isola, Zhu, Zhou & Efros (UC Berkeley) · CVPR 2017Give a network many pairs of pictures — a sketch and the photo, a map and the aerial view — and it learns to turn one into the other. The same method works for all of them; only the training pairs change.
- Conditional GAN with a U-Net generator and a PatchGAN discriminator that judges local image patches
- One general framework learns both the mapping and the loss, across many paired translation tasks
Made conditional generation practical; its community demos (e.g. #edges2cats) spread widely.
CycleGAN: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Zhu, Park, Isola & Efros (UC Berkeley) · ICCV 2017There are no photos of the same horse as a zebra. CycleGAN learns horse→zebra and zebra→horse at the same time, with one rule: turn a horse into a zebra and back, and you should get the original horse.
- Two generators (G: X→Y, F: Y→X) with adversarial losses plus a cycle-consistency loss F(G(x)) ≈ x
- Unpaired translation: horses↔zebras, summer↔winter, photos↔paintings in the style of Monet, Van Gogh and others
Removed the need for paired data; cycle consistency reappears in many later methods.
Wasserstein GAN
Arjovsky, Chintala & Bottou (NYU, Facebook AI) · ICML 2017GANs were notoriously hard to train, and their loss number told you almost nothing. WGAN measures the gap between real and fake images as the effort needed to move one pile of sand into the shape of another. That measure gives a smooth signal, so training is steadier and the loss actually tracks image quality.
- Replaces the Jensen–Shannon-based GAN objective with an approximation of the Wasserstein (earth mover's) distance, using a weight-clipped critic
- More stable training, less mode collapse, and a loss that correlates with sample quality
- Weight clipping was later replaced by a gradient penalty (WGAN-GP, NIPS 2017)
Put GAN training on firmer theoretical ground, feeding into BigGAN and StyleGAN.
10 🧿 Alternative Architectures and Representation Learning
RNNs? CNNs? Capsules? Attention? In 2017 the winner wasn't yet clear.
Dynamic Routing Between Capsules
Sabour, Frosst & Hinton (Google Brain) · NIPS 2017A normal image network detects "eye", "nose" and "mouth" but not always how they fit together. Capsules output vectors that also encode pose, and lower-level parts "vote" for the whole object; parts whose votes agree get routed together.
- Capsules: groups of neurons whose activity vector represents an entity's presence (length) and pose (orientation)
- Routing-by-agreement sends each part's output to the parent capsules its prediction agrees with
- 0.25% test error on MNIST, and much better recognition of heavily overlapping digits (MultiMNIST) than a comparable CNN
Didn't become the dominant architecture, but influenced work on equivariance, part–whole structure and object-centric learning.
11 Self-Attention, From Scratch
The Transformer's core operation takes a dozen lines. Each token builds a query, a key and a value; attention weights come from comparing queries with keys, and each token's new vector is a weighted mix of all the values:
"""Self-attention from scratch: the core operation of the Transformer.
Attention(Q, K, V) = softmax(Q K^T / sqrt(d)) V (Vaswani et al., 2017)
Each token asks a question (query), advertises what it contains (key), and
carries information (value). Every token looks at every other token in ONE
step, so all positions can be computed in parallel - unlike an RNN, which
must process position 1 before position 2. Toy 4-dimensional vectors,
hand-picked so the result is easy to read. Standard library only.
"""
import math
tokens = ["The", "animal", "was", "tired"]
# toy embeddings: dims = [is-noun, is-verb, animacy, state]
X = [[0.1, 0.0, 0.0, 0.0], [1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 0.2], [0.0, 0.3, 0.2, 1.0]]
# projections chosen so "state" words query for animate nouns
WQ = [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 0, 3, 0]]
WK = [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]
WV = WK
mat = lambda A, B: [[sum(a * b for a, b in zip(r, c)) for c in zip(*B)] for r in A]
Q, K, V = mat(X, WQ), mat(X, WK), mat(X, WV)
d = len(Q[0])
def softmax(xs):
m = max(xs); e = [math.exp(x - m) for x in xs]; s = sum(e)
return [v / s for v in e]
weights = [softmax([sum(q * k for q, k in zip(Q[i], K[j])) / math.sqrt(d) for j in range(len(tokens))]) for i in range(len(tokens))]
out = mat(weights, V)
print("Attention weights (row = token asking, column = token attended to)")
print(f"{'':>8}" + "".join(f"{t:>8}" for t in tokens))
for t, row in zip(tokens, weights):
print(f"{t:>8}" + "".join(f"{w:>8.2f}" for w in row))
i = tokens.index("tired")
print(f"\n'tired' attends most to '{tokens[max(range(4), key=lambda j: weights[i][j])]}'"
f" (weight {max(weights[i]):.2f}): its new vector mixes in the noun it describes.")
print(" new 'tired' vector:", [round(v, 2) for v in out[i]])
print("\nSequential steps needed to connect the first and last token")
print(f"{'length':>8} {'RNN':>6} {'self-attention':>15}")
for n in [4, 64, 1024]:
print(f"{n:>8} {n - 1:>6} {1:>15}")
print("\nAll pairs interact in one layer: more parallel, and shorter paths for long-range dependencies.")Output:
Attention weights (row = token asking, column = token attended to)
The animal was tired
The 0.25 0.25 0.25 0.25
animal 0.21 0.35 0.21 0.23
was 0.23 0.31 0.23 0.24
tired 0.12 0.59 0.12 0.17
'tired' attends most to 'animal' (weight 0.59): its new vector mixes in the noun it describes.
new 'tired' vector: [0.61, 0.17, 0.63, 0.19]
Sequential steps needed to connect the first and last token
length RNN self-attention
4 3 1
64 63 1
1024 1023 1
All pairs interact in one layer: more parallel, and shorter paths for long-range dependencies.In a real Transformer the projection matrices are learned, there are many attention heads per layer and dozens of layers, and vectors have hundreds or thousands of dimensions. The principle is the same: every token looks at every other in a single step. That is why Transformers train so efficiently on GPUs, and why distant words can influence each other directly.
12 The 2017 → 2026 Roadmap
The whole series in one line. Press play:
Next: AI Research in 2016: The Year Deep Learning Became a General-Purpose Engine →