AI Research in 2016: The Year Deep Learning Became a General-Purpose Engine
In 2016 more and more of an intelligent system could be learned end to end. ResNet made very deep networks trainable. AlphaGo beat a professional at Go. Networks learned to use an external memory and even to learn their own optimisers. Translation, speech and object detection each became one learned system, replacing hand-built pipelines. Generative models started producing raw audio and pixels directly. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| CVPR 2016 (Las Vegas) | Best Paper: ResNet. Also: YOLO |
| ICML 2016 (New York) | Best Paper: PixelRNN. Also: A3C · Deep Speech 2 |
| NIPS 2016 | Learning to Learn by Gradient Descent by Gradient Descent · InfoGAN |
| ACL 2016 | Subword units (BPE) |
| Nature | AlphaGo (January) · Differentiable Neural Computer (October) |
| Preprints | GNMT · WaveNet |
02 Paper Map
Twelve papers in seven themes, plus a synthesis. Filter by award and click to jump:
03 🧱 Very Deep Networks Become Trainable
Learn the change, not the whole transformation: x → F(x) + x.
ResNet: Deep Residual Learning for Image Recognition
He, Zhang, Ren & Sun (Microsoft Research) · CVPR 2016Oddly, just stacking more layers made networks worse — even on their training data. ResNet added a shortcut around each block, so a block only has to learn a small correction to its input. If the correction isn't needed, it can learn to do almost nothing.
- Residual blocks with identity shortcut connections: output = F(x) + x
- Solved the "degradation" problem: a 56-layer plain network had higher training and test error than a 20-layer one on CIFAR-10, while deeper ResNets kept improving
- 152-layer ResNets won ILSVRC 2015 (3.57% top-5 error, ensemble) and COCO 2015 detection and segmentation; CVPR 2016 Best Paper
Every Transformer layer wraps its attention and feed-forward blocks in residual connections — the pattern starts here.
04 ⚫ Deep RL Breaks a Famous Human Barrier
Experience → policy and value networks → search and action → feedback → a better policy.
AlphaGo: Mastering the Game of Go with Deep Neural Networks and Tree Search
Silver et al. (DeepMind) · Nature 2016Go has more possible positions than atoms in the universe, so no program can search them all. AlphaGo used one network to suggest promising moves and another to judge who is winning, and searched only where they pointed.
- Policy network trained first on ~30 million positions from expert games, then improved by self-play RL; value network trained on self-play outcomes
- Monte Carlo tree search guided by both networks
- Won 99.8% of games against other Go programs and beat European champion Fan Hui 5–0; in March 2016 it beat Lee Sedol 4–1
A landmark for deep RL; 2017's AlphaGo Zero removed the human data entirely.
A3C: Asynchronous Methods for Deep Reinforcement Learning
Mnih et al. (DeepMind) · ICML 2016Instead of one agent learning from a big memory of old experiences, run many agents in parallel, each in its own copy of the game, all updating one shared brain. Their experiences are varied enough to keep learning stable — and it runs on an ordinary multi-core CPU.
- Parallel actor-learners asynchronously update shared parameters; no experience replay needed
- Beat the previous state of the art on Atari in half the training time, using a single multi-core CPU instead of a GPU
- Also worked on continuous motor control and 3D maze navigation (Labyrinth)
Made deep RL faster and cheaper; its actor-critic ideas carried into PPO and large-scale RL.
05 🗄️ Neural Networks Gain External Memory
A neural controller that learns to read and write a separate memory.
Differentiable Neural Computer: Hybrid Computing Using a Neural Network with Dynamic External Memory
Graves et al. (DeepMind) · Nature 2016A normal network keeps everything in its weights. The DNC adds a notebook: the network learns where to write facts, and how to look them up again — so it can store a family tree or a subway map and then answer questions about it.
- A neural controller with differentiable read/write heads over an external memory matrix, including links recording the order of writes
- Learned graph tasks: shortest paths on the London Underground, inference over family trees; solved a block-puzzle task (Mini-SHRDLU) with reinforcement learning
An early ancestor of agent memory, retrieval and stateful AI systems.
06 🔁 AI Starts Learning How to Learn
The optimiser itself becomes something a network can learn.
Learning to Learn by Gradient Descent by Gradient Descent
Andrychowicz et al. (DeepMind, Oxford) · NIPS 2016Training uses an optimiser — a rule like "step downhill by this much" — that people design by hand. This paper trained a small network to be the optimiser, learning its own update rule from experience of training other networks.
- A coordinate-wise LSTM outputs parameter updates; it is itself trained by gradient descent on the optimisee's loss
- Beat tuned hand-designed optimisers (SGD, RMSprop, Adam) on the kinds of problems it was trained on, and transferred to somewhat larger networks and to neural style transfer
- Did not transfer everywhere: changing the network's activation function (sigmoid to ReLU) broke it
An early "improve the improver" idea — a thread that runs to today's self-improving systems.
07 🗣️ End-to-End Language and Speech Become Practical
Raw or lightly processed input → one learned system → output.
Google's Neural Machine Translation System (GNMT)
Wu et al. (Google) · arXiv Sep 2016Old translation systems stitched together thousands of phrase rules. GNMT is one big neural network that reads the whole sentence and writes the translation, and it went into Google Translate.
- Deep LSTM encoder–decoder (8 + 8 layers) with attention and residual connections, wordpiece units, and low-precision inference for speed
- In human side-by-side evaluations, translation errors fell by about 60% on average compared with the phrase-based production system
- Deployed for Chinese→English in September 2016
Proved neural sequence models ready for production at massive scale.
Neural Machine Translation of Rare Words with Subword Units (BPE)
Sennrich, Haddow & Birch (University of Edinburgh) · ACL 2016A model with a fixed word list chokes on rare or new words. Byte Pair Encoding splits words into common pieces: learn which letter pairs appear together most often, merge them, and repeat. Any word can then be spelled from known pieces.
- Adapted the Byte Pair Encoding compression algorithm to learn a subword vocabulary
- Open-vocabulary translation of rare words, names and compounds; +1.1 and +1.3 BLEU on WMT 2015 English→German and English→Russian
BPE-style tokenisers are used by GPT-2, GPT-3 and most LLMs today.
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei et al. (Baidu Research) · ICML 2016Traditional speech recognition chained together many hand-built parts. Deep Speech 2 is one network that goes from sound to text — and the same approach worked for two very different languages, English and Mandarin.
- End-to-end model: convolutions and recurrent layers trained with CTC on spectrograms
- Competitive with human transcribers on several benchmarks; high-performance-computing techniques gave a 7× speed-up over the previous system
Showed scale plus end-to-end learning beating hand-engineering; its emphasis on scale and fast experimentation foreshadowed the scaling era.
08 📦 Computer Vision Becomes End-to-End and Real-Time
Propose → classify → post-process becomes image → one network → detections.
YOLO: You Only Look Once — Unified, Real-Time Object Detection
Redmon, Divvala, Girshick & Farhadi (UW, Allen Institute for AI, Facebook AI) · CVPR 2016Earlier detectors looked at an image thousands of times — once per candidate box. YOLO splits the image into a grid and predicts all boxes and labels in one pass. Fast enough for live video.
- Detection as a single regression from image pixels to box coordinates and class probabilities, on an S×S grid
- 45 frames per second (Fast YOLO: 155) with 63.4% mAP on PASCAL VOC 2007; fewer background false positives, more localisation errors than region-based methods
Made real-time detection mainstream; unified one-pass designs became the norm.
09 🎵 Generative Models Move Toward Raw Data
Learn the data distribution itself — audio samples, pixels and interpretable factors.
WaveNet: A Generative Model for Raw Audio
van den Oord et al. (DeepMind) · arXiv Sep 2016Speech is 16,000 or more numbers per second. WaveNet predicts them one at a time, each from the ones before, using layers that look back further and further. The voices it produced sounded far more natural than earlier systems.
- Autoregressive model of raw waveforms using stacks of dilated causal convolutions
- In listening tests it cut the gap between the best previous text-to-speech systems and human speech by over 50% for US English and Mandarin
- Also generated music and could model many speakers
Launched neural speech synthesis; autoregressive generation of raw data foreshadows GPT.
Pixel Recurrent Neural Networks (PixelRNN)
van den Oord, Kalchbrenner & Kavukcuoglu (DeepMind) · ICML 2016Paint an image one pixel at a time, left to right and top to bottom, choosing each pixel's colour from the ones already painted. Treat each colour value like a word from a 256-word vocabulary.
- Row LSTM and Diagonal BiLSTM layers (plus the faster PixelCNN) model pixels autoregressively, with a 256-way softmax per colour value
- State-of-the-art log-likelihoods on MNIST and CIFAR-10 (3.00 bits per dimension) and new downsampled ImageNet benchmarks
- ICML 2016 Best Paper
Autoregressive generation with a discrete softmax — the same recipe later used for text and image tokens.
InfoGAN: Interpretable Representation Learning by Information Maximizing GANs
Chen, Duan, Houthooft, Schulman, Sutskever & Abbeel (UC Berkeley, OpenAI) · NIPS 2016A normal GAN's random input is a jumble. InfoGAN reserves a few input dials and rewards the network when those dials clearly show up in the image. Without any labels, the dials learn meaningful things — which digit, how slanted, how thick.
- Adds a mutual-information term between a subset of latent codes and the generated output
- Discovered digit type, rotation and stroke width on MNIST; pose and lighting on 3D faces — all unsupervised
An early step toward controllable, interpretable generation.
10 🧩 The Emerging 2016 Pattern: Everything Becomes Learnable
From designing the pipeline to designing a differentiable system and letting learning find the procedure.
In 2016 features, representations, sequence mappings, control policies, memory access, optimisation rules and raw-data generation were all shown to be learnable. The job changed from designing the pipeline to designing a differentiable system and letting learning discover the procedure. The Transformer, a year later, became the most general such system.
11 Why Residuals Work, Computed
Training sends a learning signal (the gradient) backwards through every layer. This toy script compares how much of it reaches the first layer in plain and residual stacks of random layers:
"""Why residual connections made very deep networks trainable (He et al., 2016).
Backpropagation multiplies one factor per layer. In a plain network each
factor is the layer's local slope; if it is a bit below 1, the gradient
reaching the first layer shrinks exponentially with depth ("vanishing
gradients"). A residual block computes x + F(x), so its slope is 1 + F'(x):
the identity path carries the gradient through untouched.
We push a gradient back through random 1-D layers (tanh units, small random
weights) and compare plain vs residual stacks. Toy model, standard library only.
"""
import math, random
def backprop_gain(depth, residual, seed):
rng = random.Random(seed)
x, grad = 0.5, 1.0
slopes = []
for _ in range(depth): # forward pass, remembering local slopes
w = rng.gauss(0, 0.9)
pre = w * x
d = w * (1 - math.tanh(pre) ** 2) # derivative of tanh(w x)
if residual:
x, s = x + 0.1 * math.tanh(pre), 1 + 0.1 * d # x + F(x), F scaled small as in practice
else:
x, s = math.tanh(pre), d
slopes.append(s)
for s in reversed(slopes): # backward pass
grad *= s
return abs(grad)
print("Gradient size reaching layer 1 (median of 200 random networks)")
print(f"{'depth':>6} {'plain':>12} {'residual':>12}")
for depth in [5, 20, 56, 152]:
med = lambda r: sorted(backprop_gain(depth, r, s) for s in range(200))[100]
print(f"{depth:>6} {med(False):>12.2e} {med(True):>12.2e}")
print("\nPlain stacks lose the signal as depth grows; residual stacks keep it near 1,")
print("so a 152-layer ResNet can still learn in its earliest layers.")Output:
Gradient size reaching layer 1 (median of 200 random networks)
depth plain residual
5 2.80e-02 1.01e+00
20 2.69e-07 9.98e-01
56 2.02e-18 9.70e-01
152 1.30e-49 8.79e-01
Plain stacks lose the signal as depth grows; residual stacks keep it near 1,
so a 152-layer ResNet can still learn in its earliest layers.A caveat: the ResNet paper trained its plain networks with batch normalisation and argued their trouble was mainly an optimisation difficulty, not simply vanishing gradients. This toy shows one mechanism, gradients flowing through the identity path, which the follow-up paper "Identity Mappings in Deep Residual Networks" (He et al., 2016) analysed directly. The practical lesson holds either way: shortcut paths let signals pass cleanly through very deep stacks.
12 The 2016 → 2026 Roadmap
The whole series in one line. Press play:
Next: AI Research in 2015: The Year Deep Learning Became Trainable, Attentive and Interactive →