Home › Blog › AI Research in 2016
AI Research Progress Through the Years · 2016 ⚙️

AI Research in 2016: The Year Deep Learning Became a General-Purpose Engine

In 2016 more and more of an intelligent system could be learned end to end. ResNet made very deep networks trainable. AlphaGo beat a professional at Go. Networks learned to use an external memory and even to learn their own optimisers. Translation, speech and object detection each became one learned system, replacing hand-built pipelines. Generative models started producing raw audio and pixels directly. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the eleventh article in AI Research Progress Through the Years. It follows 2015, and the story continues in 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025 and 2026. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.

01 Venues and Awards

🏆 Best Paper📄 Accepted paper📰 Nature📝 Preprint
VenueCovered here
CVPR 2016 (Las Vegas)Best Paper: ResNet. Also: YOLO
ICML 2016 (New York)Best Paper: PixelRNN. Also: A3C · Deep Speech 2
NIPS 2016Learning to Learn by Gradient Descent by Gradient Descent · InfoGAN
ACL 2016Subword units (BPE)
NatureAlphaGo (January) · Differentiable Neural Computer (October)
PreprintsGNMT · WaveNet

02 Paper Map

Twelve papers in seven themes, plus a synthesis. Filter by award and click to jump:

▶ 2016 paper map Click to jump

03 🧱 Very Deep Networks Become Trainable

Learn the change, not the whole transformation: x → F(x) + x.

🧱

ResNet: Deep Residual Learning for Image Recognition

He, Zhang, Ren & Sun (Microsoft Research) · CVPR 2016
🏆 Best / Outstanding Paper
🧒 In plain English

Oddly, just stacking more layers made networks worse — even on their training data. ResNet added a shortcut around each block, so a block only has to learn a small correction to its input. If the correction isn't needed, it can learn to do almost nothing.

▶ Plain vs residual Toggle
🔍 What they found
  • Residual blocks with identity shortcut connections: output = F(x) + x
  • Solved the "degradation" problem: a 56-layer plain network had higher training and test error than a 20-layer one on CIFAR-10, while deeper ResNets kept improving
  • 152-layer ResNets won ILSVRC 2015 (3.57% top-5 error, ensemble) and COCO 2015 detection and segmentation; CVPR 2016 Best Paper
💡 Why it matters

Every Transformer layer wraps its attention and feed-forward blocks in residual connections — the pattern starts here.

04 ⚫ Deep RL Breaks a Famous Human Barrier

Experience → policy and value networks → search and action → feedback → a better policy.

⚫

AlphaGo: Mastering the Game of Go with Deep Neural Networks and Tree Search

Silver et al. (DeepMind) · Nature 2016
📰 Nature
🧒 In plain English

Go has more possible positions than atoms in the universe, so no program can search them all. AlphaGo used one network to suggest promising moves and another to judge who is winning, and searched only where they pointed.

▶ How AlphaGo was built Press play
🔍 What they found
  • Policy network trained first on ~30 million positions from expert games, then improved by self-play RL; value network trained on self-play outcomes
  • Monte Carlo tree search guided by both networks
  • Won 99.8% of games against other Go programs and beat European champion Fan Hui 5–0; in March 2016 it beat Lee Sedol 4–1
💡 Why it matters

A landmark for deep RL; 2017's AlphaGo Zero removed the human data entirely.

🧵

A3C: Asynchronous Methods for Deep Reinforcement Learning

Mnih et al. (DeepMind) · ICML 2016
📄 Accepted paper
🧒 In plain English

Instead of one agent learning from a big memory of old experiences, run many agents in parallel, each in its own copy of the game, all updating one shared brain. Their experiences are varied enough to keep learning stable — and it runs on an ordinary multi-core CPU.

▶ Many actors, one network Press run
🔍 What they found
  • Parallel actor-learners asynchronously update shared parameters; no experience replay needed
  • Beat the previous state of the art on Atari in half the training time, using a single multi-core CPU instead of a GPU
  • Also worked on continuous motor control and 3D maze navigation (Labyrinth)
💡 Why it matters

Made deep RL faster and cheaper; its actor-critic ideas carried into PPO and large-scale RL.

05 🗄️ Neural Networks Gain External Memory

A neural controller that learns to read and write a separate memory.

🗄️

Differentiable Neural Computer: Hybrid Computing Using a Neural Network with Dynamic External Memory

Graves et al. (DeepMind) · Nature 2016
📰 Nature
🧒 In plain English

A normal network keeps everything in its weights. The DNC adds a notebook: the network learns where to write facts, and how to look them up again — so it can store a family tree or a subway map and then answer questions about it.

▶ Read and write a memory Press run
🔍 What they found
  • A neural controller with differentiable read/write heads over an external memory matrix, including links recording the order of writes
  • Learned graph tasks: shortest paths on the London Underground, inference over family trees; solved a block-puzzle task (Mini-SHRDLU) with reinforcement learning
💡 Why it matters

An early ancestor of agent memory, retrieval and stateful AI systems.

06 🔁 AI Starts Learning How to Learn

The optimiser itself becomes something a network can learn.

🔁

Learning to Learn by Gradient Descent by Gradient Descent

Andrychowicz et al. (DeepMind, Oxford) · NIPS 2016
📄 Accepted paper
🧒 In plain English

Training uses an optimiser — a rule like "step downhill by this much" — that people design by hand. This paper trained a small network to be the optimiser, learning its own update rule from experience of training other networks.

▶ Hand-designed vs learned optimiser Press race
🔍 What they found
  • A coordinate-wise LSTM outputs parameter updates; it is itself trained by gradient descent on the optimisee's loss
  • Beat tuned hand-designed optimisers (SGD, RMSprop, Adam) on the kinds of problems it was trained on, and transferred to somewhat larger networks and to neural style transfer
  • Did not transfer everywhere: changing the network's activation function (sigmoid to ReLU) broke it
💡 Why it matters

An early "improve the improver" idea — a thread that runs to today's self-improving systems.

07 🗣️ End-to-End Language and Speech Become Practical

Raw or lightly processed input → one learned system → output.

🌐

Google's Neural Machine Translation System (GNMT)

Wu et al. (Google) · arXiv Sep 2016
📝 Preprint / report
🧒 In plain English

Old translation systems stitched together thousands of phrase rules. GNMT is one big neural network that reads the whole sentence and writes the translation, and it went into Google Translate.

▶ Phrase-based vs neural Toggle
🔍 What they found
  • Deep LSTM encoder–decoder (8 + 8 layers) with attention and residual connections, wordpiece units, and low-precision inference for speed
  • In human side-by-side evaluations, translation errors fell by about 60% on average compared with the phrase-based production system
  • Deployed for Chinese→English in September 2016
💡 Why it matters

Proved neural sequence models ready for production at massive scale.

✂️

Neural Machine Translation of Rare Words with Subword Units (BPE)

Sennrich, Haddow & Birch (University of Edinburgh) · ACL 2016
📄 Accepted paper
🧒 In plain English

A model with a fixed word list chokes on rare or new words. Byte Pair Encoding splits words into common pieces: learn which letter pairs appear together most often, merge them, and repeat. Any word can then be spelled from known pieces.

▶ Learning merges Press merge
🔍 What they found
  • Adapted the Byte Pair Encoding compression algorithm to learn a subword vocabulary
  • Open-vocabulary translation of rare words, names and compounds; +1.1 and +1.3 BLEU on WMT 2015 English→German and English→Russian
💡 Why it matters

BPE-style tokenisers are used by GPT-2, GPT-3 and most LLMs today.

🎤

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Amodei et al. (Baidu Research) · ICML 2016
📄 Accepted paper
🧒 In plain English

Traditional speech recognition chained together many hand-built parts. Deep Speech 2 is one network that goes from sound to text — and the same approach worked for two very different languages, English and Mandarin.

▶ Pipeline vs one network Toggle
🔍 What they found
  • End-to-end model: convolutions and recurrent layers trained with CTC on spectrograms
  • Competitive with human transcribers on several benchmarks; high-performance-computing techniques gave a 7× speed-up over the previous system
💡 Why it matters

Showed scale plus end-to-end learning beating hand-engineering; its emphasis on scale and fast experimentation foreshadowed the scaling era.

08 📦 Computer Vision Becomes End-to-End and Real-Time

Propose → classify → post-process becomes image → one network → detections.

📦

YOLO: You Only Look Once — Unified, Real-Time Object Detection

Redmon, Divvala, Girshick & Farhadi (UW, Allen Institute for AI, Facebook AI) · CVPR 2016
📄 Accepted paper
🧒 In plain English

Earlier detectors looked at an image thousands of times — once per candidate box. YOLO splits the image into a grid and predicts all boxes and labels in one pass. Fast enough for live video.

▶ One look at the grid Press detect
🔍 What they found
  • Detection as a single regression from image pixels to box coordinates and class probabilities, on an S×S grid
  • 45 frames per second (Fast YOLO: 155) with 63.4% mAP on PASCAL VOC 2007; fewer background false positives, more localisation errors than region-based methods
💡 Why it matters

Made real-time detection mainstream; unified one-pass designs became the norm.

09 🎵 Generative Models Move Toward Raw Data

Learn the data distribution itself — audio samples, pixels and interpretable factors.

🎵

WaveNet: A Generative Model for Raw Audio

van den Oord et al. (DeepMind) · arXiv Sep 2016
📝 Preprint / report
🧒 In plain English

Speech is 16,000 or more numbers per second. WaveNet predicts them one at a time, each from the ones before, using layers that look back further and further. The voices it produced sounded far more natural than earlier systems.

▶ Dilated causal convolutions Press play
🔍 What they found
  • Autoregressive model of raw waveforms using stacks of dilated causal convolutions
  • In listening tests it cut the gap between the best previous text-to-speech systems and human speech by over 50% for US English and Mandarin
  • Also generated music and could model many speakers
💡 Why it matters

Launched neural speech synthesis; autoregressive generation of raw data foreshadows GPT.

🖼️

Pixel Recurrent Neural Networks (PixelRNN)

van den Oord, Kalchbrenner & Kavukcuoglu (DeepMind) · ICML 2016
🏆 Best / Outstanding Paper
🧒 In plain English

Paint an image one pixel at a time, left to right and top to bottom, choosing each pixel's colour from the ones already painted. Treat each colour value like a word from a 256-word vocabulary.

▶ Pixel by pixel Press generate
🔍 What they found
  • Row LSTM and Diagonal BiLSTM layers (plus the faster PixelCNN) model pixels autoregressively, with a 256-way softmax per colour value
  • State-of-the-art log-likelihoods on MNIST and CIFAR-10 (3.00 bits per dimension) and new downsampled ImageNet benchmarks
  • ICML 2016 Best Paper
💡 Why it matters

Autoregressive generation with a discrete softmax — the same recipe later used for text and image tokens.

🎛️

InfoGAN: Interpretable Representation Learning by Information Maximizing GANs

Chen, Duan, Houthooft, Schulman, Sutskever & Abbeel (UC Berkeley, OpenAI) · NIPS 2016
📄 Accepted paper
🧒 In plain English

A normal GAN's random input is a jumble. InfoGAN reserves a few input dials and rewards the network when those dials clearly show up in the image. Without any labels, the dials learn meaningful things — which digit, how slanted, how thick.

▶ Turn the learned dials Drag a dial
🔍 What they found
  • Adds a mutual-information term between a subset of latent codes and the generated output
  • Discovered digit type, rotation and stroke width on MNIST; pose and lighting on 3D faces — all unsupervised
💡 Why it matters

An early step toward controllable, interpretable generation.

10 🧩 The Emerging 2016 Pattern: Everything Becomes Learnable

From designing the pipeline to designing a differentiable system and letting learning find the procedure.

▶ From hand-engineered pipelines to learned systems Click a domain

In 2016 features, representations, sequence mappings, control policies, memory access, optimisation rules and raw-data generation were all shown to be learnable. The job changed from designing the pipeline to designing a differentiable system and letting learning discover the procedure. The Transformer, a year later, became the most general such system.

11 Why Residuals Work, Computed

Training sends a learning signal (the gradient) backwards through every layer. This toy script compares how much of it reaches the first layer in plain and residual stacks of random layers:

"""Why residual connections made very deep networks trainable (He et al., 2016).

Backpropagation multiplies one factor per layer. In a plain network each
factor is the layer's local slope; if it is a bit below 1, the gradient
reaching the first layer shrinks exponentially with depth ("vanishing
gradients"). A residual block computes x + F(x), so its slope is 1 + F'(x):
the identity path carries the gradient through untouched.

We push a gradient back through random 1-D layers (tanh units, small random
weights) and compare plain vs residual stacks. Toy model, standard library only.
"""
import math, random

def backprop_gain(depth, residual, seed):
    rng = random.Random(seed)
    x, grad = 0.5, 1.0
    slopes = []
    for _ in range(depth):                       # forward pass, remembering local slopes
        w = rng.gauss(0, 0.9)
        pre = w * x
        d = w * (1 - math.tanh(pre) ** 2)        # derivative of tanh(w x)
        if residual:
            x, s = x + 0.1 * math.tanh(pre), 1 + 0.1 * d   # x + F(x), F scaled small as in practice
        else:
            x, s = math.tanh(pre), d
        slopes.append(s)
    for s in reversed(slopes):                   # backward pass
        grad *= s
    return abs(grad)

print("Gradient size reaching layer 1 (median of 200 random networks)")
print(f"{'depth':>6} {'plain':>12} {'residual':>12}")
for depth in [5, 20, 56, 152]:
    med = lambda r: sorted(backprop_gain(depth, r, s) for s in range(200))[100]
    print(f"{depth:>6} {med(False):>12.2e} {med(True):>12.2e}")
print("\nPlain stacks lose the signal as depth grows; residual stacks keep it near 1,")
print("so a 152-layer ResNet can still learn in its earliest layers.")

Output:

Gradient size reaching layer 1 (median of 200 random networks)
 depth        plain     residual
     5     2.80e-02     1.01e+00
    20     2.69e-07     9.98e-01
    56     2.02e-18     9.70e-01
   152     1.30e-49     8.79e-01

Plain stacks lose the signal as depth grows; residual stacks keep it near 1,
so a 152-layer ResNet can still learn in its earliest layers.

A caveat: the ResNet paper trained its plain networks with batch normalisation and argued their trouble was mainly an optimisation difficulty, not simply vanishing gradients. This toy shows one mechanism, gradients flowing through the identity path, which the follow-up paper "Identity Mappings in Deep Residual Networks" (He et al., 2016) analysed directly. The practical lesson holds either way: shortcut paths let signals pass cleanly through very deep stacks.

12 The 2016 → 2026 Roadmap

The whole series in one line. Press play:

▶ Eleven years, eleven stages Click a stage or press play
AI research in 2016, in one paragraph
2016 demonstrated that more and more of an intelligent system could be learned end to end. ResNet made depth trainable, and AlphaGo and A3C scaled deep reinforcement learning. The DNC learned to use memory, and learned optimisers learned how to learn. GNMT, subword units and Deep Speech 2 made language and speech end-to-end, and YOLO did the same for detection. WaveNet, PixelRNN and InfoGAN generated raw data and discovered interpretable factors. Those primitives are what the Transformer-and-agent era combined from 2017 onward.

Next: AI Research in 2015: The Year Deep Learning Became Trainable, Attentive and Interactive →

⚠️ Notes

About the source infographic and brief: (1) the ResNet chart shows the plain network's error falling steadily with depth; the paper's central finding was the opposite — deeper plain networks got worse (the "degradation problem"), which residual connections fixed; the chart's log-scale axis from 0.1% to 10% error is also not from the paper; (2) the learned-optimiser panel says it "generalizes across different model architectures"; it transferred to somewhat larger networks and to style transfer, but failed when the activation function changed from sigmoid to ReLU — later work (Lv et al., 2017) documented these limits; (3) the BPE example "un@@ @@believable" mixes notations; in Sennrich et al.'s format, "@@" marks a piece that continues into the next one (e.g. "un@@ believ@@ able"); (4) the roadmap's 2017 box repeats 2016 papers (GNMT/BPE, Deep Speech 2, YOLO, WaveNet/PixelRNN, InfoGAN); (5) awards and events missing from the infographic are added: ResNet (CVPR 2016 Best Paper), PixelRNN (ICML 2016 Best Paper), and AlphaGo's 4–1 win over Lee Sedol in March 2016. Other figures checked out against the papers (152 layers, Fan Hui 5–0, 45/155 FPS, 448×448 input). Widget numbers not taken from a paper are marked illustrative.

🔗 All References