AI Research in 2015: The Year Deep Learning Became Trainable, Attentive and Interactive
2015 gave deep learning many of its essential engineering primitives. BatchNorm and Adam made networks far easier to train, and DQN learned to play Atari from raw pixels. Attention let translation and captioning models focus on what mattered. Detection, segmentation and spatial alignment moved inside the network. Distillation moved knowledge from one model into another, DRAW drew images step by step, and Pointer Networks tackled discrete problems. At year's end, ResNet opened the very-deep era. Three of these papers won Test of Time awards in 2025. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| ICLR 2015 | Adam — ICLR 2025 Test of Time Award · Bahdanau attention — Test of Time runner-up |
| ICML 2015 (Lille) | Batch Normalization — ICML 2025 Test of Time Award · Show, Attend and Tell · DRAW |
| NIPS 2015 (Montréal) | Faster R-CNN — NeurIPS 2025 Test of Time Award · Spatial Transformer Networks · Pointer Networks |
| MICCAI 2015 | U-Net |
| Nature | DQN |
| Preprints | Distillation (NIPS 2014 workshop, arXiv 2015) · ResNet (arXiv Dec 2015 → CVPR 2016) |
Test of Time awards go to the paper from ten years earlier with the most lasting impact. In 2025, three such awards and one runner-up went to papers covered here.
02 Paper Map
Twelve papers in eight themes. Filter and click to jump:
03 🏋️ Training Deep Networks Becomes Much Easier
Hard-to-train deep network → batch normalisation + better optimisation → deeper, faster, more reliable training.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe & Szegedy (Google) · ICML 2015As a network trains, the numbers flowing between layers keep drifting in size, so each layer is chasing a moving target. Batch normalisation rescales them at every layer to a steady range, batch by batch. Training becomes faster and much less fussy about settings.
- Normalise each unit over the mini-batch to zero mean and unit variance, then apply a learned scale and shift
- Allowed much higher learning rates and less careful initialisation; an Inception network reached the same accuracy in 14× fewer training steps
- A batch-normalised ensemble reached 4.9% top-5 validation error on ImageNet
- ICML 2025 Test of Time Award
Made very deep networks practical; normalisation layers (BatchNorm, then LayerNorm in Transformers) became standard.
Adam: A Method for Stochastic Optimization
Kingma & Ba (University of Amsterdam, University of Toronto) · ICLR 2015Plain gradient descent takes the same-sized step for every parameter. Adam keeps a running memory of each parameter's recent gradients: it builds momentum where they agree and takes smaller steps where they're large and noisy. It works well out of the box.
- Per-parameter adaptive step sizes from running estimates of the first and second moments of the gradient, with bias correction
- Simple, memory-efficient and robust to hyperparameters; defaults β₁ = 0.9, β₂ = 0.999
- ICLR 2025 Test of Time Award
Still the default optimiser for training Transformers and LLMs (usually as AdamW).
04 🕹️ Deep Reinforcement Learning Becomes Real
Observe the screen → neural network → action → reward → store experience → learn a better policy.
DQN: Human-Level Control through Deep Reinforcement Learning
Mnih et al. (DeepMind) · Nature 2015One program, given only the screen pixels and the score, learned to play dozens of different Atari games — with the same network and settings for every game. It learned by trial and error, replaying its past experiences to learn from them again.
- Convolutional network maps raw pixels to action values (Q-values); experience replay and a separate target network stabilise learning
- Same architecture and hyperparameters across 49 games; performance comparable to a professional human games tester, above 75% of the human score on more than half the games
Launched modern deep RL — the line runs through AlphaGo, A3C, PPO and AlphaGo Zero.
05 🔦 Attention Becomes a General Neural Mechanism
One fixed vector for the whole input → look at different parts at each step.
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, Cho & Bengio (Université de Montréal) · ICLR 2015Early neural translators squeezed a whole sentence into one vector, then translated from that — long sentences fell apart. This model lets the translator look back at the source words that matter for each word it writes: attention.
- Soft attention: at each output step, a learned weighting over all encoder states forms a new context vector
- English→French translation comparable to the phrase-based state of the art, and robust on long sentences where fixed-vector models degraded
- The learned attention weights recover plausible word alignments
- ICLR 2025 Test of Time Award, runner-up
The attention mechanism the Transformer (2017) built entirely around.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu et al. (Université de Montréal, University of Toronto) · ICML 2015When describing a photo, you look at different parts as you speak. This model does the same: for each word of the caption it focuses on a region of the image — and you can see where it was looking.
- CNN image features + LSTM decoder with soft (deterministic) or hard (stochastic) attention over image regions
- State of the art on Flickr8k, Flickr30k and MS COCO captioning; attention maps make the model's focus visible
Showed attention works across modalities — an early step toward vision-language models.
06 🖼️ Computer Vision Becomes More End-to-End
Hand-designed proposals, preprocessing and alignment move inside the differentiable network.
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, He, Girshick & Sun (Microsoft Research) · NIPS 2015Detectors used to rely on a slow hand-designed step to guess where objects might be. Faster R-CNN lets the network propose the regions itself, reusing the features it already computed — so proposals become almost free.
- A Region Proposal Network shares convolutional features with the detector and predicts object bounds and scores at every position
- About 5 frames per second with VGG-16 and state-of-the-art accuracy on PASCAL VOC and MS COCO; the basis of several first-place ILSVRC and COCO 2015 entries
- NeurIPS 2025 Test of Time Award
The template for two-stage detection for years; anchors and RPNs appear across later vision systems.
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, Fischer & Brox (University of Freiburg) · MICCAI 2015To outline every cell in a microscope image, a network must know what it sees and exactly where. U-Net shrinks the image to understand it, then grows it back, passing fine details straight across through skip connections — the U shape.
- Contracting encoder + expanding decoder with skip connections that copy high-resolution features across
- Trained from very few annotated images using heavy data augmentation (elastic deformations); won the ISBI 2015 cell-tracking challenge by a large margin; segments a 512×512 image in under a second
The standard segmentation architecture — and the backbone inside the diffusion models of 2020–22.
Spatial Transformer Networks
Jaderberg, Simonyan, Zisserman & Kavukcuoglu (DeepMind) · NIPS 2015A tilted or off-centre digit is harder to recognise. A spatial transformer is a small module that learns to straighten and zoom the input before recognition — with no extra labels telling it how.
- Differentiable module: a localisation network predicts transform parameters, a grid generator and sampler warp the feature map
- Trained end-to-end with the task loss only; state of the art on distorted MNIST, Street View House Numbers and CUB-200 bird classification
An early example of putting a classic hand-crafted step inside a learned network.
07 🧑🏫 Knowledge Becomes Transferable Between Models
Teacher or ensemble → soft probability targets → smaller student.
Distilling the Knowledge in a Neural Network
Hinton, Vinyals & Dean (Google) · arXiv Mar 2015A big model knows more than its top answer: that a picture of a 2 looks a bit like a 3 and nothing like a 7. Distillation trains a small student on the teacher's full probabilities — softened with a "temperature" — so that knowledge transfers.
- Train the student on the teacher's temperature-softened output distribution, alongside the true labels
- A single distilled model captured most of the gain of an ensemble in a production speech-recognition system; introduced specialist models for large datasets
- First presented at the NIPS 2014 Deep Learning Workshop
The mechanism behind compressed LLMs, reasoning distillation and training small models on frontier-model outputs.
08 ✏️ Generative Models Learn to Draw Iteratively
Blank canvas → attend → write a little → update → repeat.
DRAW: A Recurrent Neural Network for Image Generation
Gregor, Danihelka, Graves, Rezende & Wierstra (DeepMind) · ICML 2015Instead of producing a whole picture in one go, DRAW draws in steps, like a person sketching: look at a part of the canvas, add a few strokes, look somewhere else, add more.
- A variational autoencoder with recurrent encoder and decoder, plus differentiable attention windows for reading and writing
- Generated MNIST digits and Street View house numbers that were hard to tell from real data by eye
An early mix of generation, iterative computation and attention — ideas that return in diffusion and in reasoning over many steps.
09 📍 Neural Networks Tackle Discrete and Combinatorial Problems
Attention used as a pointer: output positions in the input.
Pointer Networks
Vinyals, Fortunato & Jaitly (Google, UC Berkeley) · NIPS 2015Some answers are choices among the inputs: which points form the outer boundary, which order to visit cities. Pointer Networks use attention to point at input positions, so the answer can be any length the input requires.
- Attention over encoder states used directly as the output distribution over input positions
- Learned approximate solutions to convex hull, Delaunay triangulation and small travelling-salesman problems, and generalised to more points than seen in training for some tasks
A precursor to neural combinatorial optimisation and to models that choose actions from structured options.
10 🧱 The Very-Deep-Network Era Begins
VGG's 16–19 layers → residual learning → 152 layers.
ResNet: Deep Residual Learning for Image Recognition (late-2015 preprint)
He, Zhang, Ren & Sun (Microsoft Research) · arXiv Dec 2015 · CVPR 2016Just stacking more layers made networks worse. ResNet added shortcuts so each block learns only a correction to its input, and depth jumped from VGG's 19 layers to 152.
- Residual blocks: output = F(x) + x
- Won ILSVRC 2015 classification (3.57% top-5 error) — posted in December 2015, published at CVPR 2016 (Best Paper)
The bridge to 2016 — see the 2016 article for the full card and a lab.
11 Adam and BatchNorm, Computed
Two of 2015's training tools in a few lines. Plain gradient descent struggles in a narrow valley, while Adam's per-parameter steps do not. Batch normalisation rescales a batch of activations to a steady range:
"""2015's training toolkit in a few lines: Adam and batch normalisation.
Part 1: a badly scaled valley, loss = 0.5 * (100 x^2 + y^2). Plain gradient
descent must use a small step or it blows up along x, so it crawls along y.
Adam (Kingma & Ba, 2015) divides each parameter's step by a running estimate
of its gradient size, so both directions move at a similar pace.
Part 2: batch normalisation (Ioffe & Szegedy, 2015) rescales a layer's
activations over a mini-batch to mean 0 and variance 1, then applies a
learnable scale (gamma) and shift (beta). Standard library only.
"""
import math
def grad(x, y):
return 100 * x, y
def loss(x, y):
return 0.5 * (100 * x * x + y * y)
def sgd(lr, steps=2000):
x, y = 1.0, 1.0
for t in range(1, steps + 1):
gx, gy = grad(x, y); x, y = x - lr * gx, y - lr * gy
if loss(x, y) < 1e-3: return t
if loss(x, y) > 1e6: return None
return 'not reached in 2000'
def adam(lr=0.05, b1=0.9, b2=0.999, eps=1e-8, steps=2000):
x, y, m, v = 1.0, 1.0, [0.0, 0.0], [0.0, 0.0]
for t in range(1, steps + 1):
g = grad(x, y)
m = [b1 * mi + (1 - b1) * gi for mi, gi in zip(m, g)]
v = [b2 * vi + (1 - b2) * gi * gi for vi, gi in zip(v, g)]
mh = [mi / (1 - b1 ** t) for mi in m]; vh = [vi / (1 - b2 ** t) for vi in v]
x, y = [p - lr * a / (math.sqrt(b) + eps) for p, a, b in zip((x, y), mh, vh)]
if loss(x, y) < 1e-3: return t
return steps
print("Part 1 - steps to reach loss < 0.001 in the valley 0.5*(100x^2 + y^2)")
for lr in [0.001, 0.01, 0.019, 0.021]:
r = sgd(lr)
print(f" gradient descent, lr={lr:<6} -> {'diverges' if r is None else (r if isinstance(r, str) else str(r) + ' steps')}")
print(f" Adam, lr=0.05 -> {adam()} steps")
print("\nPart 2 - batch normalisation of one unit's activations over a batch of 8")
acts = [12.0, 15.5, 9.8, 14.1, 30.2, 11.7, 13.3, 10.4]
mu = sum(acts) / len(acts); var = sum((a - mu) ** 2 for a in acts) / len(acts)
normed = [(a - mu) / math.sqrt(var + 1e-5) for a in acts]
gamma, beta = 1.0, 0.0
out = [gamma * z + beta for z in normed]
print(f" before: mean {mu:6.2f}, std {math.sqrt(var):5.2f} {[round(a, 1) for a in acts]}")
m2 = sum(out) / len(out); s2 = math.sqrt(sum((o - m2) ** 2 for o in out) / len(out))
print(f" after: mean {m2:6.2f}, std {s2:5.2f} {[round(o, 2) for o in out]}")
print("\nStable scales and per-parameter steps: two reasons deep nets became easier to train.")Output:
Part 1 - steps to reach loss < 0.001 in the valley 0.5*(100x^2 + y^2) gradient descent, lr=0.001 -> not reached in 2000 gradient descent, lr=0.01 -> 310 steps gradient descent, lr=0.019 -> 162 steps gradient descent, lr=0.021 -> diverges Adam, lr=0.05 -> 24 steps Part 2 - batch normalisation of one unit's activations over a batch of 8 before: mean 14.62, std 6.14 [12.0, 15.5, 9.8, 14.1, 30.2, 11.7, 13.3, 10.4] after: mean 0.00, std 1.00 [-0.43, 0.14, -0.79, -0.09, 2.54, -0.48, -0.22, -0.69] Stable scales and per-parameter steps: two reasons deep nets became easier to train.
Plain gradient descent can't take a step bigger than about 0.02 without blowing up along the steep direction, so it crawls along the shallow one. Adam scales each direction's step separately. Real networks have millions of such directions, which is why Adam became the default. See the paths in the Adam card.
12 The 2015 → 2026 Roadmap
The whole series in one line. Press play:
Next: AI Research in 2014: The Year Deep Learning Learned to Generate, Translate and Remember →