Home › Blog › AI Research in 2015
AI Research Progress Through the Years · 2015 🔧

AI Research in 2015: The Year Deep Learning Became Trainable, Attentive and Interactive

2015 gave deep learning many of its essential engineering primitives. BatchNorm and Adam made networks far easier to train, and DQN learned to play Atari from raw pixels. Attention let translation and captioning models focus on what mattered. Detection, segmentation and spatial alignment moved inside the network. Distillation moved knowledge from one model into another, DRAW drew images step by step, and Pointer Networks tackled discrete problems. At year's end, ResNet opened the very-deep era. Three of these papers won Test of Time awards in 2025. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the twelfth article in AI Research Progress Through the Years. It follows 2014, and the story continues in 2016, 2017, 2018 and on to 2026. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.
▶ Four 2015 primitives that still matter for self-improving AI Press build · click to jump

01 Venues and Awards

🕰️ Test of Time Award (2025)📄 Accepted paper📰 Nature📝 Preprint
VenueCovered here
ICLR 2015Adam — ICLR 2025 Test of Time Award · Bahdanau attention — Test of Time runner-up
ICML 2015 (Lille)Batch Normalization — ICML 2025 Test of Time Award · Show, Attend and Tell · DRAW
NIPS 2015 (Montréal)Faster R-CNN — NeurIPS 2025 Test of Time Award · Spatial Transformer Networks · Pointer Networks
MICCAI 2015U-Net
NatureDQN
PreprintsDistillation (NIPS 2014 workshop, arXiv 2015) · ResNet (arXiv Dec 2015 → CVPR 2016)

Test of Time awards go to the paper from ten years earlier with the most lasting impact. In 2025, three such awards and one runner-up went to papers covered here.

02 Paper Map

Twelve papers in eight themes. Filter and click to jump:

▶ 2015 paper map Click to jump

03 🏋️ Training Deep Networks Becomes Much Easier

Hard-to-train deep network → batch normalisation + better optimisation → deeper, faster, more reliable training.

📏

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Ioffe & Szegedy (Google) · ICML 2015
🕰️ Test of Time Award (2025)
🧒 In plain English

As a network trains, the numbers flowing between layers keep drifting in size, so each layer is chasing a moving target. Batch normalisation rescales them at every layer to a steady range, batch by batch. Training becomes faster and much less fussy about settings.

▶ Activations drifting vs normalised Toggle, then train
🔍 What they found
  • Normalise each unit over the mini-batch to zero mean and unit variance, then apply a learned scale and shift
  • Allowed much higher learning rates and less careful initialisation; an Inception network reached the same accuracy in 14× fewer training steps
  • A batch-normalised ensemble reached 4.9% top-5 validation error on ImageNet
  • ICML 2025 Test of Time Award
💡 Why it matters

Made very deep networks practical; normalisation layers (BatchNorm, then LayerNorm in Transformers) became standard.

🧭

Adam: A Method for Stochastic Optimization

Kingma & Ba (University of Amsterdam, University of Toronto) · ICLR 2015
🕰️ Test of Time Award (2025)
🧒 In plain English

Plain gradient descent takes the same-sized step for every parameter. Adam keeps a running memory of each parameter's recent gradients: it builds momentum where they agree and takes smaller steps where they're large and noisy. It works well out of the box.

▶ Descending a narrow valley Press race
🔍 What they found
  • Per-parameter adaptive step sizes from running estimates of the first and second moments of the gradient, with bias correction
  • Simple, memory-efficient and robust to hyperparameters; defaults β₁ = 0.9, β₂ = 0.999
  • ICLR 2025 Test of Time Award
💡 Why it matters

Still the default optimiser for training Transformers and LLMs (usually as AdamW).

04 🕹️ Deep Reinforcement Learning Becomes Real

Observe the screen → neural network → action → reward → store experience → learn a better policy.

🕹️

DQN: Human-Level Control through Deep Reinforcement Learning

Mnih et al. (DeepMind) · Nature 2015
📰 Nature
🧒 In plain English

One program, given only the screen pixels and the score, learned to play dozens of different Atari games — with the same network and settings for every game. It learned by trial and error, replaying its past experiences to learn from them again.

▶ The DQN loop Press play
🔍 What they found
  • Convolutional network maps raw pixels to action values (Q-values); experience replay and a separate target network stabilise learning
  • Same architecture and hyperparameters across 49 games; performance comparable to a professional human games tester, above 75% of the human score on more than half the games
💡 Why it matters

Launched modern deep RL — the line runs through AlphaGo, A3C, PPO and AlphaGo Zero.

05 🔦 Attention Becomes a General Neural Mechanism

One fixed vector for the whole input → look at different parts at each step.

🔦

Neural Machine Translation by Jointly Learning to Align and Translate

Bahdanau, Cho & Bengio (Université de Montréal) · ICLR 2015
🕰️ Test of Time Award (2025)
🧒 In plain English

Early neural translators squeezed a whole sentence into one vector, then translated from that — long sentences fell apart. This model lets the translator look back at the source words that matter for each word it writes: attention.

▶ Alignment as attention Press translate
🔍 What they found
  • Soft attention: at each output step, a learned weighting over all encoder states forms a new context vector
  • English→French translation comparable to the phrase-based state of the art, and robust on long sentences where fixed-vector models degraded
  • The learned attention weights recover plausible word alignments
  • ICLR 2025 Test of Time Award, runner-up
💡 Why it matters

The attention mechanism the Transformer (2017) built entirely around.

🐈

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Xu et al. (Université de Montréal, University of Toronto) · ICML 2015
📄 Accepted paper
🧒 In plain English

When describing a photo, you look at different parts as you speak. This model does the same: for each word of the caption it focuses on a region of the image — and you can see where it was looking.

▶ Where the model looks Press caption
🔍 What they found
  • CNN image features + LSTM decoder with soft (deterministic) or hard (stochastic) attention over image regions
  • State of the art on Flickr8k, Flickr30k and MS COCO captioning; attention maps make the model's focus visible
💡 Why it matters

Showed attention works across modalities — an early step toward vision-language models.

06 🖼️ Computer Vision Becomes More End-to-End

Hand-designed proposals, preprocessing and alignment move inside the differentiable network.

📦

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Ren, He, Girshick & Sun (Microsoft Research) · NIPS 2015
🕰️ Test of Time Award (2025)
🧒 In plain English

Detectors used to rely on a slow hand-designed step to guess where objects might be. Faster R-CNN lets the network propose the regions itself, reusing the features it already computed — so proposals become almost free.

▶ Two stages, one network Toggle
🔍 What they found
  • A Region Proposal Network shares convolutional features with the detector and predicts object bounds and scores at every position
  • About 5 frames per second with VGG-16 and state-of-the-art accuracy on PASCAL VOC and MS COCO; the basis of several first-place ILSVRC and COCO 2015 entries
  • NeurIPS 2025 Test of Time Award
💡 Why it matters

The template for two-stage detection for years; anchors and RPNs appear across later vision systems.

🔬

U-Net: Convolutional Networks for Biomedical Image Segmentation

Ronneberger, Fischer & Brox (University of Freiburg) · MICCAI 2015
📄 Accepted paper
🧒 In plain English

To outline every cell in a microscope image, a network must know what it sees and exactly where. U-Net shrinks the image to understand it, then grows it back, passing fine details straight across through skip connections — the U shape.

▶ Down, across and up Press run
🔍 What they found
  • Contracting encoder + expanding decoder with skip connections that copy high-resolution features across
  • Trained from very few annotated images using heavy data augmentation (elastic deformations); won the ISBI 2015 cell-tracking challenge by a large margin; segments a 512×512 image in under a second
💡 Why it matters

The standard segmentation architecture — and the backbone inside the diffusion models of 2020–22.

🔄

Spatial Transformer Networks

Jaderberg, Simonyan, Zisserman & Kavukcuoglu (DeepMind) · NIPS 2015
📄 Accepted paper
🧒 In plain English

A tilted or off-centre digit is harder to recognise. A spatial transformer is a small module that learns to straighten and zoom the input before recognition — with no extra labels telling it how.

▶ Learn to straighten the input Drag the distortion, then transform
🔍 What they found
  • Differentiable module: a localisation network predicts transform parameters, a grid generator and sampler warp the feature map
  • Trained end-to-end with the task loss only; state of the art on distorted MNIST, Street View House Numbers and CUB-200 bird classification
💡 Why it matters

An early example of putting a classic hand-crafted step inside a learned network.

07 🧑‍🏫 Knowledge Becomes Transferable Between Models

Teacher or ensemble → soft probability targets → smaller student.

🧑‍🏫

Distilling the Knowledge in a Neural Network

Hinton, Vinyals & Dean (Google) · arXiv Mar 2015
📝 Preprint / report
🧒 In plain English

A big model knows more than its top answer: that a picture of a 2 looks a bit like a 3 and nothing like a 7. Distillation trains a small student on the teacher's full probabilities — softened with a "temperature" — so that knowledge transfers.

▶ Hard labels vs soft targets Drag the temperature
🔍 What they found
  • Train the student on the teacher's temperature-softened output distribution, alongside the true labels
  • A single distilled model captured most of the gain of an ensemble in a production speech-recognition system; introduced specialist models for large datasets
  • First presented at the NIPS 2014 Deep Learning Workshop
💡 Why it matters

The mechanism behind compressed LLMs, reasoning distillation and training small models on frontier-model outputs.

08 ✏️ Generative Models Learn to Draw Iteratively

Blank canvas → attend → write a little → update → repeat.

✏️

DRAW: A Recurrent Neural Network for Image Generation

Gregor, Danihelka, Graves, Rezende & Wierstra (DeepMind) · ICML 2015
📄 Accepted paper
🧒 In plain English

Instead of producing a whole picture in one go, DRAW draws in steps, like a person sketching: look at a part of the canvas, add a few strokes, look somewhere else, add more.

▶ Draw in steps Press draw
🔍 What they found
  • A variational autoencoder with recurrent encoder and decoder, plus differentiable attention windows for reading and writing
  • Generated MNIST digits and Street View house numbers that were hard to tell from real data by eye
💡 Why it matters

An early mix of generation, iterative computation and attention — ideas that return in diffusion and in reasoning over many steps.

09 📍 Neural Networks Tackle Discrete and Combinatorial Problems

Attention used as a pointer: output positions in the input.

📍

Pointer Networks

Vinyals, Fortunato & Jaitly (Google, UC Berkeley) · NIPS 2015
📄 Accepted paper
🧒 In plain English

Some answers are choices among the inputs: which points form the outer boundary, which order to visit cities. Pointer Networks use attention to point at input positions, so the answer can be any length the input requires.

▶ Point to the next city Press solve
🔍 What they found
  • Attention over encoder states used directly as the output distribution over input positions
  • Learned approximate solutions to convex hull, Delaunay triangulation and small travelling-salesman problems, and generalised to more points than seen in training for some tasks
💡 Why it matters

A precursor to neural combinatorial optimisation and to models that choose actions from structured options.

10 🧱 The Very-Deep-Network Era Begins

VGG's 16–19 layers → residual learning → 152 layers.

🧱

ResNet: Deep Residual Learning for Image Recognition (late-2015 preprint)

He, Zhang, Ren & Sun (Microsoft Research) · arXiv Dec 2015 · CVPR 2016
📝 Preprint / report
🧒 In plain English

Just stacking more layers made networks worse. ResNet added shortcuts so each block learns only a correction to its input, and depth jumped from VGG's 19 layers to 152.

▶ From VGG to ResNet Press grow
🔍 What they found
  • Residual blocks: output = F(x) + x
  • Won ILSVRC 2015 classification (3.57% top-5 error) — posted in December 2015, published at CVPR 2016 (Best Paper)
💡 Why it matters

The bridge to 2016 — see the 2016 article for the full card and a lab.

11 Adam and BatchNorm, Computed

Two of 2015's training tools in a few lines. Plain gradient descent struggles in a narrow valley, while Adam's per-parameter steps do not. Batch normalisation rescales a batch of activations to a steady range:

"""2015's training toolkit in a few lines: Adam and batch normalisation.

Part 1: a badly scaled valley, loss = 0.5 * (100 x^2 + y^2). Plain gradient
descent must use a small step or it blows up along x, so it crawls along y.
Adam (Kingma & Ba, 2015) divides each parameter's step by a running estimate
of its gradient size, so both directions move at a similar pace.

Part 2: batch normalisation (Ioffe & Szegedy, 2015) rescales a layer's
activations over a mini-batch to mean 0 and variance 1, then applies a
learnable scale (gamma) and shift (beta). Standard library only.
"""
import math

def grad(x, y):
    return 100 * x, y

def loss(x, y):
    return 0.5 * (100 * x * x + y * y)

def sgd(lr, steps=2000):
    x, y = 1.0, 1.0
    for t in range(1, steps + 1):
        gx, gy = grad(x, y); x, y = x - lr * gx, y - lr * gy
        if loss(x, y) < 1e-3: return t
        if loss(x, y) > 1e6: return None
    return 'not reached in 2000'

def adam(lr=0.05, b1=0.9, b2=0.999, eps=1e-8, steps=2000):
    x, y, m, v = 1.0, 1.0, [0.0, 0.0], [0.0, 0.0]
    for t in range(1, steps + 1):
        g = grad(x, y)
        m = [b1 * mi + (1 - b1) * gi for mi, gi in zip(m, g)]
        v = [b2 * vi + (1 - b2) * gi * gi for vi, gi in zip(v, g)]
        mh = [mi / (1 - b1 ** t) for mi in m]; vh = [vi / (1 - b2 ** t) for vi in v]
        x, y = [p - lr * a / (math.sqrt(b) + eps) for p, a, b in zip((x, y), mh, vh)]
        if loss(x, y) < 1e-3: return t
    return steps

print("Part 1 - steps to reach loss < 0.001 in the valley 0.5*(100x^2 + y^2)")
for lr in [0.001, 0.01, 0.019, 0.021]:
    r = sgd(lr)
    print(f"   gradient descent, lr={lr:<6} -> {'diverges' if r is None else (r if isinstance(r, str) else str(r) + ' steps')}")
print(f"   Adam, lr=0.05             -> {adam()} steps")

print("\nPart 2 - batch normalisation of one unit's activations over a batch of 8")
acts = [12.0, 15.5, 9.8, 14.1, 30.2, 11.7, 13.3, 10.4]
mu = sum(acts) / len(acts); var = sum((a - mu) ** 2 for a in acts) / len(acts)
normed = [(a - mu) / math.sqrt(var + 1e-5) for a in acts]
gamma, beta = 1.0, 0.0
out = [gamma * z + beta for z in normed]
print(f"   before: mean {mu:6.2f}, std {math.sqrt(var):5.2f}  {[round(a, 1) for a in acts]}")
m2 = sum(out) / len(out); s2 = math.sqrt(sum((o - m2) ** 2 for o in out) / len(out))
print(f"   after:  mean {m2:6.2f}, std {s2:5.2f}  {[round(o, 2) for o in out]}")
print("\nStable scales and per-parameter steps: two reasons deep nets became easier to train.")

Output:

Part 1 - steps to reach loss < 0.001 in the valley 0.5*(100x^2 + y^2)
   gradient descent, lr=0.001  -> not reached in 2000
   gradient descent, lr=0.01   -> 310 steps
   gradient descent, lr=0.019  -> 162 steps
   gradient descent, lr=0.021  -> diverges
   Adam, lr=0.05             -> 24 steps

Part 2 - batch normalisation of one unit's activations over a batch of 8
   before: mean  14.62, std  6.14  [12.0, 15.5, 9.8, 14.1, 30.2, 11.7, 13.3, 10.4]
   after:  mean   0.00, std  1.00  [-0.43, 0.14, -0.79, -0.09, 2.54, -0.48, -0.22, -0.69]

Stable scales and per-parameter steps: two reasons deep nets became easier to train.

Plain gradient descent can't take a step bigger than about 0.02 without blowing up along the steep direction, so it crawls along the shallow one. Adam scales each direction's step separately. Real networks have millions of such directions, which is why Adam became the default. See the paths in the Adam card.

12 The 2015 → 2026 Roadmap

The whole series in one line. Press play:

▶ Twelve years, twelve stages Click a stage or press play
AI research in 2015, in one paragraph
2015 gave deep learning many of its essential engineering primitives. BatchNorm and Adam made training stable. DQN learned from interaction. Attention let models focus, in language and in vision. Faster R-CNN, U-Net and spatial transformers put perception steps inside the network. Distillation transferred knowledge between models. DRAW built images step by step, and Pointer Networks produced structured, discrete outputs. ResNet, arriving in December, opened the door to very deep networks. Together these turned deep networks from specialised models into increasingly general learning systems, ready for 2016's end-to-end everything.

Next: AI Research in 2014: The Year Deep Learning Learned to Generate, Translate and Remember →

⚠️ Notes

About the source infographic and brief: (1) the ResNet chart again shows the plain network's error falling steadily with depth; the paper's key finding was that deeper plain networks got worse (the degradation problem); (2) the roadmap strip labels two consecutive boxes "2019" — the first ("Pretraining changes everything") should be 2018; (3) distillation is dated 2015 by its arXiv posting, but it was first presented at the NIPS 2014 Deep Learning Workshop; similarly Adam and the Bahdanau attention paper were posted in late 2014 and published at ICLR 2015; (4) recognition missing from the infographic is added: Adam (ICLR 2025 Test of Time Award), Bahdanau et al. (runner-up), Batch Normalization (ICML 2025 Test of Time Award) and Faster R-CNN (NeurIPS 2025 Test of Time Award). Other figures checked out against the papers (14× fewer steps, 49 Atari games, 5 fps with VGG-16, 152 layers). Widget numbers not taken from a paper are marked illustrative.

🔗 All References