AI Research in 2014: The Year Deep Learning Learned to Generate, Translate and Remember
In 2014 deep learning expanded from recognising patterns to generating data, translating between sequences, focusing on what matters and using an external memory. GANs learned through competition. Sequence-to-sequence models and GRUs translated whole sentences, and attention broke the fixed-vector bottleneck. Neural Turing Machines learned simple algorithms. Vision networks got much deeper and took on detection and faces. Adversarial examples exposed a new weakness, and dropout and GloVe matured training and word representations. Four of these papers have since won ten-year test-of-time awards, and a fifth was a runner-up. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| NIPS 2014 (Montréal) | GANs and Seq2Seq — joint NeurIPS 2024 Test of Time Award (the first time it was shared) |
| CVPR 2014 | R-CNN — CVPR 2024 Longuet-Higgins Prize · DeepFace |
| EMNLP 2014 (Doha) | GloVe — ACL 2024 Test of Time Award · RNN Encoder–Decoder / GRU |
| Posted 2014, published at ICLR 2015 | Bahdanau attention — ICLR 2025 Test of Time runner-up · VGG · FGSM |
| Other | Dropout (JMLR) · GoogLeNet (CVPR 2015) · Neural Turing Machines (preprint) |
02 Paper Map
Twelve papers in eight themes. Filter and click to jump:
03 🎭 Generative Adversarial Networks: AI Learns Through Competition
Generator vs discriminator: one network creates the challenge that drives the other to improve.
Generative Adversarial Nets
Goodfellow et al. (Université de Montréal) · NIPS 2014Two networks play a game. A forger (generator) turns random noise into fake images; a detective (discriminator) tries to tell fakes from real photos. Each improves by beating the other, until the fakes are hard to spot.
- A generator G and discriminator D trained simultaneously in a two-player minimax game; no Markov chains or approximate inference needed
- Theory: at the game's optimum the generator reproduces the data distribution
- Demonstrated on MNIST, the Toronto Face Database and CIFAR-10
- NeurIPS 2024 Test of Time Award (shared with Seq2Seq)
Led to DCGAN, pix2pix, CycleGAN, BigGAN and StyleGAN — and the idea of learning from a learned adversary.
04 🔁 Sequence-to-Sequence Learning
One network reads a sequence and another writes one: arbitrary input → arbitrary output.
Sequence to Sequence Learning with Neural Networks
Sutskever, Vinyals & Le (Google) · NIPS 2014One network reads a whole English sentence and sums it up as a list of numbers; a second network writes the French sentence from that summary. No grammar rules, no phrase tables — and a surprising trick: feed the English in backwards.
- Deep (4-layer) LSTM encoder and decoder trained end to end on WMT 2014 English→French
- 34.8 BLEU with an ensemble — above a phrase-based baseline's 33.3 — and 36.5 when used to rerank that system's outputs
- Reversing the source sentence created short-range dependencies and markedly improved results
- NeurIPS 2024 Test of Time Award (shared with GANs)
Set the encoder–decoder template that attention, the Transformer and today's LLM generation build on.
Learning Phrase Representations using RNN Encoder–Decoder (GRU)
Cho et al. (Université de Montréal and others) · EMNLP 2014Simple recurrent networks forget quickly. This paper's new unit has two small gates: one decides how much of the old memory to keep, the other how much to reset. It became known as the GRU, a lighter cousin of the LSTM.
- Introduced an RNN encoder–decoder and a gated hidden unit (update and reset gates) — the GRU
- Scoring phrase pairs with the learned model improved a phrase-based translation system; the representations captured linguistic regularities
The GRU became a standard recurrent unit; the encoder–decoder idea merged with Seq2Seq and attention.
05 🔦 Attention Appears
The fixed-vector bottleneck is broken: each output word looks back at the relevant inputs.
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, Cho & Bengio · arXiv Sep 2014 · ICLR 2015Seq2Seq squeezes a whole sentence into one summary vector, so long sentences suffer. Attention lets the writer look back at every source word and decide, for each word it writes, which ones matter.
- Soft attention over encoder states gives a fresh context vector for each output word
- Translation quality held up on long sentences where fixed-vector models degraded; comparable to the phrase-based state of the art on English→French
- ICLR 2025 Test of Time Award runner-up
The direct ancestor of self-attention and the Transformer — see the full card in the 2015 article.
06 🗄️ Neural Networks Gain External Memory
A controller that learns to read from and write to a memory bank.
Neural Turing Machines
Graves, Wayne & Danihelka (DeepMind) · arXiv Oct 2014Give a neural network a scratch pad it can read and write, with soft "heads" that decide where to look. Shown only input–output examples, it learned simple algorithms — like copying a sequence — and applied them to longer sequences than it was trained on.
- A neural controller with differentiable read and write heads over a memory matrix, addressed by content and by location
- Learned copying, repeat-copying, associative recall and sorting from examples; the copy skill generalised to sequences several times longer than those seen in training
The idea that intelligence need not live only in weights — leading to the DNC (2016), retrieval and agent memory.
07 🏛️ CNNs Become Dramatically Deeper
Depth works — but architecture and efficiency decide whether it is practical.
VGG: Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan & Zisserman (University of Oxford) · arXiv Sep 2014 · ICLR 2015A simple recipe: use only tiny 3×3 filters, and stack lots of them. Two small filters see as much as one big one, with fewer parameters and more non-linearity. Going to 16–19 layers made a big difference.
- Networks of 16 and 19 weight layers built only from 3×3 convolutions and max-pooling
- ILSVRC 2014: first in localisation and second in classification; its features transferred well to other datasets
A simple, widely reused backbone — and proof that depth pays.
GoogLeNet: Going Deeper with Convolutions
Szegedy et al. (Google and others) · arXiv Sep 2014 · CVPR 2015Should a layer look through a small, medium or large window? The Inception module does all of them at once and lets training decide, using cheap 1×1 filters to keep the cost down.
- Inception modules run 1×1, 3×3 and 5×5 convolutions and pooling in parallel, with 1×1 "bottleneck" layers to reduce cost
- 22 layers deep with about 12× fewer parameters than the 2012 winning network (AlexNet); won ILSVRC 2014 classification with 6.67% top-5 error
Showed that careful architecture, not just size, makes depth affordable.
08 🖼️ Deep Learning Takes Over Real Vision Tasks
Large data + deep representations + fine-tuning → practical perception systems.
R-CNN: Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, Donahue, Darrell & Malik (UC Berkeley) · CVPR 2014To find objects, first propose about 2,000 candidate boxes, then run a CNN on each one to decide what — if anything — is inside. Slow, but a huge jump in accuracy over hand-crafted features.
- Region proposals (Selective Search) + CNN features fine-tuned from ImageNet + per-class SVMs and box regression
- 53.3% mAP on PASCAL VOC 2012 — more than 30% relative improvement over the previous best
- CVPR 2024 Longuet-Higgins Prize (test of time)
Started the R-CNN line: Fast R-CNN (2015), Faster R-CNN (2015) and Mask R-CNN (2017).
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Yang, Ranzato & Wolf (Facebook AI Research, Tel Aviv University) · CVPR 2014Is this the same person in both photos? DeepFace first straightens each face using a 3D model, then a deep network turns it into a compact description. Photos of the same person land close together.
- 3D face alignment + a nine-layer deep network trained on about 4.4 million faces of 4,030 people
- 97.35% accuracy on the Labeled Faces in the Wild benchmark, close to the 97.53% reported for humans
Showed deep networks reaching near-human accuracy on a real-world task — along with the privacy debates face recognition raised.
09 🐼 Neural Networks Reveal a New Security Problem
High accuracy does not mean robust behaviour under adversarial conditions.
Explaining and Harnessing Adversarial Examples (FGSM)
Goodfellow, Shlens & Szegedy (Google) · arXiv Dec 2014 · ICLR 2015Add a layer of noise so faint you can't see it, and a network that was 57.7% sure it saw a panda becomes 99.3% sure it's a gibbon. The paper explained why: in very high dimensions, many tiny nudges add up.
- Fast Gradient Sign Method: perturb every input value by ε in the direction that increases the loss
- Argued adversarial examples come mainly from models behaving too linearly in high dimensions, not from overfitting or nonlinearity
- Adversarial training with FGSM examples improved robustness and acted as a regulariser
Started the robustness line that runs to jailbreaks, prompt injection and evaluator gaming.
10 🧩 Regularisation and Representation Learning Mature
Stop units co-adapting; make word representations reusable objects.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Srivastava, Hinton, Krizhevsky, Sutskever & Salakhutdinov (University of Toronto) · JMLR 2014During training, randomly switch off half the neurons each time. No neuron can rely on a particular partner, so each learns something useful on its own — like a team where anyone might be absent.
- Randomly drop units during training; at test time use all units with scaled weights, approximating an average over many thinned networks
- Reduced overfitting across vision, speech recognition, document classification and computational biology benchmarks
A standard regulariser for a decade — still used inside many Transformers.
GloVe: Global Vectors for Word Representation
Pennington, Socher & Manning (Stanford) · EMNLP 2014Words that appear in similar company mean similar things. GloVe counts how often words co-occur across billions of words of text, then learns vectors whose geometry reflects those counts — so directions carry meaning.
- Log-bilinear regression on global word–word co-occurrence counts, combining count-based and prediction-based methods
- 75% accuracy on a word-analogy benchmark and strong word-similarity and NER results
- ACL 2024 Test of Time Award
Reusable word vectors — the step before ELMo's contextual representations in 2018.
11 Why Tiny Noise Fools Networks, Computed
Goodfellow and colleagues argued that adversarial examples come from models behaving too linearly in high dimensions. The script tests that with linear classifiers of growing input size, applying the same invisible perturbation to each:
"""Why tiny perturbations fool neural networks: the linearity argument.
Goodfellow, Shlens & Szegedy (2014) argued that adversarial examples arise
because models behave roughly linearly. For a linear score w.x, the Fast
Gradient Sign Method (FGSM) adds eta = epsilon * sign(gradient) to the input.
Each input changes by at most epsilon - invisible - but the score changes by
epsilon * sum(|w|), which grows with the number of input dimensions.
We build linear classifiers of increasing input size, start from an input the
model classifies confidently, and apply FGSM with a fixed tiny epsilon.
Standard library only.
"""
import math, random
sigmoid = lambda z: 1 / (1 + math.exp(-z))
EPS = 0.007 # the epsilon used in the paper's panda example
print(f"FGSM with epsilon = {EPS} per input value (inputs range 0..1)")
print(f"{'input dims':>11} {'score before':>13} {'p(class A)':>11} {'score after':>12} {'p(class A)':>11}")
for n in [100, 1_000, 10_000, 150_528]: # 150,528 = 224 x 224 x 3, an ImageNet-sized image
rng = random.Random(n)
w = [rng.gauss(0, 0.05) for _ in range(n)]
x = [rng.random() for _ in range(n)]
b = 3.0 - sum(wi * xi for wi, xi in zip(w, x)) # shift so the clean score is +3 (95% confident)
score = sum(wi * xi for wi, xi in zip(w, x)) + b
x_adv = [xi - EPS * (1 if wi > 0 else -1) for wi, xi in zip(w, x)] # step against the gradient
adv = sum(wi * xi for wi, xi in zip(w, x_adv)) + b
print(f"{n:>11,} {score:>13.2f} {sigmoid(score):>11.1%} {adv:>12.2f} {sigmoid(adv):>11.1%}")
print(f"\nThe score drops by epsilon * sum|w|, so the same invisible change matters more as")
print("inputs get bigger: a flip for image-sized inputs, almost nothing for small ones.")Output:
FGSM with epsilon = 0.007 per input value (inputs range 0..1)
input dims score before p(class A) score after p(class A)
100 3.00 95.3% 2.97 95.1%
1,000 3.00 95.3% 2.71 93.8%
10,000 3.00 95.3% 0.18 54.5%
150,528 3.00 95.3% -39.11 0.0%
The score drops by epsilon * sum|w|, so the same invisible change matters more as
inputs get bigger: a flip for image-sized inputs, almost nothing for small ones.With a hundred inputs, the attack barely moves the score. With an image-sized input it flips a confident prediction completely, even though no single pixel changed by more than 0.007. Deep networks are not linear, but the paper showed they behave linearly enough for the same effect, which is why FGSM works on them. Try it in the FGSM card.
12 The 2014 → 2026 Roadmap
The whole series in one line. Press play:
Next: AI Research in 2013: The Year Representation Learning Became the Language of AI →