Home › Blog › AI Research in 2014
AI Research Progress Through the Years · 2014 🎭

AI Research in 2014: The Year Deep Learning Learned to Generate, Translate and Remember

In 2014 deep learning expanded from recognising patterns to generating data, translating between sequences, focusing on what matters and using an external memory. GANs learned through competition. Sequence-to-sequence models and GRUs translated whole sentences, and attention broke the fixed-vector bottleneck. Neural Turing Machines learned simple algorithms. Vision networks got much deeper and took on detection and faces. Adversarial examples exposed a new weakness, and dropout and GloVe matured training and word representations. Four of these papers have since won ten-year test-of-time awards, and a fifth was a runner-up. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the thirteenth article in AI Research Progress Through the Years. It follows 2013, and the story continues in 2015, 2016, 2017 and on to 2026. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.
▶ Four 2014 ideas modern AI still runs on Press build · click to jump

01 Venues and Awards

🕰️ Test of Time award (2024–25)📄 Accepted paper📚 Journal📝 Preprint
VenueCovered here
NIPS 2014 (Montréal)GANs and Seq2Seq — joint NeurIPS 2024 Test of Time Award (the first time it was shared)
CVPR 2014R-CNN — CVPR 2024 Longuet-Higgins Prize · DeepFace
EMNLP 2014 (Doha)GloVe — ACL 2024 Test of Time Award · RNN Encoder–Decoder / GRU
Posted 2014, published at ICLR 2015Bahdanau attention — ICLR 2025 Test of Time runner-up · VGG · FGSM
OtherDropout (JMLR) · GoogLeNet (CVPR 2015) · Neural Turing Machines (preprint)

02 Paper Map

Twelve papers in eight themes. Filter and click to jump:

▶ 2014 paper map Click to jump

03 🎭 Generative Adversarial Networks: AI Learns Through Competition

Generator vs discriminator: one network creates the challenge that drives the other to improve.

🎭

Generative Adversarial Nets

Goodfellow et al. (Université de Montréal) · NIPS 2014
🕰️ Test of Time award (2024–25)
🧒 In plain English

Two networks play a game. A forger (generator) turns random noise into fake images; a detective (discriminator) tries to tell fakes from real photos. Each improves by beating the other, until the fakes are hard to spot.

▶ Forger vs detective Press train
🔍 What they found
  • A generator G and discriminator D trained simultaneously in a two-player minimax game; no Markov chains or approximate inference needed
  • Theory: at the game's optimum the generator reproduces the data distribution
  • Demonstrated on MNIST, the Toronto Face Database and CIFAR-10
  • NeurIPS 2024 Test of Time Award (shared with Seq2Seq)
💡 Why it matters

Led to DCGAN, pix2pix, CycleGAN, BigGAN and StyleGAN — and the idea of learning from a learned adversary.

04 🔁 Sequence-to-Sequence Learning

One network reads a sequence and another writes one: arbitrary input → arbitrary output.

🔁

Sequence to Sequence Learning with Neural Networks

Sutskever, Vinyals & Le (Google) · NIPS 2014
🕰️ Test of Time award (2024–25)
🧒 In plain English

One network reads a whole English sentence and sums it up as a list of numbers; a second network writes the French sentence from that summary. No grammar rules, no phrase tables — and a surprising trick: feed the English in backwards.

▶ Encoder → vector → decoder Toggle, then translate
🔍 What they found
  • Deep (4-layer) LSTM encoder and decoder trained end to end on WMT 2014 English→French
  • 34.8 BLEU with an ensemble — above a phrase-based baseline's 33.3 — and 36.5 when used to rerank that system's outputs
  • Reversing the source sentence created short-range dependencies and markedly improved results
  • NeurIPS 2024 Test of Time Award (shared with GANs)
💡 Why it matters

Set the encoder–decoder template that attention, the Transformer and today's LLM generation build on.

🚪

Learning Phrase Representations using RNN Encoder–Decoder (GRU)

Cho et al. (Université de Montréal and others) · EMNLP 2014
📄 Accepted paper
🧒 In plain English

Simple recurrent networks forget quickly. This paper's new unit has two small gates: one decides how much of the old memory to keep, the other how much to reset. It became known as the GRU, a lighter cousin of the LSTM.

▶ Update and reset gates Drag the gates
🔍 What they found
  • Introduced an RNN encoder–decoder and a gated hidden unit (update and reset gates) — the GRU
  • Scoring phrase pairs with the learned model improved a phrase-based translation system; the representations captured linguistic regularities
💡 Why it matters

The GRU became a standard recurrent unit; the encoder–decoder idea merged with Seq2Seq and attention.

05 🔦 Attention Appears

The fixed-vector bottleneck is broken: each output word looks back at the relevant inputs.

🔦

Neural Machine Translation by Jointly Learning to Align and Translate

Bahdanau, Cho & Bengio · arXiv Sep 2014 · ICLR 2015
🕰️ Test of Time award (2024–25)
🧒 In plain English

Seq2Seq squeezes a whole sentence into one summary vector, so long sentences suffer. Attention lets the writer look back at every source word and decide, for each word it writes, which ones matter.

▶ Fixed vector vs attention Drag the sentence length
🔍 What they found
  • Soft attention over encoder states gives a fresh context vector for each output word
  • Translation quality held up on long sentences where fixed-vector models degraded; comparable to the phrase-based state of the art on English→French
  • ICLR 2025 Test of Time Award runner-up
💡 Why it matters

The direct ancestor of self-attention and the Transformer — see the full card in the 2015 article.

06 🗄️ Neural Networks Gain External Memory

A controller that learns to read from and write to a memory bank.

🗄️

Neural Turing Machines

Graves, Wayne & Danihelka (DeepMind) · arXiv Oct 2014
📝 Preprint / report
🧒 In plain English

Give a neural network a scratch pad it can read and write, with soft "heads" that decide where to look. Shown only input–output examples, it learned simple algorithms — like copying a sequence — and applied them to longer sequences than it was trained on.

▶ Learning to copy Press run
🔍 What they found
  • A neural controller with differentiable read and write heads over a memory matrix, addressed by content and by location
  • Learned copying, repeat-copying, associative recall and sorting from examples; the copy skill generalised to sequences several times longer than those seen in training
💡 Why it matters

The idea that intelligence need not live only in weights — leading to the DNC (2016), retrieval and agent memory.

07 🏛️ CNNs Become Dramatically Deeper

Depth works — but architecture and efficiency decide whether it is practical.

🏛️

VGG: Very Deep Convolutional Networks for Large-Scale Image Recognition

Simonyan & Zisserman (University of Oxford) · arXiv Sep 2014 · ICLR 2015
📄 Accepted paper
🧒 In plain English

A simple recipe: use only tiny 3×3 filters, and stack lots of them. Two small filters see as much as one big one, with fewer parameters and more non-linearity. Going to 16–19 layers made a big difference.

▶ Stacking small filters Drag the number of 3×3 layers
🔍 What they found
  • Networks of 16 and 19 weight layers built only from 3×3 convolutions and max-pooling
  • ILSVRC 2014: first in localisation and second in classification; its features transferred well to other datasets
💡 Why it matters

A simple, widely reused backbone — and proof that depth pays.

🔗 References
🌀

GoogLeNet: Going Deeper with Convolutions

Szegedy et al. (Google and others) · arXiv Sep 2014 · CVPR 2015
📄 Accepted paper
🧒 In plain English

Should a layer look through a small, medium or large window? The Inception module does all of them at once and lets training decide, using cheap 1×1 filters to keep the cost down.

▶ The Inception module Press run
🔍 What they found
  • Inception modules run 1×1, 3×3 and 5×5 convolutions and pooling in parallel, with 1×1 "bottleneck" layers to reduce cost
  • 22 layers deep with about 12× fewer parameters than the 2012 winning network (AlexNet); won ILSVRC 2014 classification with 6.67% top-5 error
💡 Why it matters

Showed that careful architecture, not just size, makes depth affordable.

08 🖼️ Deep Learning Takes Over Real Vision Tasks

Large data + deep representations + fine-tuning → practical perception systems.

📦

R-CNN: Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Girshick, Donahue, Darrell & Malik (UC Berkeley) · CVPR 2014
🕰️ Test of Time award (2024–25)
🧒 In plain English

To find objects, first propose about 2,000 candidate boxes, then run a CNN on each one to decide what — if anything — is inside. Slow, but a huge jump in accuracy over hand-crafted features.

▶ Propose, then classify Press run
🔍 What they found
  • Region proposals (Selective Search) + CNN features fine-tuned from ImageNet + per-class SVMs and box regression
  • 53.3% mAP on PASCAL VOC 2012 — more than 30% relative improvement over the previous best
  • CVPR 2024 Longuet-Higgins Prize (test of time)
💡 Why it matters

Started the R-CNN line: Fast R-CNN (2015), Faster R-CNN (2015) and Mask R-CNN (2017).

🙂

DeepFace: Closing the Gap to Human-Level Performance in Face Verification

Taigman, Yang, Ranzato & Wolf (Facebook AI Research, Tel Aviv University) · CVPR 2014
📄 Accepted paper
🧒 In plain English

Is this the same person in both photos? DeepFace first straightens each face using a 3D model, then a deep network turns it into a compact description. Photos of the same person land close together.

▶ Same person or not? Press compare
🔍 What they found
  • 3D face alignment + a nine-layer deep network trained on about 4.4 million faces of 4,030 people
  • 97.35% accuracy on the Labeled Faces in the Wild benchmark, close to the 97.53% reported for humans
💡 Why it matters

Showed deep networks reaching near-human accuracy on a real-world task — along with the privacy debates face recognition raised.

🔗 References

09 🐼 Neural Networks Reveal a New Security Problem

High accuracy does not mean robust behaviour under adversarial conditions.

🐼

Explaining and Harnessing Adversarial Examples (FGSM)

Goodfellow, Shlens & Szegedy (Google) · arXiv Dec 2014 · ICLR 2015
📄 Accepted paper
🧒 In plain English

Add a layer of noise so faint you can't see it, and a network that was 57.7% sure it saw a panda becomes 99.3% sure it's a gibbon. The paper explained why: in very high dimensions, many tiny nudges add up.

▶ The panda attack Drag epsilon
🔍 What they found
  • Fast Gradient Sign Method: perturb every input value by ε in the direction that increases the loss
  • Argued adversarial examples come mainly from models behaving too linearly in high dimensions, not from overfitting or nonlinearity
  • Adversarial training with FGSM examples improved robustness and acted as a regulariser
💡 Why it matters

Started the robustness line that runs to jailbreaks, prompt injection and evaluator gaming.

10 🧩 Regularisation and Representation Learning Mature

Stop units co-adapting; make word representations reusable objects.

🎲

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Srivastava, Hinton, Krizhevsky, Sutskever & Salakhutdinov (University of Toronto) · JMLR 2014
📚 Journal
🧒 In plain English

During training, randomly switch off half the neurons each time. No neuron can rely on a particular partner, so each learns something useful on its own — like a team where anyone might be absent.

▶ Random thinning Press train
🔍 What they found
  • Randomly drop units during training; at test time use all units with scaled weights, approximating an average over many thinned networks
  • Reduced overfitting across vision, speech recognition, document classification and computational biology benchmarks
💡 Why it matters

A standard regulariser for a decade — still used inside many Transformers.

🔗 References
🧩

GloVe: Global Vectors for Word Representation

Pennington, Socher & Manning (Stanford) · EMNLP 2014
🕰️ Test of Time award (2024–25)
🧒 In plain English

Words that appear in similar company mean similar things. GloVe counts how often words co-occur across billions of words of text, then learns vectors whose geometry reflects those counts — so directions carry meaning.

▶ Word arithmetic Pick an analogy
🔍 What they found
  • Log-bilinear regression on global word–word co-occurrence counts, combining count-based and prediction-based methods
  • 75% accuracy on a word-analogy benchmark and strong word-similarity and NER results
  • ACL 2024 Test of Time Award
💡 Why it matters

Reusable word vectors — the step before ELMo's contextual representations in 2018.

11 Why Tiny Noise Fools Networks, Computed

Goodfellow and colleagues argued that adversarial examples come from models behaving too linearly in high dimensions. The script tests that with linear classifiers of growing input size, applying the same invisible perturbation to each:

"""Why tiny perturbations fool neural networks: the linearity argument.

Goodfellow, Shlens & Szegedy (2014) argued that adversarial examples arise
because models behave roughly linearly. For a linear score w.x, the Fast
Gradient Sign Method (FGSM) adds eta = epsilon * sign(gradient) to the input.
Each input changes by at most epsilon - invisible - but the score changes by
epsilon * sum(|w|), which grows with the number of input dimensions.

We build linear classifiers of increasing input size, start from an input the
model classifies confidently, and apply FGSM with a fixed tiny epsilon.
Standard library only.
"""
import math, random

sigmoid = lambda z: 1 / (1 + math.exp(-z))
EPS = 0.007                                   # the epsilon used in the paper's panda example

print(f"FGSM with epsilon = {EPS} per input value (inputs range 0..1)")
print(f"{'input dims':>11} {'score before':>13} {'p(class A)':>11} {'score after':>12} {'p(class A)':>11}")
for n in [100, 1_000, 10_000, 150_528]:        # 150,528 = 224 x 224 x 3, an ImageNet-sized image
    rng = random.Random(n)
    w = [rng.gauss(0, 0.05) for _ in range(n)]
    x = [rng.random() for _ in range(n)]
    b = 3.0 - sum(wi * xi for wi, xi in zip(w, x))           # shift so the clean score is +3 (95% confident)
    score = sum(wi * xi for wi, xi in zip(w, x)) + b
    x_adv = [xi - EPS * (1 if wi > 0 else -1) for wi, xi in zip(w, x)]   # step against the gradient
    adv = sum(wi * xi for wi, xi in zip(w, x_adv)) + b
    print(f"{n:>11,} {score:>13.2f} {sigmoid(score):>11.1%} {adv:>12.2f} {sigmoid(adv):>11.1%}")
print(f"\nThe score drops by epsilon * sum|w|, so the same invisible change matters more as")
print("inputs get bigger: a flip for image-sized inputs, almost nothing for small ones.")

Output:

FGSM with epsilon = 0.007 per input value (inputs range 0..1)
 input dims  score before  p(class A)  score after  p(class A)
        100          3.00       95.3%         2.97       95.1%
      1,000          3.00       95.3%         2.71       93.8%
     10,000          3.00       95.3%         0.18       54.5%
    150,528          3.00       95.3%       -39.11        0.0%

The score drops by epsilon * sum|w|, so the same invisible change matters more as
inputs get bigger: a flip for image-sized inputs, almost nothing for small ones.

With a hundred inputs, the attack barely moves the score. With an image-sized input it flips a confident prediction completely, even though no single pixel changed by more than 0.007. Deep networks are not linear, but the paper showed they behave linearly enough for the same effect, which is why FGSM works on them. Try it in the FGSM card.

12 The 2014 → 2026 Roadmap

The whole series in one line. Press play:

▶ Thirteen years, thirteen stages Click a stage or press play
AI research in 2014, in one paragraph
2014 was the year deep learning expanded from recognising patterns to learning generative distributions, translating arbitrary sequences, dynamically selecting information and using external memory. GANs, Seq2Seq, GRUs, attention and Neural Turing Machines supplied those ingredients. VGG, GoogLeNet, R-CNN and DeepFace made deep vision practical. FGSM revealed that high accuracy is not robustness, and dropout and GloVe matured training and representation. Generation, sequence modelling, attention and memory later converged in Transformers, foundation models, retrieval and agents.

Next: AI Research in 2013: The Year Representation Learning Became the Language of AI →

⚠️ Notes

About the source infographic and brief: (1) the GAN panel's "Examples of generated images (DCGAN, 2014)" are wrong twice over — DCGAN is Radford et al., posted in November 2015, and the photorealistic faces shown resemble much later models (StyleGAN-era); the 2014 GAN paper showed small, blurry MNIST, face and CIFAR-10 samples; (2) the R-CNN panel's "+30% mAP" reads as 30 points; the paper reported a more than 30% relative improvement, reaching 53.3% mAP on PASCAL VOC 2012; R-CNN was also first posted in November 2013; (3) the attention panel says it "achieved state-of-the-art translation results"; it was comparable to the phrase-based state of the art, with its main gain on long sentences; (4) in "Other notable 2014 contributions", word2vec is 2013 work, and the best-known 2014 memory paper, Memory Networks (Weston, Chopra & Bordes), is missing; (5) awards missing from the infographic are added: GANs and Seq2Seq (joint NeurIPS 2024 Test of Time Award), R-CNN (CVPR 2024 Longuet-Higgins Prize), GloVe (ACL 2024 Test of Time Award) and Bahdanau attention (ICLR 2025 Test of Time runner-up). Other figures checked out against the papers (DeepFace 97.35% on LFW, VGG 16–19 layers, GoogLeNet 22 layers, panda 57.7% → gibbon 99.3%). Widget numbers not taken from a paper are marked illustrative.

🔗 All References