AI Research in 2013: The Year Representation Learning Became the Language of AI
2013 shifted deep learning from "learn a classifier" toward "learn the representation itself". Words became vectors, and images and language began to share one space. Recurrent networks generated speech transcripts, text and handwriting. Latent-variable models became trainable with the VAE, and a network learned to play Atari straight from pixels. Researchers also started looking inside CNNs, and discovered that learned representations can be fooled by invisible changes. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| NIPS 2013 (Lake Tahoe) | word2vec phrases paper — NeurIPS 2023 Test of Time Award · DeViSE · Atari DQN (Deep Learning Workshop) |
| Posted 2013, ICLR 2014 | VAE — the first ICLR Test of Time Award (2024) · Intriguing Properties — runner-up · Network In Network · OverFeat |
| Other venues | Representation learning review (IEEE TPAMI) · Deep speech RNNs (ICASSP) · Zeiler & Fergus (ECCV 2014) · word2vec (ICLR 2013 workshop) |
| Preprint | Generating Sequences with RNNs |
02 Paper Map
Twelve papers in eight themes. Filter and click to jump:
03 🧠 Representation Learning Becomes the Central Paradigm
Hand-designed features → layers that learn simple features, then combinations, then concepts.
Representation Learning: A Review and New Perspectives
Bengio, Courville & Vincent (Université de Montréal) · IEEE TPAMI 2013How well a learning system works depends heavily on how the data is described to it. For decades people hand-designed those descriptions (features). This review argued that deep networks can learn good descriptions themselves, layer by layer, pulling apart the underlying factors in the data.
- Surveyed unsupervised feature learning, deep networks, autoencoders and manifold learning
- Framed good representations as ones that disentangle the explanatory factors of variation, and discussed priors that help (hierarchy, sparsity, shared factors across tasks)
The conceptual base under embeddings, pretraining and foundation models.
04 📐 Words Become Geometry
Meaning emerges as directions and distances in a learned vector space.
word2vec: Efficient Estimation of Word Representations in Vector Space
Mikolov, Chen, Corrado & Dean (Google) · arXiv Jan 2013 · ICLR 2013 workshopTrain a very simple network to guess a word from its neighbours (or the neighbours from the word). Its internal numbers for each word — its vector — end up placing words with similar meanings close together, and relationships become directions.
- Two cheap architectures — CBOW (predict a word from its context) and Skip-gram (predict the context from a word)
- High-quality vectors from a 1.6-billion-word dataset in under a day; strong results on a new word-analogy test set
Word vectors became standard inputs to NLP systems — and the start of embeddings everywhere.
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, Sutskever, Chen, Corrado & Dean (Google) · NIPS 2013The follow-up made word2vec faster and better: instead of scoring every word in the vocabulary, compare the true neighbour with a few random "negative" words. It also learned vectors for phrases like "New York" — and famously, king − man + woman ≈ queen.
- Negative sampling as a cheap alternative to the full softmax; subsampling of frequent words for speed and quality
- Phrase vectors (e.g. "Air Canada") and additive compositionality of vectors
- NeurIPS 2023 Test of Time Award
The idea that meaning is geometry runs straight to sentence embeddings, CLIP, vector databases and RAG.
05 🕹️ Deep Learning Starts Acting
Game pixels → CNN → action values → reward → better play.
Playing Atari with Deep Reinforcement Learning
Mnih et al. (DeepMind) · arXiv Dec 2013 · NIPS 2013 Deep Learning WorkshopA network sees only the screen pixels and the score, and learns by trial and error which joystick move is worth most. The same network and settings learned seven different Atari games.
- Convolutional network trained with a variant of Q-learning and experience replay, directly from raw pixels
- Beat all previous approaches on six of seven games and surpassed a human expert on three (Breakout, Enduro, Pong)
The seed of the 2015 Nature DQN — see the 2015 article — and of modern deep RL.
06 🌌 Deep Generative Models Get a Powerful Latent Space
Encode → sample a latent code (reparameterised) → decode.
Auto-Encoding Variational Bayes (the VAE)
Kingma & Welling (University of Amsterdam) · arXiv Dec 2013 · ICLR 2014A VAE squeezes each image into a small cloud of possible codes, then learns to rebuild the image from a code drawn from that cloud. Because the codes are organised smoothly, you can walk between them — a 1 slowly morphs into a 7 — or draw brand-new samples.
- Stochastic variational inference with an encoder (recognition model) and decoder trained jointly
- The reparameterisation trick, z = μ + σ·ε, makes sampling differentiable so everything trains with ordinary gradient descent
- ICLR 2024 Test of Time Award (the first one awarded)
A pillar of generative modelling; latent diffusion (2022) runs its denoising inside a VAE's latent space.
07 ✍️ Recurrent Networks Learn Long Sequences
Deep LSTMs recognise speech and generate text and handwriting, one step at a time.
Speech Recognition with Deep Recurrent Neural Networks
Graves, Mohamed & Hinton (University of Toronto) · ICASSP 2013Speech unfolds over time, so the network should too. Stacking several layers of LSTM memory cells, reading the audio in both directions, gave the best phoneme recognition then recorded.
- Deep bidirectional LSTM networks trained end to end (CTC and a transducer variant)
- 17.7% phoneme error on the TIMIT test set — the best recorded result at the time
Helped make deep recurrent networks the default for speech and sequences until the Transformer.
Generating Sequences with Recurrent Neural Networks
Graves (University of Toronto) · arXiv Aug 2013Train an LSTM to predict the next character — or the next pen movement — and then let it run. It writes plausible text, and realistic handwriting in many styles. Give it a sentence, and it writes that sentence in handwriting.
- Deep LSTM predicts one data point at a time; sampling from its predictions generates new sequences
- Text generation on Wikipedia; online handwriting synthesis, including a version conditioned on a character sequence via a soft attention window
Autoregressive generation with a neural network — and an early soft attention mechanism.
08 🔗 Vision and Language Begin Sharing a Semantic Space
Images and words mapped into the same space — recognition of classes never seen in training.
DeViSE: A Deep Visual-Semantic Embedding Model
Frome et al. (Google) · NIPS 2013A normal image classifier knows only its fixed list of labels. DeViSE maps each image to a point in word2vec's word space. A photo of an animal it was never trained on can still land near the right word, because words for similar things sit close together.
- A CNN is trained to predict the word2vec vector of each image's label instead of a one-of-N class
- Matched strong classifiers on 1,000-class ImageNet while making more semantically reasonable errors, and recognised thousands of labels never seen in training
A clear ancestor of CLIP (2021) and today's vision-language embeddings.
09 🔍 CNNs Become More General, Interpretable and Efficient
Beyond classifying whole images: look inside, localise, detect and share features.
Network In Network
Lin, Chen & Yan (National University of Singapore) · arXiv Dec 2013 · ICLR 2014Replace each simple filter with a tiny network (in practice, 1×1 convolutions), and replace the big final classifier with a simple average over each feature map. Fewer parameters, less overfitting.
- "mlpconv" layers (equivalent to 1×1 convolutions) and global average pooling instead of fully connected layers
- State of the art on CIFAR-10, CIFAR-100 and SVHN at the time
1×1 convolutions and global average pooling became standard (GoogLeNet 2014, ResNet 2015).
OverFeat: Integrated Recognition, Localization and Detection using ConvNets
Sermanet et al. (NYU) · arXiv Dec 2013 · ICLR 2014One ConvNet slides efficiently across the image to classify, locate and detect objects, sharing its computation between tasks instead of using separate systems.
- Efficient multiscale sliding-window ConvNet with box regression and accumulated predictions
- Won the ILSVRC 2013 localisation task; competitive in detection and classification; released features used widely
Showed CNN features could do far more than whole-image classification — setting up R-CNN (2014).
Visualizing and Understanding Convolutional Networks
Zeiler & Fergus (NYU) · arXiv Nov 2013 · ECCV 2014What does each layer of a CNN actually respond to? A "deconvolutional" network projects features back into pixels: early layers catch edges and colours, middle layers textures and parts, deep layers whole objects.
- Deconvnet visualisations of what activates each feature, and occlusion tests to find which image regions matter
- The insights led to an improved AlexNet-style architecture; Zeiler's Clarifai entry based on this work topped ILSVRC 2013 classification
Started CNN interpretability — ancestors of saliency maps and today's mechanistic interpretability.
10 🧩 The Emerging Pattern — and a First Warning
Learn the representation, not the features. But learned representations can be brittle.
Intriguing Properties of Neural Networks
Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow & Fergus · arXiv Dec 2013 · ICLR 2014The first clear demonstration that tiny, invisible changes to an image can make an accurate network misclassify it with confidence — and that the same altered image often fools other networks too. It named these "adversarial examples".
- Found adversarial examples with an optimisation method (box-constrained L-BFGS); they transferred across models and training sets
- Also showed individual units were no more interpretable than random directions in the representation space
- ICLR 2024 Test of Time Award runner-up
Launched adversarial robustness research; FGSM followed in 2014 — see the 2014 article.
11 The Reparameterisation Trick, Computed
The VAE's key trick is to write a random sample as z = μ + σ·ε, so gradients can flow through it. The script compares that estimator with the older "score function" method on a problem whose exact answer is known, then trains with it:
"""The VAE's reparameterisation trick in a few lines (Kingma & Welling, 2013).
A VAE must backpropagate through a random sample z ~ Normal(mu, sigma^2).
Sampling isn't differentiable, so the trick rewrites it as
z = mu + sigma * eps, eps ~ Normal(0, 1)
The randomness now sits in eps, and z is a smooth function of mu and sigma.
Test problem: minimise E[(z - 3)^2]. The exact gradient with respect to mu is
2 (mu - 3). We estimate it from samples two ways - the reparameterisation
trick, and the older "score function" (REINFORCE) estimator - and compare how
noisy each is. Standard library only.
"""
import random, statistics
mu, sigma, TARGET = 1.0, 1.0, 3.0
f = lambda z: (z - TARGET) ** 2
exact = 2 * (mu - TARGET)
def reparam(n, rng): # d/dmu f(mu + sigma*eps) = f'(z)
return sum(2 * (mu + sigma * e - TARGET) for e in (rng.gauss(0, 1) for _ in range(n))) / n
def score(n, rng): # f(z) * d/dmu log N(z; mu, sigma)
tot = 0.0
for _ in range(n):
z = rng.gauss(mu, sigma); tot += f(z) * (z - mu) / sigma ** 2
return tot / n
rng = random.Random(0)
print(f"Exact gradient d/dmu E[(z-3)^2] at mu = 1, sigma = 1: {exact:.2f}")
print(f"{'samples':>8} {'reparam mean':>13} {'reparam std':>12} {'score-fn mean':>14} {'score-fn std':>13}")
for n in [1, 10, 100]:
r = [reparam(n, rng) for _ in range(2000)]; s = [score(n, rng) for _ in range(2000)]
print(f"{n:>8} {statistics.mean(r):>13.2f} {statistics.stdev(r):>12.2f} {statistics.mean(s):>14.2f} {statistics.stdev(s):>13.2f}")
print("\nGradient descent on mu with ONE reparameterised sample per step:")
m, rng = 1.0, random.Random(1)
for step in range(1, 61):
z = m + sigma * rng.gauss(0, 1); m -= 0.05 * 2 * (z - TARGET)
if step in (1, 10, 30, 60): print(f" step {step:>2}: mu = {m:.2f}")
print("\nBoth estimators are unbiased, but the reparameterised one is far less noisy,")
print("so a VAE can train its encoder with ordinary stochastic gradient descent.")Output:
Exact gradient d/dmu E[(z-3)^2] at mu = 1, sigma = 1: -4.00
samples reparam mean reparam std score-fn mean score-fn std
1 -3.99 2.02 -3.91 8.70
10 -3.98 0.63 -3.98 2.95
100 -3.99 0.21 -3.99 0.90
Gradient descent on mu with ONE reparameterised sample per step:
step 1: mu = 1.07
step 10: mu = 2.45
step 30: mu = 2.71
step 60: mu = 3.04
Both estimators are unbiased, but the reparameterised one is far less noisy,
so a VAE can train its encoder with ordinary stochastic gradient descent.Both methods get the right gradient on average, but the reparameterised one is about four times less noisy. Even a single sample per step is enough to drive μ to the target. In a real VAE, μ and σ come from the encoder network for each image, and the same trick trains the encoder and decoder together.
12 The 2013 → 2026 Roadmap
The whole series in one line. Press play:
Next: AI Research in 2012: The Year Deep Learning Broke Through →