AI Research in 2011: The Year Before the Breakthrough
2011 was the eve of the deep-learning era. The pieces that made AlexNet possible a year later were falling into place. ReLUs and AdaGrad made training easier. GPU-trained CNNs beat humans on a traffic-sign benchmark, and lock-free parallel SGD kept many cores busy. Deep networks cut speech-recognition errors by a third, and language models learned word vectors and wrote text character by character. Meanwhile the year's most famous AI systems, IBM's Watson and Microsoft's Kinect, were still built mostly without deep learning. This guide covers thirteen papers and systems in eight themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| NIPS 2011 (Granada) | Hogwild! — NeurIPS 2020 Test of Time Award · Algorithms for Hyper-Parameter Optimization (TPE) |
| ICML 2011 (Bellevue) | Socher et al. — Distinguished Application Paper Award · Generating Text with RNNs |
| CVPR 2011 | Kinect body-part recognition — Best Paper |
| IJCNN 2011 | IDSIA's CNN committee — winner of the traffic-sign recognition competition, above human accuracy |
| AISTATS, Interspeech 2011 | Rectifier networks · single-layer feature analysis · CD-DNN-HMM speech |
| JMLR 2011 | AdaGrad · NLP (Almost) from Scratch · scikit-learn |
| Events | IBM Watson on Jeopardy! (February 2011) |
02 Paper Map
Thirteen papers and systems in eight themes. Filter and click to jump:
03 🧱 Better Building Blocks for Training
Simpler activations and smarter step sizes make deep and sparse models easier to train.
Deep Sparse Rectifier Neural Networks (ReLU)
Glorot, Bordes & Bengio (Université de Montréal) · AISTATS 2011Neurons used smooth S-shaped switches (sigmoid, tanh) that flatten out and stall learning. The rectifier is almost embarrassingly simple: output the input if it's positive, otherwise zero. Deep networks built from it trained well — even without the elaborate unsupervised pretraining people thought was necessary.
- Rectifier units, max(0, x), give sparse representations (many exact zeros) and don't saturate for positive inputs
- Deep rectifier networks matched or beat tanh networks and reached their best results on supervised tasks with large labelled sets without unsupervised pretraining
ReLU became the default activation — AlexNet used it a year later, and variants (GELU, SwiGLU) still power Transformers.
AdaGrad: Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
Duchi, Hazan & Singer (UC Berkeley, Technion, Google) · JMLR 2011Plain gradient descent uses one step size for everything. AdaGrad gives every parameter its own step size, shrinking it for parameters that get updated a lot and keeping it large for rare ones — so rare but important features still get learned.
- Adapts each coordinate's learning rate using the history of its squared gradients, with regret guarantees
- Especially effective for sparse, high-dimensional data such as text and ads
The ancestor of RMSprop and Adam (2015) — the optimisers that train almost every network today.
04 ⚡ GPUs and Parallelism Make Training Fast
GPU-trained CNNs win a benchmark against humans; lock-free parallel SGD keeps many cores busy.
A Committee of Neural Networks for Traffic Sign Classification (IJCNN 2011 competition)
Cireşan, Meier, Masci & Schmidhuber (IDSIA) · IJCNN 2011The German Traffic Sign Recognition Benchmark pitted algorithms against each other — and against people. IDSIA's GPU-trained convolutional networks, combined into a committee, won the competition and beat human accuracy.
- Deep CNNs trained on GPUs, combined into a committee with differently pre-processed inputs
- Won the IJCNN 2011 traffic-sign competition; the committee's 99.46% accuracy in the final results exceeded the 98.84% human performance on the benchmark
- One of the first times a machine beat people on a visual recognition benchmark
A year before AlexNet, a clear sign that GPU-trained CNNs were ready — developed into 2012's multi-column networks.
HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, Recht, Ré & Wright (University of Wisconsin–Madison) · NIPS 2011To use many processor cores, the usual approach was to lock the shared model so only one core writes at a time — slow. Hogwild! just lets every core write whenever it wants, occasionally overwriting each other. When each update touches only a few parameters, collisions are rare and it works — much faster.
- Lock-free parallel SGD with provable near-optimal convergence when updates are sparse
- About an order of magnitude faster than schemes that use locking on sparse problems
- NeurIPS 2020 Test of Time Award
Asynchronous, lock-free updates became part of large-scale training, from DistBelief-style systems to today's frameworks.
05 🎙️ Deep Networks Crack Conversational Speech
A deep network replaces Gaussian mixtures on the hardest speech benchmark — and cuts errors by a third.
Conversational Speech Transcription Using Context-Dependent Deep Neural Networks
Seide, Li & Yu (Microsoft Research) · Interspeech 2011Transcribing casual phone calls is one of speech recognition's hardest problems. Microsoft replaced the Gaussian-mixture part of a standard recogniser with a deep network with up to nine layers, predicting thousands of fine-grained sound states. Errors dropped by a third.
- Context-dependent DNN-HMM: deep network (up to 9 hidden layers, pretrained) predicting over 9,000 tied triphone states, trained on over 300 hours
- Word error rate on the RT03S Fisher test set fell from 27.4% (discriminatively trained GMM-HMM) to 18.5% — a 33% relative reduction
The result that convinced industry — summarised with Toronto, Google and IBM in 2012's joint speech paper.
06 📝 NLP (Almost) from Scratch
Learned word vectors and tree-structured networks instead of hand-built linguistic features.
Natural Language Processing (Almost) from Scratch
Collobert, Weston, Bottou, Karlen, Kavukcuoglu & Kuksa (NEC Labs) · JMLR 2011NLP systems relied on piles of hand-crafted features for each task. This work used one neural architecture for four tasks — part-of-speech tagging, chunking, named entities and semantic roles — with word vectors learned from unlabelled Wikipedia text.
- Shared lookup-table word embeddings, pretrained with a language-model-style ranking objective on large unlabelled text, then fine-tuned with multitask learning
- Near state-of-the-art results on all four tasks with minimal feature engineering; released as the SENNA toolkit
A direct precursor of word2vec (2013) and of the pretrain-then-fine-tune recipe.
Parsing Natural Scenes and Natural Language with Recursive Neural Networks
Socher, Lin, Ng & Manning (Stanford) · ICML 2011Sentences and scenes have structure: words group into phrases, pixels into objects. A recursive neural network repeatedly merges the two most compatible pieces into one, building a tree — the same method for language and images.
- One network scores and merges adjacent segments (words or image regions) into a tree, trained with a max-margin objective
- Competitive syntactic parser on the Penn Treebank, and better scene segmentation and annotation than alternatives
- ICML 2011 Distinguished Application Paper Award
Launched recursive neural networks for sentiment and parsing — part of the path to compositional representations.
07 🔤 Recurrent Networks Learn to Write
Character by character, a recurrent network generates surprisingly fluent text.
Generating Text with Recurrent Neural Networks
Sutskever, Martens & Hinton (University of Toronto) · ICML 2011Train a recurrent network to predict the next character of Wikipedia text, then let it write. Recurrent networks were thought too hard to train; a new optimiser (Hessian-free) and a "multiplicative" design made it work. The output was often grammatical nonsense — but remarkably word-like.
- Multiplicative RNN (the input character changes the recurrent weights) trained with Hessian-free optimisation on large text corpora
- Strong character-level language modelling; samples showed learned spelling, punctuation and some syntax
An early demonstration of generative language modelling with neural networks — a long line runs from here to GPT.
08 🧮 Unsupervised Features: Simpler Than Expected
Many features, dense extraction and simple k-means beat complex deep models on CIFAR-10.
An Analysis of Single-Layer Networks in Unsupervised Feature Learning
Coates, Lee & Ng (Stanford, University of Michigan) · AISTATS 2011Researchers were building ever more complicated unsupervised models. This paper found that a single layer of simple k-means features could beat them — if you used lots of features and extracted them densely. The setup mattered more than the algorithm.
- Compared sparse autoencoders, RBMs, k-means and Gaussian mixtures as single-layer feature learners
- Number of features, stride and receptive-field size mattered most; k-means (triangle) features reached 79.6% on CIFAR-10, state of the art at the time
A lesson that returns often: scale and careful setup can matter more than clever algorithms.
09 🏆 Classical AI's Showpieces
Two landmark 2011 systems — Watson and Kinect — built mostly without deep learning.
IBM Watson Wins Jeopardy! (DeepQA)
Ferrucci et al. (IBM Research) · televised Feb 14–16, 2011IBM's Watson played the quiz show Jeopardy! against its two greatest champions — and won. It combined hundreds of language-analysis and evidence-scoring components, running on a large computer cluster, to answer questions posed in tricky natural language.
- DeepQA: question analysis, many candidate answers from search, hundreds of evidence scorers, and learned confidence estimation
- Final two-game totals: Watson $77,147, Ken Jennings $24,000, Brad Rutter $21,600
A landmark for question answering — built mostly from classical NLP, search and machine learning rather than deep networks.
Real-Time Human Pose Recognition in Parts from Single Depth Images (Kinect)
Shotton et al. (Microsoft Research Cambridge, Xbox) · CVPR 2011The Kinect camera sees depth, not colour. To find your body in real time, Microsoft trained a random forest to label every pixel with a body part, on a huge set of synthetic depth images, then located the joints — fast enough for a game console.
- Per-pixel body-part classification with randomised decision forests on simple depth-comparison features, trained on large synthetic data
- Real-time joint estimation on consumer hardware; shipped in Xbox Kinect
- CVPR 2011 Best Paper
Showed the power of large synthetic training data — an idea central to later robotics and vision.
10 🧰 Tools and Automated Tuning
Open-source ML for everyone, and algorithms that search hyperparameters for you.
Scikit-learn: Machine Learning in Python
Pedregosa et al. · JMLR 2011A free, consistent Python toolbox for machine learning: the same fit / predict interface for dozens of methods. It put solid ML into the hands of students, scientists and engineers everywhere.
- A consistent API (fit, predict, transform) over a wide range of supervised and unsupervised algorithms, built on NumPy and SciPy
- Emphasis on ease of use, documentation and performance; BSD-licensed
Part of the open-source Python ecosystem that later made deep-learning frameworks feel familiar.
Algorithms for Hyper-Parameter Optimization (TPE)
Bergstra, Bardenet, Bengio & Kégl · NIPS 2011Deep networks have many settings to tune. Instead of guessing, model which settings tended to do well so far and sample new ones from there. The Tree-structured Parzen Estimator did this efficiently — and found better networks than people tuning by hand.
- Two sequential model-based methods — a Gaussian-process approach and the Tree-structured Parzen Estimator — for tuning many hyperparameters
- Beat manual and random search when tuning deep belief networks; became the basis of the Hyperopt library
With 2012's random search and Bayesian optimisation, the start of automated experimentation.
11 AdaGrad, Computed
Why per-parameter step sizes matter for sparse data. The script trains the same classifier with plain SGD at three step sizes and with AdaGrad. The data mixes common features that carry no signal with rare ones that decide the label:
"""AdaGrad vs plain SGD on sparse features (Duchi, Hazan & Singer, 2011).
Text and click data are sparse: a few features appear constantly, most appear
rarely - and rare features are often the informative ones. Plain SGD gives
every weight the same step size, so a rare feature gets few, small updates.
AdaGrad divides each weight's step by the square root of the sum of its past
squared gradients: frequently updated weights slow down, rare ones keep large
steps.
Toy task: logistic regression with 20 common, uninformative features and
10 rare features (each present in ~1% of examples) that decide the label
when present. One pass over 5,000 examples. Standard library only.
"""
import math, random
rng = random.Random(0)
N_COMMON, N_RARE, N = 20, 10, 5000
data = []
for _ in range(N):
x = {i: rng.gauss(0, 1) for i in range(N_COMMON)}
y = rng.random() < 0.5
if rng.random() < 0.1: # ~10% of rows carry one rare feature
j = N_COMMON + rng.randrange(N_RARE)
x[j] = 1.0
y = (j % 2 == 0) # even rare features => label 1
data.append((x, 1.0 if y else 0.0))
sig = lambda z: 1 / (1 + math.exp(-max(-30, min(30, z))))
def train(lr, adagrad):
w, g2 = {}, {}
for x, y in data:
p = sig(sum(w.get(i, 0.0) * v for i, v in x.items()))
for i, v in x.items():
g = (p - y) * v
if adagrad:
g2[i] = g2.get(i, 0.0) + g * g
w[i] = w.get(i, 0.0) - lr * g / (math.sqrt(g2[i]) + 1e-8)
else:
w[i] = w.get(i, 0.0) - lr * g
return w
def evaluate(w):
rare = [(x, y) for x, y in data if any(i >= N_COMMON for i in x)]
conf = sum(sig(sum(w.get(i, 0.0) * v for i, v in x.items())) if y else 1 - sig(sum(w.get(i, 0.0) * v for i, v in x.items())) for x, y in rare) / len(rare)
noise = sum(abs(w.get(i, 0.0)) for i in range(N_COMMON)) / N_COMMON
rw = sum(abs(w.get(j, 0.0)) for j in range(N_COMMON, N_COMMON + N_RARE)) / N_RARE
return conf, rw, noise
print(f"{'method':>18} {'p(correct) on rare rows':>24} {'|rare weight|':>14} {'|useless weight|':>17}")
for name, lr, ada in [("SGD, lr 0.01", 0.01, False), ("SGD, lr 0.1", 0.1, False), ("SGD, lr 0.5", 0.5, False), ("AdaGrad, lr 0.5", 0.5, True)]:
conf, rw, noise = evaluate(train(lr, ada))
print(f"{name:>18} {conf:>24.0%} {rw:>14.2f} {noise:>17.2f}")
print("\nA small SGD step barely learns the rare features; a large one learns them but also")
print("lets the constantly-updated useless weights swing around. AdaGrad gets both right:")
print("big steps for rare features, shrinking steps for frequent ones.")Output:
method p(correct) on rare rows |rare weight| |useless weight|
SGD, lr 0.01 57% 0.25 0.05
SGD, lr 0.1 77% 1.68 0.25
SGD, lr 0.5 83% 6.51 1.17
AdaGrad, lr 0.5 96% 3.17 0.07
A small SGD step barely learns the rare features; a large one learns them but also
lets the constantly-updated useless weights swing around. AdaGrad gets both right:
big steps for rare features, shrinking steps for frequent ones.No single SGD step size works. A small one barely learns the rare features (57% confidence on rows that contain them), and a large one learns them but lets the useless weights drift far from zero. AdaGrad gets 96% confidence while keeping the useless weights near zero. The same per-parameter idea, with running averages instead of full sums, became RMSprop and then Adam in 2015.
12 The 2011 → 2026 Roadmap
The whole series in one line. Press play:
Next: AI Research in 2001–2010: The Decade That Set the Stage for Deep Learning →