Home › Blog › AI Research in 2011
AI Research Progress Through the Years · 2011 🌅

AI Research in 2011: The Year Before the Breakthrough

2011 was the eve of the deep-learning era. The pieces that made AlexNet possible a year later were falling into place. ReLUs and AdaGrad made training easier. GPU-trained CNNs beat humans on a traffic-sign benchmark, and lock-free parallel SGD kept many cores busy. Deep networks cut speech-recognition errors by a third, and language models learned word vectors and wrote text character by character. Meanwhile the year's most famous AI systems, IBM's Watson and Microsoft's Kinect, were still built mostly without deep learning. This guide covers thirteen papers and systems in eight themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the sixteenth article in AI Research Progress Through the Years. It follows 2001–2010, and the story continues in 2012, 2013, 2014 and on to 2026. Unlike the other articles, this one had no source infographic, so the paper selection is ours: the 2011 work that most directly set up the 2012 breakthrough, plus the two classical systems that defined AI in the public eye that year. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.
▶ Two paths in 2011 Toggle

01 Venues and Awards

🕰️ NeurIPS 2020 Test of Time Award🏆 Best paper / award / competition win📄 Accepted paper📚 Journal📝 System / event
VenueCovered here
NIPS 2011 (Granada)Hogwild! — NeurIPS 2020 Test of Time Award · Algorithms for Hyper-Parameter Optimization (TPE)
ICML 2011 (Bellevue)Socher et al. — Distinguished Application Paper Award · Generating Text with RNNs
CVPR 2011Kinect body-part recognition — Best Paper
IJCNN 2011IDSIA's CNN committee — winner of the traffic-sign recognition competition, above human accuracy
AISTATS, Interspeech 2011Rectifier networks · single-layer feature analysis · CD-DNN-HMM speech
JMLR 2011AdaGrad · NLP (Almost) from Scratch · scikit-learn
EventsIBM Watson on Jeopardy! (February 2011)

02 Paper Map

Thirteen papers and systems in eight themes. Filter and click to jump:

▶ 2011 paper map Click to jump

03 🧱 Better Building Blocks for Training

Simpler activations and smarter step sizes make deep and sparse models easier to train.

📐

Deep Sparse Rectifier Neural Networks (ReLU)

Glorot, Bordes & Bengio (Université de Montréal) · AISTATS 2011
📄 Accepted paper
🧒 In plain English

Neurons used smooth S-shaped switches (sigmoid, tanh) that flatten out and stall learning. The rectifier is almost embarrassingly simple: output the input if it's positive, otherwise zero. Deep networks built from it trained well — even without the elaborate unsupervised pretraining people thought was necessary.

▶ Sigmoid vs rectifier Drag the input
🔍 What they found
  • Rectifier units, max(0, x), give sparse representations (many exact zeros) and don't saturate for positive inputs
  • Deep rectifier networks matched or beat tanh networks and reached their best results on supervised tasks with large labelled sets without unsupervised pretraining
💡 Why it matters

ReLU became the default activation — AlexNet used it a year later, and variants (GELU, SwiGLU) still power Transformers.

🎚️

AdaGrad: Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Duchi, Hazan & Singer (UC Berkeley, Technion, Google) · JMLR 2011
📚 Journal
🧒 In plain English

Plain gradient descent uses one step size for everything. AdaGrad gives every parameter its own step size, shrinking it for parameters that get updated a lot and keeping it large for rare ones — so rare but important features still get learned.

▶ Per-parameter step sizes Press train
🔍 What they found
  • Adapts each coordinate's learning rate using the history of its squared gradients, with regret guarantees
  • Especially effective for sparse, high-dimensional data such as text and ads
💡 Why it matters

The ancestor of RMSprop and Adam (2015) — the optimisers that train almost every network today.

🔗 References

04 ⚡ GPUs and Parallelism Make Training Fast

GPU-trained CNNs win a benchmark against humans; lock-free parallel SGD keeps many cores busy.

🚦

A Committee of Neural Networks for Traffic Sign Classification (IJCNN 2011 competition)

Cireşan, Meier, Masci & Schmidhuber (IDSIA) · IJCNN 2011
🏆 Best paper / award / competition win
🧒 In plain English

The German Traffic Sign Recognition Benchmark pitted algorithms against each other — and against people. IDSIA's GPU-trained convolutional networks, combined into a committee, won the competition and beat human accuracy.

▶ Machine vs human Press reveal
🔍 What they found
  • Deep CNNs trained on GPUs, combined into a committee with differently pre-processed inputs
  • Won the IJCNN 2011 traffic-sign competition; the committee's 99.46% accuracy in the final results exceeded the 98.84% human performance on the benchmark
  • One of the first times a machine beat people on a visual recognition benchmark
💡 Why it matters

A year before AlexNet, a clear sign that GPU-trained CNNs were ready — developed into 2012's multi-column networks.

🐗

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Niu, Recht, Ré & Wright (University of Wisconsin–Madison) · NIPS 2011
🕰️ NeurIPS 2020 Test of Time Award
🧒 In plain English

To use many processor cores, the usual approach was to lock the shared model so only one core writes at a time — slow. Hogwild! just lets every core write whenever it wants, occasionally overwriting each other. When each update touches only a few parameters, collisions are rare and it works — much faster.

▶ Locking vs Hogwild! Toggle, then run
🔍 What they found
  • Lock-free parallel SGD with provable near-optimal convergence when updates are sparse
  • About an order of magnitude faster than schemes that use locking on sparse problems
  • NeurIPS 2020 Test of Time Award
💡 Why it matters

Asynchronous, lock-free updates became part of large-scale training, from DistBelief-style systems to today's frameworks.

05 🎙️ Deep Networks Crack Conversational Speech

A deep network replaces Gaussian mixtures on the hardest speech benchmark — and cuts errors by a third.

🎙️

Conversational Speech Transcription Using Context-Dependent Deep Neural Networks

Seide, Li & Yu (Microsoft Research) · Interspeech 2011
📄 Accepted paper
🧒 In plain English

Transcribing casual phone calls is one of speech recognition's hardest problems. Microsoft replaced the Gaussian-mixture part of a standard recogniser with a deep network with up to nine layers, predicting thousands of fine-grained sound states. Errors dropped by a third.

▶ Word error rate on phone calls Press compare
🔍 What they found
  • Context-dependent DNN-HMM: deep network (up to 9 hidden layers, pretrained) predicting over 9,000 tied triphone states, trained on over 300 hours
  • Word error rate on the RT03S Fisher test set fell from 27.4% (discriminatively trained GMM-HMM) to 18.5% — a 33% relative reduction
💡 Why it matters

The result that convinced industry — summarised with Toronto, Google and IBM in 2012's joint speech paper.

06 📝 NLP (Almost) from Scratch

Learned word vectors and tree-structured networks instead of hand-built linguistic features.

📝

Natural Language Processing (Almost) from Scratch

Collobert, Weston, Bottou, Karlen, Kavukcuoglu & Kuksa (NEC Labs) · JMLR 2011
📚 Journal
🧒 In plain English

NLP systems relied on piles of hand-crafted features for each task. This work used one neural architecture for four tasks — part-of-speech tagging, chunking, named entities and semantic roles — with word vectors learned from unlabelled Wikipedia text.

▶ One network, four tasks Pick a task
🔍 What they found
  • Shared lookup-table word embeddings, pretrained with a language-model-style ranking objective on large unlabelled text, then fine-tuned with multitask learning
  • Near state-of-the-art results on all four tasks with minimal feature engineering; released as the SENNA toolkit
💡 Why it matters

A direct precursor of word2vec (2013) and of the pretrain-then-fine-tune recipe.

🌳

Parsing Natural Scenes and Natural Language with Recursive Neural Networks

Socher, Lin, Ng & Manning (Stanford) · ICML 2011
🏆 Best paper / award / competition win
🧒 In plain English

Sentences and scenes have structure: words group into phrases, pixels into objects. A recursive neural network repeatedly merges the two most compatible pieces into one, building a tree — the same method for language and images.

▶ Build the tree Press merge
🔍 What they found
  • One network scores and merges adjacent segments (words or image regions) into a tree, trained with a max-margin objective
  • Competitive syntactic parser on the Penn Treebank, and better scene segmentation and annotation than alternatives
  • ICML 2011 Distinguished Application Paper Award
💡 Why it matters

Launched recursive neural networks for sentiment and parsing — part of the path to compositional representations.

07 🔤 Recurrent Networks Learn to Write

Character by character, a recurrent network generates surprisingly fluent text.

🔤

Generating Text with Recurrent Neural Networks

Sutskever, Martens & Hinton (University of Toronto) · ICML 2011
📄 Accepted paper
🧒 In plain English

Train a recurrent network to predict the next character of Wikipedia text, then let it write. Recurrent networks were thought too hard to train; a new optimiser (Hessian-free) and a "multiplicative" design made it work. The output was often grammatical nonsense — but remarkably word-like.

▶ Next character, repeatedly Press generate
🔍 What they found
  • Multiplicative RNN (the input character changes the recurrent weights) trained with Hessian-free optimisation on large text corpora
  • Strong character-level language modelling; samples showed learned spelling, punctuation and some syntax
💡 Why it matters

An early demonstration of generative language modelling with neural networks — a long line runs from here to GPT.

08 🧮 Unsupervised Features: Simpler Than Expected

Many features, dense extraction and simple k-means beat complex deep models on CIFAR-10.

🧮

An Analysis of Single-Layer Networks in Unsupervised Feature Learning

Coates, Lee & Ng (Stanford, University of Michigan) · AISTATS 2011
📄 Accepted paper
🧒 In plain English

Researchers were building ever more complicated unsupervised models. This paper found that a single layer of simple k-means features could beat them — if you used lots of features and extracted them densely. The setup mattered more than the algorithm.

▶ What really matters Drag the number of features
🔍 What they found
  • Compared sparse autoencoders, RBMs, k-means and Gaussian mixtures as single-layer feature learners
  • Number of features, stride and receptive-field size mattered most; k-means (triangle) features reached 79.6% on CIFAR-10, state of the art at the time
💡 Why it matters

A lesson that returns often: scale and careful setup can matter more than clever algorithms.

09 🏆 Classical AI's Showpieces

Two landmark 2011 systems — Watson and Kinect — built mostly without deep learning.

🟦

IBM Watson Wins Jeopardy! (DeepQA)

Ferrucci et al. (IBM Research) · televised Feb 14–16, 2011
📝 Preprint / report
🧒 In plain English

IBM's Watson played the quiz show Jeopardy! against its two greatest champions — and won. It combined hundreds of language-analysis and evidence-scoring components, running on a large computer cluster, to answer questions posed in tricky natural language.

▶ The DeepQA pipeline Press answer
🔍 What they found
  • DeepQA: question analysis, many candidate answers from search, hundreds of evidence scorers, and learned confidence estimation
  • Final two-game totals: Watson $77,147, Ken Jennings $24,000, Brad Rutter $21,600
💡 Why it matters

A landmark for question answering — built mostly from classical NLP, search and machine learning rather than deep networks.

🕺

Real-Time Human Pose Recognition in Parts from Single Depth Images (Kinect)

Shotton et al. (Microsoft Research Cambridge, Xbox) · CVPR 2011
🏆 Best paper / award / competition win
🧒 In plain English

The Kinect camera sees depth, not colour. To find your body in real time, Microsoft trained a random forest to label every pixel with a body part, on a huge set of synthetic depth images, then located the joints — fast enough for a game console.

▶ Pixels → body parts → joints Press run
🔍 What they found
  • Per-pixel body-part classification with randomised decision forests on simple depth-comparison features, trained on large synthetic data
  • Real-time joint estimation on consumer hardware; shipped in Xbox Kinect
  • CVPR 2011 Best Paper
💡 Why it matters

Showed the power of large synthetic training data — an idea central to later robotics and vision.

10 🧰 Tools and Automated Tuning

Open-source ML for everyone, and algorithms that search hyperparameters for you.

🧰

Scikit-learn: Machine Learning in Python

Pedregosa et al. · JMLR 2011
📚 Journal
🧒 In plain English

A free, consistent Python toolbox for machine learning: the same fit / predict interface for dozens of methods. It put solid ML into the hands of students, scientists and engineers everywhere.

▶ One interface for many models Pick a model
🔍 What they found
  • A consistent API (fit, predict, transform) over a wide range of supervised and unsupervised algorithms, built on NumPy and SciPy
  • Emphasis on ease of use, documentation and performance; BSD-licensed
💡 Why it matters

Part of the open-source Python ecosystem that later made deep-learning frameworks feel familiar.

🔗 References
🔎

Algorithms for Hyper-Parameter Optimization (TPE)

Bergstra, Bardenet, Bengio & Kégl · NIPS 2011
📄 Accepted paper
🧒 In plain English

Deep networks have many settings to tune. Instead of guessing, model which settings tended to do well so far and sample new ones from there. The Tree-structured Parzen Estimator did this efficiently — and found better networks than people tuning by hand.

▶ Good vs bad trials Press run
🔍 What they found
  • Two sequential model-based methods — a Gaussian-process approach and the Tree-structured Parzen Estimator — for tuning many hyperparameters
  • Beat manual and random search when tuning deep belief networks; became the basis of the Hyperopt library
💡 Why it matters

With 2012's random search and Bayesian optimisation, the start of automated experimentation.

11 AdaGrad, Computed

Why per-parameter step sizes matter for sparse data. The script trains the same classifier with plain SGD at three step sizes and with AdaGrad. The data mixes common features that carry no signal with rare ones that decide the label:

"""AdaGrad vs plain SGD on sparse features (Duchi, Hazan & Singer, 2011).

Text and click data are sparse: a few features appear constantly, most appear
rarely - and rare features are often the informative ones. Plain SGD gives
every weight the same step size, so a rare feature gets few, small updates.
AdaGrad divides each weight's step by the square root of the sum of its past
squared gradients: frequently updated weights slow down, rare ones keep large
steps.

Toy task: logistic regression with 20 common, uninformative features and
10 rare features (each present in ~1% of examples) that decide the label
when present. One pass over 5,000 examples. Standard library only.
"""
import math, random

rng = random.Random(0)
N_COMMON, N_RARE, N = 20, 10, 5000
data = []
for _ in range(N):
    x = {i: rng.gauss(0, 1) for i in range(N_COMMON)}
    y = rng.random() < 0.5
    if rng.random() < 0.1:                                 # ~10% of rows carry one rare feature
        j = N_COMMON + rng.randrange(N_RARE)
        x[j] = 1.0
        y = (j % 2 == 0)                                     # even rare features => label 1
    data.append((x, 1.0 if y else 0.0))

sig = lambda z: 1 / (1 + math.exp(-max(-30, min(30, z))))

def train(lr, adagrad):
    w, g2 = {}, {}
    for x, y in data:
        p = sig(sum(w.get(i, 0.0) * v for i, v in x.items()))
        for i, v in x.items():
            g = (p - y) * v
            if adagrad:
                g2[i] = g2.get(i, 0.0) + g * g
                w[i] = w.get(i, 0.0) - lr * g / (math.sqrt(g2[i]) + 1e-8)
            else:
                w[i] = w.get(i, 0.0) - lr * g
    return w

def evaluate(w):
    rare = [(x, y) for x, y in data if any(i >= N_COMMON for i in x)]
    conf = sum(sig(sum(w.get(i, 0.0) * v for i, v in x.items())) if y else 1 - sig(sum(w.get(i, 0.0) * v for i, v in x.items())) for x, y in rare) / len(rare)
    noise = sum(abs(w.get(i, 0.0)) for i in range(N_COMMON)) / N_COMMON
    rw = sum(abs(w.get(j, 0.0)) for j in range(N_COMMON, N_COMMON + N_RARE)) / N_RARE
    return conf, rw, noise

print(f"{'method':>18} {'p(correct) on rare rows':>24} {'|rare weight|':>14} {'|useless weight|':>17}")
for name, lr, ada in [("SGD, lr 0.01", 0.01, False), ("SGD, lr 0.1", 0.1, False), ("SGD, lr 0.5", 0.5, False), ("AdaGrad, lr 0.5", 0.5, True)]:
    conf, rw, noise = evaluate(train(lr, ada))
    print(f"{name:>18} {conf:>24.0%} {rw:>14.2f} {noise:>17.2f}")
print("\nA small SGD step barely learns the rare features; a large one learns them but also")
print("lets the constantly-updated useless weights swing around. AdaGrad gets both right:")
print("big steps for rare features, shrinking steps for frequent ones.")

Output:

            method  p(correct) on rare rows  |rare weight|  |useless weight|
      SGD, lr 0.01                      57%           0.25              0.05
       SGD, lr 0.1                      77%           1.68              0.25
       SGD, lr 0.5                      83%           6.51              1.17
   AdaGrad, lr 0.5                      96%           3.17              0.07

A small SGD step barely learns the rare features; a large one learns them but also
lets the constantly-updated useless weights swing around. AdaGrad gets both right:
big steps for rare features, shrinking steps for frequent ones.

No single SGD step size works. A small one barely learns the rare features (57% confidence on rows that contain them), and a large one learns them but lets the useless weights drift far from zero. AdaGrad gets 96% confidence while keeping the useless weights near zero. The same per-parameter idea, with running averages instead of full sums, became RMSprop and then Adam in 2015.

12 The 2011 → 2026 Roadmap

The whole series in one line. Press play:

▶ Sixteen years, sixteen stages Click a stage or press play
AI research in 2011, in one paragraph
2011 was the year before the breakthrough. Rectifiers and AdaGrad made training easier, and Hogwild! made it parallel. GPU-trained CNNs beat humans on traffic signs, and deep networks cut conversational speech errors by a third. Collobert's work and Socher's recursive networks showed NLP could learn its own features, and a recurrent network wrote text character by character. Coates, Lee and Ng showed that scale and setup could matter more than clever algorithms. Scikit-learn and TPE improved tooling and tuning. Yet the year's headline AI, Watson and Kinect, was engineered mostly without deep learning. Within a year, AlexNet would tip the balance.

Next: AI Research in 2001–2010: The Decade That Set the Stage for Deep Learning →

⚠️ Notes

About this article: (1) no source infographic was supplied for 2011, so we chose the papers; alternatives worth reading include contractive autoencoders (Rifai et al., ICML 2011), Hessian-free training of recurrent networks (Martens & Sutskever, ICML 2011) and the founding of Google Brain in 2011; (2) the traffic-sign figures come from the competition's final results — the IDSIA committee's 99.46% (vs 98.84% human) is the figure reported with the multi-column paper, while the IJCNN 2011 workshop paper reports its own earlier committee at up to 99.15%; (3) Watson's DeepQA was described in AI Magazine in 2010, and the televised match took place in February 2011; (4) Siri's October 2011 launch is not covered because it was a product rather than a research result. Awards were checked against official or institutional sources: Hogwild! (NeurIPS 2020 Test of Time Award), Kinect pose recognition (CVPR 2011 Best Paper) and Socher et al. (ICML 2011 Distinguished Application Paper). Widget numbers not taken from a paper are marked illustrative.

🔗 All References