Home › Blog › AI Research in 2021
AI Research Progress Through the Years · 2021 🧱

AI Research in 2021: The Year the Foundation-Model Stack Emerged

In 2021 the parts of modern AI came together. Models grew larger and more efficient, and a Stanford report gave the field its name: foundation models. Cheap adaptation methods let one giant model serve many tasks. Images and words were learned in one shared space. Language models started writing working code, looking things up and browsing the web. Diffusion overtook GANs, and AlphaFold 2 solved a fifty-year-old problem in biology. This guide covers fourteen papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the sixth article in AI Research Progress Through the Years, following 2020 and continued in 2022, 2023, 2024, 2025 and 2026. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.
▶ The modern AI stack, as it stood at the end of 2021 Press build · click a layer to jump

01 Venues and Recognition

🏆 Science Breakthrough of the Year📄 Accepted paper📚 Journal📰 Nature📝 Preprint / report
VenueCovered here
ICML 2021CLIP · ALIGN · DALL·E
NeurIPS 2021Diffusion Models Beat GANs (spotlight)
EMNLP 2021Prompt Tuning
Posted 2021, published 2022FLAN and LoRA (ICLR 2022) · RETRO (ICML 2022) · Switch Transformers (JMLR)
NatureAlphaFold 2 — Science's 2021 Breakthrough of the Year (with RoseTTAFold) and Nature Methods' Method of the Year
Reports and preprintsFoundation Models (Stanford) · Codex · Gopher · WebGPT

02 Paper Map

Fourteen papers in eight themes. Filter and click to jump:

▶ 2021 paper map Click to jump

03 📈 Scaling Beyond Dense Models

Dense scaling (use every parameter) vs sparse conditional computation (use a few experts per token).

🔀

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Fedus, Zoph & Shazeer (Google) · arXiv Jan 2021 · JMLR 2022
📚 Journal
🧒 In plain English

Imagine a hospital with hundreds of specialists. A patient doesn't see all of them — a receptionist sends each one to the single right specialist. Switch Transformer does this for words: a small router sends each token to one "expert" sub-network, so the model can have a trillion parameters while each word only uses a small slice of them.

▶ Dense vs sparse Toggle, then route
🔍 What they found
  • Simplified Mixture-of-Experts routing to one expert per token (earlier MoE used two or more), cutting communication and complexity
  • Up to 7× pretraining speed-up over T5-Base at the same compute per token; models up to 1.6 trillion parameters
  • Stabilised sparse training with selective lower precision and careful initialisation
💡 Why it matters

Established sparse Mixture-of-Experts as a serious scaling path — today used in many frontier models.

🐹

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Rae et al. (DeepMind) · arXiv Dec 2021
📝 Preprint / report
🧒 In plain English

DeepMind trained a family of dense models up to 280 billion parameters and tested them on 152 tasks to ask a simple question: what does scale actually buy you? Lots of knowledge and reading comprehension — but much less improvement on logic and maths.

▶ What scale bought Toggle
🔍 What they found
  • Six models from 44M to 280B parameters, evaluated on 152 diverse tasks
  • Largest gains from scale in reading comprehension, fact-checking and identifying toxic language; smaller gains in logical and mathematical reasoning
  • Also analysed training data, toxicity and bias across scales
💡 Why it matters

A thorough map of dense scaling — and the model Chinchilla beat in 2022 by training smaller on more data.

🔗 References

04 🏛️ “Foundation Models” Become a Research Paradigm

Pretraining → foundation model → instruction tuning → general-purpose task solver.

🏛️

On the Opportunities and Risks of Foundation Models

Bommasani et al. (Stanford CRFM, 100+ authors) · arXiv Aug 2021
📝 Preprint / report
🧒 In plain English

This long Stanford report gave the field a name: a foundation model is one big model trained on broad data and then adapted to many jobs. That is a huge lever — but it also means one flaw in the foundation can show up in everything built on top of it.

▶ One model, many applications Press play
🔍 What they found
  • Coined "foundation model" and analysed capabilities, applications, technology and society
  • Named two key properties: emergence (behaviour arises rather than being designed) and homogenisation (many systems share one base, so defects propagate)
💡 Why it matters

Still the vocabulary for most AI policy and research discussion.

🍮

Finetuned Language Models Are Zero-Shot Learners (FLAN)

Wei et al. (Google) · arXiv Sep 2021 · ICLR 2022
📄 Accepted paper
🧒 In plain English

Take a pretrained model and practise it on dozens of tasks, each written as a plain instruction ("Translate this sentence to French"). It becomes much better at following instructions for tasks it has never seen.

▶ Unseen tasks: FLAN vs GPT-3 zero-shot Press evaluate
🔍 What they found
  • Instruction-tuned a 137B model on 60+ NLP datasets described by natural-language templates
  • Beat zero-shot 175B GPT-3 on 20 of 25 evaluated tasks, and even few-shot GPT-3 on several
  • Held out whole task clusters to test genuine generalisation; benefits appeared only at sufficient scale
💡 Why it matters

Instruction tuning became a standard stage of every assistant model — scaled up in 2022's Flan-PaLM.

🔗 References

05 🔧 Efficient Adaptation

Full fine-tuning (update 175B weights) vs frozen model + a tiny trainable adapter or prompt.

🪶

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al. (Microsoft) · arXiv Jun 2021 · ICLR 2022
📄 Accepted paper
🧒 In plain English

Instead of rewriting a whole encyclopaedia to add a new topic, clip in a thin supplement. LoRA freezes the giant model and trains two small matrices beside each layer. After training, the supplement can be folded back in, so the model runs just as fast.

▶ Trainable parameters for GPT-3 Drag the rank
🔍 What they found
  • Learns a low-rank update BA beside frozen weights
  • On GPT-3 175B: up to 10,000× fewer trainable parameters and 3× less GPU memory than full fine-tuning, with matching or better quality
  • No extra inference latency, because the update merges into the weights
💡 Why it matters

The default way to fine-tune open models today (with 2023's QLoRA on top).

💬

The Power of Scale for Parameter-Efficient Prompt Tuning

Lester, Al-Rfou & Constant (Google) · EMNLP 2021
📄 Accepted paper
🧒 In plain English

Keep the whole model frozen and learn only a few "soft" prompt tokens — not words, but numbers the model reads as if they were words. For small models this falls short of full fine-tuning, but as models get bigger the gap closes.

▶ Does a tiny prompt suffice? Drag model size
🔍 What they found
  • Learns continuous prompt embeddings; the model itself stays frozen
  • Gap to full model tuning shrinks with scale and closes at about 11B parameters (T5-XXL) on SuperGLUE
  • One frozen model can serve many tasks by swapping small prompts
💡 Why it matters

A direct ancestor of today's parameter-efficient fine-tuning toolkit.

06 👁️ Vision and Language Enter the Same Representation Space

Image ↔ shared meaning space ↔ language, then text prompt → generative model → image.

📎

CLIP: Learning Transferable Visual Models From Natural Language Supervision

Radford et al. (OpenAI) · ICML 2021
📄 Accepted paper
🧒 In plain English

Show a model 400 million pictures with their captions and train it to match each picture to its caption. It learns a shared "meaning space" for images and words — so it can classify a photo just by comparing it with sentences like "a photo of a dog", with no labelled training set.

▶ Zero-shot classification Pick an image
🔍 What they found
  • Contrastive training of an image encoder and a text encoder on 400M image–text pairs
  • Zero-shot ImageNet accuracy matched the original ResNet-50 without using any of its 1.28M labelled training images
  • Strong transfer across 30+ datasets from natural-language class descriptions
💡 Why it matters

The vision backbone behind DALL·E 2, Stable Diffusion and LLaVA.

🔗 References
🔗

ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Jia et al. (Google) · ICML 2021
📄 Accepted paper
🧒 In plain English

Instead of carefully cleaning data, ALIGN used 1.8 billion images with their messy web alt-text. The noise mostly cancelled out at that scale: sheer quantity made up for quality.

▶ Noisy pairs, strong model Press play
🔍 What they found
  • Dual encoder (EfficientNet + BERT) trained contrastively on 1.8B noisy image–alt-text pairs with minimal filtering
  • State of the art on Flickr30K and MSCOCO image–text retrieval at the time; strong ImageNet transfer
💡 Why it matters

Independent confirmation that web-scale contrastive learning works — even with noisy data.

🔗 References
🥑

DALL·E: Zero-Shot Text-to-Image Generation

Ramesh et al. (OpenAI) · ICML 2021
📄 Accepted paper
🧒 In plain English

Treat a picture like a sentence: chop it into a grid of 1,024 "image words", put the caption's words in front, and train a 12B Transformer to predict the next token. Ask for "an armchair in the shape of an avocado" and it writes the image token by token.

▶ Text tokens, then image tokens Press generate
🔍 What they found
  • 12B autoregressive Transformer over up to 256 text tokens + 1,024 image tokens (a 32×32 grid from a discrete VAE)
  • Trained on 250M image–text pairs; CLIP was used to rerank samples
  • Competitive zero-shot on MS-COCO with models trained on it, by human evaluation
💡 Why it matters

Showed language could drive image generation — then diffusion took over in 2022.

07 💻 Code Becomes a Language-Model Domain

Specification → model writes code → run the tests → sample alternatives → working program.

⌨️

Codex: Evaluating Large Language Models Trained on Code

Chen et al. (OpenAI) · arXiv Jul 2021
📝 Preprint / report
🧒 In plain English

Fine-tune GPT on public GitHub code, then test it with HumanEval: 164 hand-written programming problems checked by unit tests. One try often fails — but let it try many times and one attempt usually passes.

▶ More samples, more solved Drag samples per problem
🔍 What they found
  • Introduced HumanEval (164 problems, judged by functional correctness)
  • Codex-12B solved 28.8% with one sample (GPT-3: 0%); with 100 samples per problem 70.2% had at least one correct
  • A production descendant powered the first GitHub Copilot
💡 Why it matters

The start of the road from autocomplete to coding agents — and pass@k is still the standard metric.

08 🔎 External Knowledge and Tool Use

Parametric model + external retrieval + a browser environment + citations: the beginnings of the modern harness.

📚

RETRO: Improving Language Models by Retrieving from Trillions of Tokens

Borgeaud et al. (DeepMind) · arXiv Dec 2021 · ICML 2022
📄 Accepted paper
🧒 In plain English

Rather than memorising everything in its weights, RETRO keeps a 2-trillion-token library and looks things up while it writes. A 7.5B model with a library performed like models 25 times its size.

▶ Look it up while writing Press run
🔍 What they found
  • Splits text into chunks; retrieves nearest neighbours from a 2T-token database with a frozen BERT retriever; attends to them via chunked cross-attention
  • Comparable to GPT-3 and Jurassic-1 on the Pile with 25× fewer parameters
  • The database can be updated without retraining
💡 Why it matters

Made external memory a serious alternative to ever-larger weights — a pillar of RAG.

🌐

WebGPT: Browser-Assisted Question-Answering with Human Feedback

Nakano et al. (OpenAI) · arXiv Dec 2021
📝 Preprint / report
🧒 In plain English

Give GPT-3 a simple text web browser: it can search, click, scroll and quote passages, then write an answer with citations. Humans then judged the answers, and that feedback trained it further.

▶ A model that browses Press run
🔍 What they found
  • GPT-3 fine-tuned to use a text browser (search, click, quote) on ELI5 questions
  • Trained with imitation learning and a reward model from human comparisons, with best-of-n sampling
  • Best answers preferred to human demonstrators' 56% of the time, and to the top Reddit answer 69%
💡 Why it matters

An early tool-using, citation-giving agent — ancestor of research agents and search-grounded chat.

🔗 References

09 🎨 Diffusion Becomes the New Generative Paradigm

GAN dominance → diffusion catches up → diffusion surpasses GANs → the 2022 text-to-image explosion.

🌫️

Diffusion Models Beat GANs on Image Synthesis

Dhariwal & Nichol (OpenAI) · NeurIPS 2021
📄 Accepted paper
🧒 In plain English

Diffusion models learn to turn noise into pictures one small clean-up step at a time. This paper improved the architecture and added classifier guidance — a dial that nudges the image toward looking more like its class — and for the first time beat the best GANs.

▶ The guidance dial Drag guidance
🔍 What they found
  • Architecture improvements plus classifier guidance to trade diversity for fidelity
  • FID 2.97 on ImageNet 128×128, 4.59 at 256×256 and 7.72 at 512×512 — better than BigGAN-deep
  • Better distribution coverage than GANs
💡 Why it matters

The turning point that set up latent diffusion, DALL·E 2 and Imagen in 2022.

10 🧬 AI-for-Science Breakthrough

Protein sequence → AlphaFold → 3D structure + confidence → biological hypotheses and drug discovery.

🧬

Highly Accurate Protein Structure Prediction with AlphaFold

Jumper et al. (DeepMind) · Nature 2021
📰 Nature🏆 Science Breakthrough of the Year
🧒 In plain English

Proteins are chains of amino acids that fold into 3D shapes, and the shape decides what they do. Working out a shape experimentally can take months or years. AlphaFold 2 predicts it from the sequence alone — often as accurately as the experiment — and says how confident it is in each part.

▶ Sequence → structure Press fold
🔍 What they found
  • Median GDT score of 92.4 in the CASP14 assessment, competitive with experimental structures for most targets
  • Evoformer attention over sequence alignments and residue pairs, with per-residue confidence (pLDDT)
  • Applied to 98.5% of the human proteome; Science's 2021 Breakthrough of the Year (with RoseTTAFold) and Nature Methods' 2021 Method of the Year
💡 Why it matters

The clearest proof that deep learning can solve a long-standing scientific problem — 2024 Nobel Prize in Chemistry followed.

11 LoRA, Counted

LoRA's "10,000× fewer trainable parameters" is easy to check. The script counts trainable parameters for GPT-3 175B, then shows on a small matrix why the adapter adds no inference cost: it merges into the frozen weights.

"""LoRA in a few lines: how few parameters does adaptation need?

Hu et al. (2021) freeze a pretrained weight matrix W (d x d) and learn a
low-rank update B @ A, with B (d x r) and A (r x d). Only A and B train.
Part 1 counts trainable parameters for GPT-3 175B (96 layers, d = 12288),
adapting the query and value projections as in the paper.
Part 2 checks, on a tiny matrix, that the update can be merged into W,
so the adapted model runs exactly as fast as the original. Standard library only.
"""
import random

LAYERS, D, FULL = 96, 12288, 175e9
print("Part 1 - trainable parameters, GPT-3 175B")
print(f"{'method':>30} {'trainable':>12} {'fraction':>10} {'reduction':>10}")
print(f"{'full fine-tuning':>30} {FULL / 1e9:>11.0f}B {1:>10.0%} {'1x':>10}")
for r in [1, 4, 8, 64]:
    n = LAYERS * 2 * (2 * D * r)                    # Wq and Wv, each gets B (d x r) + A (r x d)
    print(f"{'LoRA r=' + str(r) + ' on Wq, Wv':>30} {n / 1e6:>11.1f}M {n / FULL:>10.4%} {FULL / n:>9,.0f}x")

print("\nPart 2 - merging the update (d = 6, r = 2)")
random.seed(0)
d, r = 6, 2
mat = lambda rows, cols, s=1.0: [[random.gauss(0, s) for _ in range(cols)] for _ in range(rows)]
mul = lambda X, Y: [[sum(X[i][k] * Y[k][j] for k in range(len(Y))) for j in range(len(Y[0]))] for i in range(len(X))]
add = lambda X, Y: [[a + b for a, b in zip(x, y)] for x, y in zip(X, Y)]
W, B, A, x = mat(d, d), mat(d, r, 0.3), mat(r, d, 0.3), mat(d, 1)
two_paths = add(mul(W, x), mul(B, mul(A, x)))       # frozen path + adapter path
merged = mul(add(W, mul(B, A)), x)                  # W' = W + BA, one matmul
err = max(abs(a[0] - b[0]) for a, b in zip(two_paths, merged))
print(f"   W has {d * d} numbers; the adapter B, A has {2 * d * r}.")
print(f"   max |(Wx + BAx) - (W + BA)x| = {err:.1e}  -> identical outputs after merging")
print("\nA 175B model can be adapted by training ~0.01% of its size, with no extra inference cost.")

Output:

Part 1 - trainable parameters, GPT-3 175B
                        method    trainable   fraction  reduction
              full fine-tuning         175B       100%         1x
            LoRA r=1 on Wq, Wv         4.7M    0.0027%    37,087x
            LoRA r=4 on Wq, Wv        18.9M    0.0108%     9,272x
            LoRA r=8 on Wq, Wv        37.7M    0.0216%     4,636x
           LoRA r=64 on Wq, Wv       302.0M    0.1726%       579x

Part 2 - merging the update (d = 6, r = 2)
   W has 36 numbers; the adapter B, A has 24.
   max |(Wx + BAx) - (W + BA)x| = 4.4e-16  -> identical outputs after merging

A 175B model can be adapted by training ~0.01% of its size, with no extra inference cost.

The 4.7M (rank 1) and 37.7M (rank 8) counts match the GPT-3 rows in the LoRA paper; the headline 10,000× sits between them. Try it in the LoRA card. The same idea, with 4-bit quantisation added, became QLoRA in 2023.

12 The 2020 → 2026 Roadmap

From scaling to self-improving systems. Press play:

▶ Nine stages in six years Click a stage or press play
2021Build the foundation-model stack
→
2022Teach it to reason, align and act
→
2023Turn it into a programmable platform
→
2024Build contexts, environments and harnesses
→
2025Scale reasoning, agency and discovery
→
2026Begin persistent self-improvement
AI research in 2021, in one paragraph
By the end of 2021 the modern AI stack had taken shape: a large pretrained model, cheap ways to adapt it, shared image–text representations, retrieval and tools, code generation and scalable generative models. Switch Transformer and Gopher explored sparse and dense scaling. The Stanford report named foundation models, and FLAN showed instruction tuning. LoRA and prompt tuning made adaptation cheap. CLIP, ALIGN and DALL·E joined vision and language. Codex made code a language-model domain. RETRO and WebGPT connected models to external knowledge. Diffusion overtook GANs, and AlphaFold 2 showed deep learning could solve a long-standing scientific problem. ChatGPT and later agents were built on top of this stack.

Next: AI Research in 2020: The Year Scaling Became the AI Strategy →

⚠️ Notes

About the source infographic and brief: (1) the Switch Transformer panel says each token activates "a few experts (e.g., 1–2)"; Switch's key simplification is routing each token to exactly one expert — top-2 was the earlier MoE design; (2) the RETRO panel describes "a 7.5B-parameter encoder"; 7.5B is the whole RETRO model — retrieval uses a frozen BERT retriever and a 2T-token database; (3) the timeline puts FLAN in mid 2021; it was posted in September 2021, so the brief's "late 2021" is closer; (4) the Gopher-style chart stops at 100B, while the Gopher family runs from 44M to 280B parameters, and its curves are illustrative; (5) recognition missing from the infographic is added: AlphaFold 2 was Science's 2021 Breakthrough of the Year (shared with RoseTTAFold) and Nature Methods' 2021 Method of the Year; venues are added for every paper. Other figures checked out against the papers (1.6T parameters and 7× speed-up, 152 tasks, FLAN's 20 of 25, LoRA's 10,000×, CLIP's 400M pairs, ALIGN's 1.8B, Codex's 28.8% and 70.2%, RETRO's 2T tokens). Widget numbers not taken from a paper are marked illustrative.

🔗 All References