AI Research in 2021: The Year the Foundation-Model Stack Emerged
In 2021 the parts of modern AI came together. Models grew larger and more efficient, and a Stanford report gave the field its name: foundation models. Cheap adaptation methods let one giant model serve many tasks. Images and words were learned in one shared space. Language models started writing working code, looking things up and browsing the web. Diffusion overtook GANs, and AlphaFold 2 solved a fifty-year-old problem in biology. This guide covers fourteen papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Recognition
| Venue | Covered here |
|---|---|
| ICML 2021 | CLIP · ALIGN · DALL·E |
| NeurIPS 2021 | Diffusion Models Beat GANs (spotlight) |
| EMNLP 2021 | Prompt Tuning |
| Posted 2021, published 2022 | FLAN and LoRA (ICLR 2022) · RETRO (ICML 2022) · Switch Transformers (JMLR) |
| Nature | AlphaFold 2 — Science's 2021 Breakthrough of the Year (with RoseTTAFold) and Nature Methods' Method of the Year |
| Reports and preprints | Foundation Models (Stanford) · Codex · Gopher · WebGPT |
02 Paper Map
Fourteen papers in eight themes. Filter and click to jump:
03 📈 Scaling Beyond Dense Models
Dense scaling (use every parameter) vs sparse conditional computation (use a few experts per token).
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, Zoph & Shazeer (Google) · arXiv Jan 2021 · JMLR 2022Imagine a hospital with hundreds of specialists. A patient doesn't see all of them — a receptionist sends each one to the single right specialist. Switch Transformer does this for words: a small router sends each token to one "expert" sub-network, so the model can have a trillion parameters while each word only uses a small slice of them.
- Simplified Mixture-of-Experts routing to one expert per token (earlier MoE used two or more), cutting communication and complexity
- Up to 7× pretraining speed-up over T5-Base at the same compute per token; models up to 1.6 trillion parameters
- Stabilised sparse training with selective lower precision and careful initialisation
Established sparse Mixture-of-Experts as a serious scaling path — today used in many frontier models.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae et al. (DeepMind) · arXiv Dec 2021DeepMind trained a family of dense models up to 280 billion parameters and tested them on 152 tasks to ask a simple question: what does scale actually buy you? Lots of knowledge and reading comprehension — but much less improvement on logic and maths.
- Six models from 44M to 280B parameters, evaluated on 152 diverse tasks
- Largest gains from scale in reading comprehension, fact-checking and identifying toxic language; smaller gains in logical and mathematical reasoning
- Also analysed training data, toxicity and bias across scales
A thorough map of dense scaling — and the model Chinchilla beat in 2022 by training smaller on more data.
04 🏛️ “Foundation Models” Become a Research Paradigm
Pretraining → foundation model → instruction tuning → general-purpose task solver.
On the Opportunities and Risks of Foundation Models
Bommasani et al. (Stanford CRFM, 100+ authors) · arXiv Aug 2021This long Stanford report gave the field a name: a foundation model is one big model trained on broad data and then adapted to many jobs. That is a huge lever — but it also means one flaw in the foundation can show up in everything built on top of it.
- Coined "foundation model" and analysed capabilities, applications, technology and society
- Named two key properties: emergence (behaviour arises rather than being designed) and homogenisation (many systems share one base, so defects propagate)
Still the vocabulary for most AI policy and research discussion.
Finetuned Language Models Are Zero-Shot Learners (FLAN)
Wei et al. (Google) · arXiv Sep 2021 · ICLR 2022Take a pretrained model and practise it on dozens of tasks, each written as a plain instruction ("Translate this sentence to French"). It becomes much better at following instructions for tasks it has never seen.
- Instruction-tuned a 137B model on 60+ NLP datasets described by natural-language templates
- Beat zero-shot 175B GPT-3 on 20 of 25 evaluated tasks, and even few-shot GPT-3 on several
- Held out whole task clusters to test genuine generalisation; benefits appeared only at sufficient scale
Instruction tuning became a standard stage of every assistant model — scaled up in 2022's Flan-PaLM.
05 🔧 Efficient Adaptation
Full fine-tuning (update 175B weights) vs frozen model + a tiny trainable adapter or prompt.
LoRA: Low-Rank Adaptation of Large Language Models
Hu et al. (Microsoft) · arXiv Jun 2021 · ICLR 2022Instead of rewriting a whole encyclopaedia to add a new topic, clip in a thin supplement. LoRA freezes the giant model and trains two small matrices beside each layer. After training, the supplement can be folded back in, so the model runs just as fast.
- Learns a low-rank update BA beside frozen weights
- On GPT-3 175B: up to 10,000× fewer trainable parameters and 3× less GPU memory than full fine-tuning, with matching or better quality
- No extra inference latency, because the update merges into the weights
The default way to fine-tune open models today (with 2023's QLoRA on top).
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, Al-Rfou & Constant (Google) · EMNLP 2021Keep the whole model frozen and learn only a few "soft" prompt tokens — not words, but numbers the model reads as if they were words. For small models this falls short of full fine-tuning, but as models get bigger the gap closes.
- Learns continuous prompt embeddings; the model itself stays frozen
- Gap to full model tuning shrinks with scale and closes at about 11B parameters (T5-XXL) on SuperGLUE
- One frozen model can serve many tasks by swapping small prompts
A direct ancestor of today's parameter-efficient fine-tuning toolkit.
06 👁️ Vision and Language Enter the Same Representation Space
Image ↔ shared meaning space ↔ language, then text prompt → generative model → image.
CLIP: Learning Transferable Visual Models From Natural Language Supervision
Radford et al. (OpenAI) · ICML 2021Show a model 400 million pictures with their captions and train it to match each picture to its caption. It learns a shared "meaning space" for images and words — so it can classify a photo just by comparing it with sentences like "a photo of a dog", with no labelled training set.
- Contrastive training of an image encoder and a text encoder on 400M image–text pairs
- Zero-shot ImageNet accuracy matched the original ResNet-50 without using any of its 1.28M labelled training images
- Strong transfer across 30+ datasets from natural-language class descriptions
The vision backbone behind DALL·E 2, Stable Diffusion and LLaVA.
ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia et al. (Google) · ICML 2021Instead of carefully cleaning data, ALIGN used 1.8 billion images with their messy web alt-text. The noise mostly cancelled out at that scale: sheer quantity made up for quality.
- Dual encoder (EfficientNet + BERT) trained contrastively on 1.8B noisy image–alt-text pairs with minimal filtering
- State of the art on Flickr30K and MSCOCO image–text retrieval at the time; strong ImageNet transfer
Independent confirmation that web-scale contrastive learning works — even with noisy data.
DALL·E: Zero-Shot Text-to-Image Generation
Ramesh et al. (OpenAI) · ICML 2021Treat a picture like a sentence: chop it into a grid of 1,024 "image words", put the caption's words in front, and train a 12B Transformer to predict the next token. Ask for "an armchair in the shape of an avocado" and it writes the image token by token.
- 12B autoregressive Transformer over up to 256 text tokens + 1,024 image tokens (a 32×32 grid from a discrete VAE)
- Trained on 250M image–text pairs; CLIP was used to rerank samples
- Competitive zero-shot on MS-COCO with models trained on it, by human evaluation
Showed language could drive image generation — then diffusion took over in 2022.
07 💻 Code Becomes a Language-Model Domain
Specification → model writes code → run the tests → sample alternatives → working program.
Codex: Evaluating Large Language Models Trained on Code
Chen et al. (OpenAI) · arXiv Jul 2021Fine-tune GPT on public GitHub code, then test it with HumanEval: 164 hand-written programming problems checked by unit tests. One try often fails — but let it try many times and one attempt usually passes.
- Introduced HumanEval (164 problems, judged by functional correctness)
- Codex-12B solved 28.8% with one sample (GPT-3: 0%); with 100 samples per problem 70.2% had at least one correct
- A production descendant powered the first GitHub Copilot
The start of the road from autocomplete to coding agents — and pass@k is still the standard metric.
08 🔎 External Knowledge and Tool Use
Parametric model + external retrieval + a browser environment + citations: the beginnings of the modern harness.
RETRO: Improving Language Models by Retrieving from Trillions of Tokens
Borgeaud et al. (DeepMind) · arXiv Dec 2021 · ICML 2022Rather than memorising everything in its weights, RETRO keeps a 2-trillion-token library and looks things up while it writes. A 7.5B model with a library performed like models 25 times its size.
- Splits text into chunks; retrieves nearest neighbours from a 2T-token database with a frozen BERT retriever; attends to them via chunked cross-attention
- Comparable to GPT-3 and Jurassic-1 on the Pile with 25× fewer parameters
- The database can be updated without retraining
Made external memory a serious alternative to ever-larger weights — a pillar of RAG.
WebGPT: Browser-Assisted Question-Answering with Human Feedback
Nakano et al. (OpenAI) · arXiv Dec 2021Give GPT-3 a simple text web browser: it can search, click, scroll and quote passages, then write an answer with citations. Humans then judged the answers, and that feedback trained it further.
- GPT-3 fine-tuned to use a text browser (search, click, quote) on ELI5 questions
- Trained with imitation learning and a reward model from human comparisons, with best-of-n sampling
- Best answers preferred to human demonstrators' 56% of the time, and to the top Reddit answer 69%
An early tool-using, citation-giving agent — ancestor of research agents and search-grounded chat.
09 🎨 Diffusion Becomes the New Generative Paradigm
GAN dominance → diffusion catches up → diffusion surpasses GANs → the 2022 text-to-image explosion.
Diffusion Models Beat GANs on Image Synthesis
Dhariwal & Nichol (OpenAI) · NeurIPS 2021Diffusion models learn to turn noise into pictures one small clean-up step at a time. This paper improved the architecture and added classifier guidance — a dial that nudges the image toward looking more like its class — and for the first time beat the best GANs.
- Architecture improvements plus classifier guidance to trade diversity for fidelity
- FID 2.97 on ImageNet 128×128, 4.59 at 256×256 and 7.72 at 512×512 — better than BigGAN-deep
- Better distribution coverage than GANs
The turning point that set up latent diffusion, DALL·E 2 and Imagen in 2022.
10 🧬 AI-for-Science Breakthrough
Protein sequence → AlphaFold → 3D structure + confidence → biological hypotheses and drug discovery.
Highly Accurate Protein Structure Prediction with AlphaFold
Jumper et al. (DeepMind) · Nature 2021Proteins are chains of amino acids that fold into 3D shapes, and the shape decides what they do. Working out a shape experimentally can take months or years. AlphaFold 2 predicts it from the sequence alone — often as accurately as the experiment — and says how confident it is in each part.
- Median GDT score of 92.4 in the CASP14 assessment, competitive with experimental structures for most targets
- Evoformer attention over sequence alignments and residue pairs, with per-residue confidence (pLDDT)
- Applied to 98.5% of the human proteome; Science's 2021 Breakthrough of the Year (with RoseTTAFold) and Nature Methods' 2021 Method of the Year
The clearest proof that deep learning can solve a long-standing scientific problem — 2024 Nobel Prize in Chemistry followed.
11 LoRA, Counted
LoRA's "10,000× fewer trainable parameters" is easy to check. The script counts trainable parameters for GPT-3 175B, then shows on a small matrix why the adapter adds no inference cost: it merges into the frozen weights.
"""LoRA in a few lines: how few parameters does adaptation need?
Hu et al. (2021) freeze a pretrained weight matrix W (d x d) and learn a
low-rank update B @ A, with B (d x r) and A (r x d). Only A and B train.
Part 1 counts trainable parameters for GPT-3 175B (96 layers, d = 12288),
adapting the query and value projections as in the paper.
Part 2 checks, on a tiny matrix, that the update can be merged into W,
so the adapted model runs exactly as fast as the original. Standard library only.
"""
import random
LAYERS, D, FULL = 96, 12288, 175e9
print("Part 1 - trainable parameters, GPT-3 175B")
print(f"{'method':>30} {'trainable':>12} {'fraction':>10} {'reduction':>10}")
print(f"{'full fine-tuning':>30} {FULL / 1e9:>11.0f}B {1:>10.0%} {'1x':>10}")
for r in [1, 4, 8, 64]:
n = LAYERS * 2 * (2 * D * r) # Wq and Wv, each gets B (d x r) + A (r x d)
print(f"{'LoRA r=' + str(r) + ' on Wq, Wv':>30} {n / 1e6:>11.1f}M {n / FULL:>10.4%} {FULL / n:>9,.0f}x")
print("\nPart 2 - merging the update (d = 6, r = 2)")
random.seed(0)
d, r = 6, 2
mat = lambda rows, cols, s=1.0: [[random.gauss(0, s) for _ in range(cols)] for _ in range(rows)]
mul = lambda X, Y: [[sum(X[i][k] * Y[k][j] for k in range(len(Y))) for j in range(len(Y[0]))] for i in range(len(X))]
add = lambda X, Y: [[a + b for a, b in zip(x, y)] for x, y in zip(X, Y)]
W, B, A, x = mat(d, d), mat(d, r, 0.3), mat(r, d, 0.3), mat(d, 1)
two_paths = add(mul(W, x), mul(B, mul(A, x))) # frozen path + adapter path
merged = mul(add(W, mul(B, A)), x) # W' = W + BA, one matmul
err = max(abs(a[0] - b[0]) for a, b in zip(two_paths, merged))
print(f" W has {d * d} numbers; the adapter B, A has {2 * d * r}.")
print(f" max |(Wx + BAx) - (W + BA)x| = {err:.1e} -> identical outputs after merging")
print("\nA 175B model can be adapted by training ~0.01% of its size, with no extra inference cost.")Output:
Part 1 - trainable parameters, GPT-3 175B
method trainable fraction reduction
full fine-tuning 175B 100% 1x
LoRA r=1 on Wq, Wv 4.7M 0.0027% 37,087x
LoRA r=4 on Wq, Wv 18.9M 0.0108% 9,272x
LoRA r=8 on Wq, Wv 37.7M 0.0216% 4,636x
LoRA r=64 on Wq, Wv 302.0M 0.1726% 579x
Part 2 - merging the update (d = 6, r = 2)
W has 36 numbers; the adapter B, A has 24.
max |(Wx + BAx) - (W + BA)x| = 4.4e-16 -> identical outputs after merging
A 175B model can be adapted by training ~0.01% of its size, with no extra inference cost.The 4.7M (rank 1) and 37.7M (rank 8) counts match the GPT-3 rows in the LoRA paper; the headline 10,000× sits between them. Try it in the LoRA card. The same idea, with 4-bit quantisation added, became QLoRA in 2023.
12 The 2020 → 2026 Roadmap
From scaling to self-improving systems. Press play:
Next: AI Research in 2020: The Year Scaling Became the AI Strategy →