AI Research in 2020: The Year Scaling Became the AI Strategy
2020 set down the basic rules of modern AI. Loss turned out to fall predictably with scale, and GPT-3 showed that one very large model could take on new tasks from a few examples. Models learned to look things up and to learn from human preferences. Transformers moved into vision, and self-supervised learning spread to images and speech. Diffusion models, neural 3D scenes and trillion-parameter-scale engineering all started this year. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 Venues and Awards
| Venue | Covered here |
|---|---|
| NeurIPS 2020 (virtual) | Best Paper: GPT-3 (one of three). Also: RAG · Learning to Summarize from Human Feedback · wav2vec 2.0 · DDPM |
| ECCV 2020 | NeRF — Best Paper Honorable Mention · DETR |
| ICML 2020 | SimCLR |
| JMLR 2020 | T5 (first posted October 2019) |
| Posted 2020, published later | ViT and GShard (ICLR 2021) · Scaling Laws (arXiv preprint) |
02 Paper Map
Twelve papers in eight themes. Filter by award and click to jump:
03 📈 Scaling Laws + GPT-3: Bigger Models Become General Learners
Scale → emergent generality → prompting becomes an interface.
Scaling Laws for Neural Language Models
Kaplan et al. (OpenAI, Johns Hopkins) · arXiv Jan 2020Train lots of language models of different sizes and plot how good they get. The result is strikingly regular: loss falls along a straight line on a log–log chart as you add parameters, data or compute. That turned "how big should we build?" into something you could calculate in advance.
- Loss follows power laws in parameters, dataset size and compute, over more than seven orders of magnitude
- Architecture details such as depth versus width matter much less than scale, within a wide range
- Larger models are more sample-efficient; for a fixed budget, train very large models and stop well before convergence
Gave labs a reason to bet on GPT-3-sized models. Its compute-allocation advice was revised by Chinchilla in 2022.
GPT-3: Language Models are Few-Shot Learners
Brown et al. (OpenAI) · NeurIPS 2020Before GPT-3, you trained a separate model for each job. GPT-3 — 175 billion parameters — could do many jobs just from a few examples typed into the prompt. No retraining: the examples are the instructions.
- 175B-parameter autoregressive Transformer trained on about 300B tokens
- Strong zero-, one- and few-shot performance across translation, question answering, cloze tasks and arithmetic, with no gradient updates
- People could barely tell its generated news articles from human-written ones (about 52% detection accuracy for the largest model)
- NeurIPS 2020 Best Paper Award
Made prompting an interface and general-purpose models convincing — the direct ancestor of ChatGPT.
04 🔁 Text-to-Text Transfer and External Knowledge
One model for many tasks (T5), and one model plus external knowledge for grounded answers (RAG).
T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel et al. (Google) · arXiv Oct 2019 · JMLR 2020Translation, classification, summarising, answering questions — T5 treats all of them the same way: text in, text out. You tell it the task with a short prefix like "translate English to German:" and it writes the answer. One model, one training objective, many tasks.
- Cast every NLP task as text-to-text, with a task prefix
- Introduced C4, a ~750GB cleaned Common Crawl corpus
- Systematic study of objectives, architectures, datasets and transfer methods; the 11B model set state of the art on GLUE, SuperGLUE, SQuAD and CNN/Daily Mail
"Everything is text" became the standard interface — instruction tuning in 2021–22 builds directly on it.
RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al. (Facebook AI, UCL, NYU) · NeurIPS 2020Should a model memorise every fact? RAG says no: first look up relevant passages (here, from Wikipedia), then write the answer using them. The model's own memory plus an external library — like an open-book exam.
- Combines a pretrained generator (BART) with a dense retriever over a Wikipedia index
- State of the art on three open-domain question-answering tasks at the time
- Answers were more specific and factual than a parametric-only model; the index can be swapped to update knowledge
Named the pattern behind enterprise search, agent memory and context engineering.
05 🧑⚖️ Human Feedback Becomes a Training Signal
Alignment begins moving from labels to preferences: outputs → human comparisons → reward model → RL.
Learning to Summarize from Human Feedback
Stiennon et al. (OpenAI) · NeurIPS 2020Instead of teaching a model to copy example summaries, ask people which of two summaries is better. Train a "judge" model on thousands of those choices, then use reinforcement learning to make summaries the judge likes. The result beat much larger supervised models.
- Collected human comparisons between summaries of Reddit TL;DR posts; trained a reward model; optimised with PPO
- Human-feedback models were preferred to much larger supervised models and to the human-written reference summaries
- Transferred to news summarisation (CNN/Daily Mail) without news-specific tuning; showed the reward can be over-optimised
The pipeline InstructGPT and ChatGPT scaled up in 2022.
06 🖼️ Transformers Escape NLP
The Transformer goes from a language architecture to image classification and object detection.
Vision Transformer (ViT): An Image is Worth 16×16 Words
Dosovitskiy et al. (Google) · arXiv Oct 2020 · ICLR 2021Cut a photo into a grid of small squares, treat each square like a word, and feed the "sentence" to a standard Transformer. With enough pretraining images, this beats the convolutional networks that had ruled vision for a decade.
- Splits images into 16×16 patches, embeds them and adds position information; almost no vision-specific design
- Pretrained on large datasets (up to JFT-300M), it matched or beat state-of-the-art CNNs on ImageNet and other benchmarks with substantially less training compute
- Underperforms CNNs when trained on small datasets — scale matters
Made the Transformer the common architecture for vision, and later for multimodal models.
DETR: End-to-End Object Detection with Transformers
Carion et al. (Facebook AI) · ECCV 2020Older object detectors guessed thousands of overlapping boxes, then used hand-written rules to clean them up. DETR simply asks a Transformer to output a short list of objects, and trains by matching that list one-to-one with the true objects.
- Object detection as direct set prediction: learned object queries + Transformer encoder–decoder + bipartite (Hungarian) matching loss
- Removes anchors and non-maximum suppression
- Matched a well-tuned Faster R-CNN on COCO; extends easily to panoptic segmentation
Showed Transformers could replace hand-designed pipelines outside language.
07 🧲 Self-Supervised Learning Expands Beyond Language
Massive unlabelled text, images and audio → pretraining without human labels.
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Chen, Kornblith, Norouzi & Hinton (Google) · ICML 2020No labels needed: take a photo, make two altered copies (cropped, recoloured), and teach a network that those two belong together while every other photo is different. It learns what matters in images by itself.
- Two augmented views per image, an encoder, a projection head and a contrastive loss over large batches
- Strong augmentation (especially crop plus colour distortion) and big batches were key
- 76.5% ImageNet top-1 with a linear classifier — matching a supervised ResNet-50; 85.8% top-5 with only 1% of labels
Self-supervision became a serious way to learn vision — a path to CLIP in 2021.
wav2vec 2.0: Self-Supervised Learning of Speech Representations
Baevski et al. (Facebook AI) · NeurIPS 2020Listen to tens of thousands of hours of unlabelled speech, hide parts of it, and learn to pick out what was hidden. After that, the model needs only a tiny amount of transcribed speech to recognise words well.
- Masks latent speech features and solves a contrastive task over learned quantised targets
- With just 10 minutes of labelled data (plus 53k hours unlabelled): 4.8 / 8.2 word error rate on LibriSpeech test clean/other
- With all 960 hours: 1.8 / 3.3
Carried the pretraining recipe into speech — and into low-resource languages.
08 🌫️ Diffusion Models Arrive
Add noise step by step, then learn to reverse it: the quiet beginning of the diffusion era.
Denoising Diffusion Probabilistic Models (DDPM)
Ho, Jain & Abbeel (UC Berkeley) · NeurIPS 2020Take a photo and add a little static, again and again, until it is pure noise. Then train a network to undo one step at a time. Start from fresh noise, run the undo steps, and a brand-new image appears.
- Simple training objective: predict the noise that was added (linked to denoising score matching)
- 1,000 noising steps; FID 3.17 and Inception score 9.46 on CIFAR-10
- Sample quality comparable to strong GANs at the time
The quiet beginning of Stable Diffusion, DALL·E 2, Imagen and video generators.
09 🏛️ Neural Fields and the Beginning of Neural 3D Worlds
3D position + view direction → small network → density and colour → rendering → a new camera view.
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall et al. (UC Berkeley, Google, UC San Diego) · ECCV 2020Give a network a few dozen photos of a scene. It learns a function that answers: "at this point in space, looking from this direction, what colour is there, and how solid?" Shoot imaginary rays through that function and you can render the scene from views no camera took.
- A small MLP maps 5D input (3D position + 2D viewing direction) to density and colour
- Classical volume rendering turns samples along camera rays into pixels; positional encoding captures fine detail
- Photorealistic novel views from input photos with known camera poses; ECCV 2020 Best Paper Honorable Mention
Founded neural rendering: 3D reconstruction, Gaussian splatting, digital twins and generative 3D.
10 🖧 AI Systems Engineering Becomes Necessary
Sparse experts + automatic sharding + compiler support = models too big for any single chip.
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin et al. (Google) · arXiv Jun 2020 · ICLR 2021A 600-billion-parameter model doesn't fit on one chip. GShard splits it across 2,048 chips: each token visits only two of many "expert" sub-networks, and a compiler works out automatically which chip holds what.
- Sparsely gated Mixture-of-Experts with top-2 gating, plus lightweight sharding annotations and the XLA compiler
- A 600B-parameter multilingual translation model (100 languages to English) trained on 2,048 TPU v3 cores in 4 days
- Far better translation quality than dense baselines at lower training cost
From here on, AI progress depended as much on systems engineering as on architecture. Switch Transformer (2021) simplified it to one expert.
11 Scaling Laws, Computed
Kaplan's power law fits in one line of Python. The script predicts loss as models grow, then compares where extra compute should go under the 2020 recipe and under Chinchilla's 2022 correction:
"""Kaplan's scaling laws in a few lines - and how the recipe changed in 2022.
Kaplan et al. (2020) fitted test loss (nats per token) of Transformer language
models as power laws. With enough data, loss depends on non-embedding
parameters N as
L(N) = (N_c / N) ** alpha_N, N_c = 8.8e13, alpha_N = 0.076
They also found that as compute grows, the optimal model grows as C^0.73
and the data as C^0.27. Chinchilla (2022) refitted this and found C^0.5 each.
Standard library only; the fits are from the papers, the losses are
extrapolations of those fits, not measured values.
"""
N_C, ALPHA_N = 8.8e13, 0.076
def loss(n):
return (N_C / n) ** ALPHA_N
print("Part 1 - loss falls as a power law in model size")
print(f"{'parameters':>12} {'predicted loss':>15}")
for n in [1e6, 1e7, 1e8, 1e9, 1e10, 1e11, 1.75e11]:
print(f"{n:>12.2e} {loss(n):>15.3f}")
print(f" each 10x in size multiplies loss by {10 ** -ALPHA_N:.3f} (a {1 - 10 ** -ALPHA_N:.1%} cut)")
print("\nPart 2 - where should 10x more compute go?")
print(f"{'recipe':>22} {'params x':>9} {'data x':>8}")
for name, a in [("Kaplan et al. 2020", 0.73), ("Chinchilla 2022", 0.50)]:
print(f"{name:>22} {10 ** a:>9.2f} {10 ** (1 - a):>8.2f}")
print("\nStarting from GPT-3's 175B params / 300B tokens, grow compute 100x:")
for name, a in [("Kaplan et al. 2020", 0.73), ("Chinchilla 2022", 0.50)]:
n, d = 175e9 * 100 ** a, 300e9 * 100 ** (1 - a)
print(f" {name:<20} -> {n / 1e12:5.1f}T params, {d / 1e12:5.1f}T tokens")
print("\n2020's recipe favoured ever-bigger models; 2022 showed data matters as much.")Output:
Part 1 - loss falls as a power law in model size
parameters predicted loss
1.00e+06 4.016
1.00e+07 3.371
1.00e+08 2.830
1.00e+09 2.376
1.00e+10 1.994
1.00e+11 1.674
1.75e+11 1.604
each 10x in size multiplies loss by 0.839 (a 16.1% cut)
Part 2 - where should 10x more compute go?
recipe params x data x
Kaplan et al. 2020 5.37 1.86
Chinchilla 2022 3.16 3.16
Starting from GPT-3's 175B params / 300B tokens, grow compute 100x:
Kaplan et al. 2020 -> 5.0T params, 1.0T tokens
Chinchilla 2022 -> 1.8T params, 3.0T tokens
2020's recipe favoured ever-bigger models; 2022 showed data matters as much.Each tenfold increase in parameters cuts predicted loss by about 16%, however large the model already is. That regularity is what made scaling a strategy rather than a gamble. The allocation advice didn't last: the 2020 recipe put most new compute into parameters, while Chinchilla showed data should grow just as fast. The losses are extrapolations of the fit, not measurements.
12 The 2017 → 2026 Roadmap
From the Transformer to self-improving systems. Press play:
Next: AI Research in 2019: The Year Pretraining Became the Default →