Home › Blog › AI Research in 2020
AI Research Progress Through the Years · 2020 📐

AI Research in 2020: The Year Scaling Became the AI Strategy

2020 set down the basic rules of modern AI. Loss turned out to fall predictably with scale, and GPT-3 showed that one very large model could take on new tasks from a few examples. Models learned to look things up and to learn from human preferences. Transformers moved into vision, and self-supervised learning spread to images and speech. Diffusion models, neural 3D scenes and trillion-parameter-scale engineering all started this year. This guide covers twelve papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the seventh article in AI Research Progress Through the Years. It follows 2019, and the story continues in 2021, 2022, 2023, 2024, 2025 and 2026. Awards come from the official NeurIPS 2020 and ECCV 2020 announcements. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.
▶ The paradigm shift of 2020 Toggle

01 Venues and Awards

🏆 Best Paper🏅 Best Paper Honorable Mention📄 Accepted paper📚 Journal📝 Preprint
VenueCovered here
NeurIPS 2020 (virtual)Best Paper: GPT-3 (one of three). Also: RAG · Learning to Summarize from Human Feedback · wav2vec 2.0 · DDPM
ECCV 2020NeRF — Best Paper Honorable Mention · DETR
ICML 2020SimCLR
JMLR 2020T5 (first posted October 2019)
Posted 2020, published laterViT and GShard (ICLR 2021) · Scaling Laws (arXiv preprint)

02 Paper Map

Twelve papers in eight themes. Filter by award and click to jump:

▶ 2020 paper map Click to jump

03 📈 Scaling Laws + GPT-3: Bigger Models Become General Learners

Scale → emergent generality → prompting becomes an interface.

📉

Scaling Laws for Neural Language Models

Kaplan et al. (OpenAI, Johns Hopkins) · arXiv Jan 2020
📝 Preprint / report
🧒 In plain English

Train lots of language models of different sizes and plot how good they get. The result is strikingly regular: loss falls along a straight line on a log–log chart as you add parameters, data or compute. That turned "how big should we build?" into something you could calculate in advance.

▶ Loss vs model size Drag model size
🔍 What they found
  • Loss follows power laws in parameters, dataset size and compute, over more than seven orders of magnitude
  • Architecture details such as depth versus width matter much less than scale, within a wide range
  • Larger models are more sample-efficient; for a fixed budget, train very large models and stop well before convergence
💡 Why it matters

Gave labs a reason to bet on GPT-3-sized models. Its compute-allocation advice was revised by Chinchilla in 2022.

🧠

GPT-3: Language Models are Few-Shot Learners

Brown et al. (OpenAI) · NeurIPS 2020
🏆 Best Paper
🧒 In plain English

Before GPT-3, you trained a separate model for each job. GPT-3 — 175 billion parameters — could do many jobs just from a few examples typed into the prompt. No retraining: the examples are the instructions.

▶ Zero-, one- and few-shot Pick a prompt, then run
🔍 What they found
  • 175B-parameter autoregressive Transformer trained on about 300B tokens
  • Strong zero-, one- and few-shot performance across translation, question answering, cloze tasks and arithmetic, with no gradient updates
  • People could barely tell its generated news articles from human-written ones (about 52% detection accuracy for the largest model)
  • NeurIPS 2020 Best Paper Award
💡 Why it matters

Made prompting an interface and general-purpose models convincing — the direct ancestor of ChatGPT.

04 🔁 Text-to-Text Transfer and External Knowledge

One model for many tasks (T5), and one model plus external knowledge for grounded answers (RAG).

🔁

T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Raffel et al. (Google) · arXiv Oct 2019 · JMLR 2020
📚 Journal
🧒 In plain English

Translation, classification, summarising, answering questions — T5 treats all of them the same way: text in, text out. You tell it the task with a short prefix like "translate English to German:" and it writes the answer. One model, one training objective, many tasks.

▶ One model, many tasks Pick a task
🔍 What they found
  • Cast every NLP task as text-to-text, with a task prefix
  • Introduced C4, a ~750GB cleaned Common Crawl corpus
  • Systematic study of objectives, architectures, datasets and transfer methods; the 11B model set state of the art on GLUE, SuperGLUE, SQuAD and CNN/Daily Mail
💡 Why it matters

"Everything is text" became the standard interface — instruction tuning in 2021–22 builds directly on it.

📚

RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Lewis et al. (Facebook AI, UCL, NYU) · NeurIPS 2020
📄 Accepted paper
🧒 In plain English

Should a model memorise every fact? RAG says no: first look up relevant passages (here, from Wikipedia), then write the answer using them. The model's own memory plus an external library — like an open-book exam.

▶ Open-book answering Press run
🔍 What they found
  • Combines a pretrained generator (BART) with a dense retriever over a Wikipedia index
  • State of the art on three open-domain question-answering tasks at the time
  • Answers were more specific and factual than a parametric-only model; the index can be swapped to update knowledge
💡 Why it matters

Named the pattern behind enterprise search, agent memory and context engineering.

🔗 References

05 🧑‍⚖️ Human Feedback Becomes a Training Signal

Alignment begins moving from labels to preferences: outputs → human comparisons → reward model → RL.

🧑‍⚖️

Learning to Summarize from Human Feedback

Stiennon et al. (OpenAI) · NeurIPS 2020
📄 Accepted paper
🧒 In plain English

Instead of teaching a model to copy example summaries, ask people which of two summaries is better. Train a "judge" model on thousands of those choices, then use reinforcement learning to make summaries the judge likes. The result beat much larger supervised models.

▶ The first RLHF loop Press train
🔍 What they found
  • Collected human comparisons between summaries of Reddit TL;DR posts; trained a reward model; optimised with PPO
  • Human-feedback models were preferred to much larger supervised models and to the human-written reference summaries
  • Transferred to news summarisation (CNN/Daily Mail) without news-specific tuning; showed the reward can be over-optimised
💡 Why it matters

The pipeline InstructGPT and ChatGPT scaled up in 2022.

06 🖼️ Transformers Escape NLP

The Transformer goes from a language architecture to image classification and object detection.

🧩

Vision Transformer (ViT): An Image is Worth 16×16 Words

Dosovitskiy et al. (Google) · arXiv Oct 2020 · ICLR 2021
📄 Accepted paper
🧒 In plain English

Cut a photo into a grid of small squares, treat each square like a word, and feed the "sentence" to a standard Transformer. With enough pretraining images, this beats the convolutional networks that had ruled vision for a decade.

▶ Image → patches → tokens Press run
🔍 What they found
  • Splits images into 16×16 patches, embeds them and adds position information; almost no vision-specific design
  • Pretrained on large datasets (up to JFT-300M), it matched or beat state-of-the-art CNNs on ImageNet and other benchmarks with substantially less training compute
  • Underperforms CNNs when trained on small datasets — scale matters
💡 Why it matters

Made the Transformer the common architecture for vision, and later for multimodal models.

🎯

DETR: End-to-End Object Detection with Transformers

Carion et al. (Facebook AI) · ECCV 2020
📄 Accepted paper
🧒 In plain English

Older object detectors guessed thousands of overlapping boxes, then used hand-written rules to clean them up. DETR simply asks a Transformer to output a short list of objects, and trains by matching that list one-to-one with the true objects.

▶ Many guesses vs a set Toggle
🔍 What they found
  • Object detection as direct set prediction: learned object queries + Transformer encoder–decoder + bipartite (Hungarian) matching loss
  • Removes anchors and non-maximum suppression
  • Matched a well-tuned Faster R-CNN on COCO; extends easily to panoptic segmentation
💡 Why it matters

Showed Transformers could replace hand-designed pipelines outside language.

🔗 References

07 🧲 Self-Supervised Learning Expands Beyond Language

Massive unlabelled text, images and audio → pretraining without human labels.

🧲

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations

Chen, Kornblith, Norouzi & Hinton (Google) · ICML 2020
📄 Accepted paper
🧒 In plain English

No labels needed: take a photo, make two altered copies (cropped, recoloured), and teach a network that those two belong together while every other photo is different. It learns what matters in images by itself.

▶ Pull together, push apart Press train
🔍 What they found
  • Two augmented views per image, an encoder, a projection head and a contrastive loss over large batches
  • Strong augmentation (especially crop plus colour distortion) and big batches were key
  • 76.5% ImageNet top-1 with a linear classifier — matching a supervised ResNet-50; 85.8% top-5 with only 1% of labels
💡 Why it matters

Self-supervision became a serious way to learn vision — a path to CLIP in 2021.

🎙️

wav2vec 2.0: Self-Supervised Learning of Speech Representations

Baevski et al. (Facebook AI) · NeurIPS 2020
📄 Accepted paper
🧒 In plain English

Listen to tens of thousands of hours of unlabelled speech, hide parts of it, and learn to pick out what was hidden. After that, the model needs only a tiny amount of transcribed speech to recognise words well.

▶ How much transcribed speech is needed? Drag labelled data
🔍 What they found
  • Masks latent speech features and solves a contrastive task over learned quantised targets
  • With just 10 minutes of labelled data (plus 53k hours unlabelled): 4.8 / 8.2 word error rate on LibriSpeech test clean/other
  • With all 960 hours: 1.8 / 3.3
💡 Why it matters

Carried the pretraining recipe into speech — and into low-resource languages.

08 🌫️ Diffusion Models Arrive

Add noise step by step, then learn to reverse it: the quiet beginning of the diffusion era.

🌫️

Denoising Diffusion Probabilistic Models (DDPM)

Ho, Jain & Abbeel (UC Berkeley) · NeurIPS 2020
📄 Accepted paper
🧒 In plain English

Take a photo and add a little static, again and again, until it is pure noise. Then train a network to undo one step at a time. Start from fresh noise, run the undo steps, and a brand-new image appears.

▶ Noise in, image out Drag the step, or press generate
🔍 What they found
  • Simple training objective: predict the noise that was added (linked to denoising score matching)
  • 1,000 noising steps; FID 3.17 and Inception score 9.46 on CIFAR-10
  • Sample quality comparable to strong GANs at the time
💡 Why it matters

The quiet beginning of Stable Diffusion, DALL·E 2, Imagen and video generators.

09 🏛️ Neural Fields and the Beginning of Neural 3D Worlds

3D position + view direction → small network → density and colour → rendering → a new camera view.

🏛️

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Mildenhall et al. (UC Berkeley, Google, UC San Diego) · ECCV 2020
🏅 Best Paper Honorable Mention
🧒 In plain English

Give a network a few dozen photos of a scene. It learns a function that answers: "at this point in space, looking from this direction, what colour is there, and how solid?" Shoot imaginary rays through that function and you can render the scene from views no camera took.

▶ Rendering one ray Press render
🔍 What they found
  • A small MLP maps 5D input (3D position + 2D viewing direction) to density and colour
  • Classical volume rendering turns samples along camera rays into pixels; positional encoding captures fine detail
  • Photorealistic novel views from input photos with known camera poses; ECCV 2020 Best Paper Honorable Mention
💡 Why it matters

Founded neural rendering: 3D reconstruction, Gaussian splatting, digital twins and generative 3D.

10 🖧 AI Systems Engineering Becomes Necessary

Sparse experts + automatic sharding + compiler support = models too big for any single chip.

🖧

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Lepikhin et al. (Google) · arXiv Jun 2020 · ICLR 2021
📄 Accepted paper
🧒 In plain English

A 600-billion-parameter model doesn't fit on one chip. GShard splits it across 2,048 chips: each token visits only two of many "expert" sub-networks, and a compiler works out automatically which chip holds what.

▶ Tokens, experts and chips Press route
🔍 What they found
  • Sparsely gated Mixture-of-Experts with top-2 gating, plus lightweight sharding annotations and the XLA compiler
  • A 600B-parameter multilingual translation model (100 languages to English) trained on 2,048 TPU v3 cores in 4 days
  • Far better translation quality than dense baselines at lower training cost
💡 Why it matters

From here on, AI progress depended as much on systems engineering as on architecture. Switch Transformer (2021) simplified it to one expert.

🔗 References

11 Scaling Laws, Computed

Kaplan's power law fits in one line of Python. The script predicts loss as models grow, then compares where extra compute should go under the 2020 recipe and under Chinchilla's 2022 correction:

"""Kaplan's scaling laws in a few lines - and how the recipe changed in 2022.

Kaplan et al. (2020) fitted test loss (nats per token) of Transformer language
models as power laws. With enough data, loss depends on non-embedding
parameters N as
    L(N) = (N_c / N) ** alpha_N,   N_c = 8.8e13, alpha_N = 0.076
They also found that as compute grows, the optimal model grows as C^0.73
and the data as C^0.27. Chinchilla (2022) refitted this and found C^0.5 each.
Standard library only; the fits are from the papers, the losses are
extrapolations of those fits, not measured values.
"""
N_C, ALPHA_N = 8.8e13, 0.076

def loss(n):
    return (N_C / n) ** ALPHA_N

print("Part 1 - loss falls as a power law in model size")
print(f"{'parameters':>12} {'predicted loss':>15}")
for n in [1e6, 1e7, 1e8, 1e9, 1e10, 1e11, 1.75e11]:
    print(f"{n:>12.2e} {loss(n):>15.3f}")
print(f"   each 10x in size multiplies loss by {10 ** -ALPHA_N:.3f} (a {1 - 10 ** -ALPHA_N:.1%} cut)")

print("\nPart 2 - where should 10x more compute go?")
print(f"{'recipe':>22} {'params x':>9} {'data x':>8}")
for name, a in [("Kaplan et al. 2020", 0.73), ("Chinchilla 2022", 0.50)]:
    print(f"{name:>22} {10 ** a:>9.2f} {10 ** (1 - a):>8.2f}")
print("\nStarting from GPT-3's 175B params / 300B tokens, grow compute 100x:")
for name, a in [("Kaplan et al. 2020", 0.73), ("Chinchilla 2022", 0.50)]:
    n, d = 175e9 * 100 ** a, 300e9 * 100 ** (1 - a)
    print(f"   {name:<20} -> {n / 1e12:5.1f}T params, {d / 1e12:5.1f}T tokens")
print("\n2020's recipe favoured ever-bigger models; 2022 showed data matters as much.")

Output:

Part 1 - loss falls as a power law in model size
  parameters  predicted loss
    1.00e+06           4.016
    1.00e+07           3.371
    1.00e+08           2.830
    1.00e+09           2.376
    1.00e+10           1.994
    1.00e+11           1.674
    1.75e+11           1.604
   each 10x in size multiplies loss by 0.839 (a 16.1% cut)

Part 2 - where should 10x more compute go?
                recipe  params x   data x
    Kaplan et al. 2020      5.37     1.86
       Chinchilla 2022      3.16     3.16

Starting from GPT-3's 175B params / 300B tokens, grow compute 100x:
   Kaplan et al. 2020   ->   5.0T params,   1.0T tokens
   Chinchilla 2022      ->   1.8T params,   3.0T tokens

2020's recipe favoured ever-bigger models; 2022 showed data matters as much.

Each tenfold increase in parameters cuts predicted loss by about 16%, however large the model already is. That regularity is what made scaling a strategy rather than a gamble. The allocation advice didn't last: the 2020 recipe put most new compute into parameters, while Chinchilla showed data should grow just as fast. The losses are extrapolations of the fit, not measurements.

12 The 2017 → 2026 Roadmap

From the Transformer to self-improving systems. Press play:

▶ Ten years, thirteen stages Click a stage or press play
2020Discover the scaling recipe
→
2021Assemble the foundation-model stack
→
2022Add reasoning, alignment and action
→
2023Programmable platforms
→
2024Environments and harnesses
→
2025Reasoning, agency and discovery
→
2026Persistent self-improvement
AI research in 2020, in one paragraph
2020 established the technical grammar of modern AI. Scaling laws made progress predictable, and GPT-3 showed that one large model could be prompted into many tasks. T5 made text the universal interface, and RAG gave models an external library. Learning to Summarize turned human preferences into a training signal. ViT and DETR carried Transformers into vision, while SimCLR and wav2vec 2.0 showed that images and speech could be learned without labels. DDPM quietly began the diffusion era, NeRF turned photos into neural 3D scenes, and GShard showed that giant models needed new systems engineering. Every later article in this series builds on these pieces.

Next: AI Research in 2019: The Year Pretraining Became the Default →

⚠️ Notes

About the source infographic and brief: (1) the "Related 2020 Milestones" box lists DALL·E as a "late 2020 preprint"; DALL·E was announced in January 2021 and its paper posted in February 2021, so it is covered in the 2021 article; (2) the scaling-laws chart's loss axis runs from 10⁻² to 10¹ and its curves fall by orders of magnitude; in the paper, loss falls slowly (roughly from 6 to 2 nats per token) along straight lines on a log–log plot — see the widget and lab; (3) T5 is labelled 2020 but was first posted in October 2019 (published in JMLR in 2020), which the timeline's "2019 · T5-era transfer" already hints at; (4) awards missing from the infographic are added: GPT-3 won a NeurIPS 2020 Best Paper Award, and NeRF a ECCV 2020 Best Paper Honorable Mention. Other figures checked out against the papers (seven orders of magnitude, 175B parameters, C4, SimCLR's 76.5%, DDPM's FID 3.17, GShard's 600B parameters on 2,048 TPU v3 cores, wav2vec 2.0 with minutes of labelled data). Widget numbers not taken from a paper are marked illustrative.

🔗 All References