Home › Blog › AI Research in 2022
AI Research Progress Through the Years · 2022 🌱

AI Research in 2022: The Year Reasoning, Alignment and Generative AI Converged

2022 laid the technical foundation for ChatGPT, reasoning models, agents and multimodal AI. Scaling became a science: Chinchilla showed models should grow with their data. Instruction tuning and RLHF turned raw models into assistants. Chain-of-thought made reasoning visible, and ReAct joined reasoning to action. Diffusion models set off the generative-image explosion. Flamingo and FlashAttention showed that multimodality and systems engineering mattered as much as size. This guide covers seventeen papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the fifth article in AI Research Progress Through the Years, and the starting point of the story told in 2023, 2024, 2025 and 2026. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.

01 Venues and Awards

🏆 Outstanding Paper📄 Accepted paper📚 Journal📰 Science📝 Preprint / report
VenueCovered here
NeurIPS 2022 (New Orleans)Outstanding Papers: Chinchilla ("An empirical analysis of compute-optimal LLM training") · Imagen. Also: InstructGPT · Chain-of-Thought · Minerva · Flamingo · FlashAttention
ICLR 2023ReAct (Oral) · Self-Consistency (both posted in 2022)
Other venuesLatent Diffusion (CVPR 2022) · SayCan (CoRL 2022) · RT-1 (RSS 2023) · AlphaCode (Science) · Emergent Abilities (TMLR) · PaLM and Flan (JMLR) · DALL·E 2 (preprint)

02 Paper Map

Seventeen papers in eight themes. Filter and click to jump:

▶ 2022 paper map Click to jump

03 📈 Scaling Gets Smarter

More parameters alone → compute-optimal parameters + data → unexpected capabilities at scale.

🌴

PaLM: Scaling Language Modeling with Pathways

Chowdhery et al. (Google) · arXiv Apr 2022 · JMLR 2023
📚 Journal
🧒 In plain English

PaLM was a 540-billion-parameter model trained across thousands of Google's TPU chips at once. At that size it could explain jokes, do multi-step reasoning with worked examples and write code — and on some tasks performance jumped sharply as the model grew.

▶ Scale and capability Drag model size
🔍 What they found
  • 540B dense decoder-only Transformer trained on 6,144 TPU v4 chips with the Pathways system
  • Breakthrough few-shot results on reasoning, code and language tasks; some tasks showed discontinuous improvement from 62B to 540B
💡 Why it matters

Showed what extreme scale plus good prompting could do — the base for Flan-PaLM and Minerva.

🔗 References
🐭

Chinchilla: Training Compute-Optimal Large Language Models

Hoffmann et al. (DeepMind) · NeurIPS 2022
🏆 Outstanding Paper
🧒 In plain English

If you have a fixed amount of computing power, should you build a bigger model or train it on more text? DeepMind trained over 400 models to find out: grow both together. Their 70B Chinchilla used the same compute as the 280B Gopher but four times more data — and beat it.

▶ Split a compute budget Drag the split
🔍 What they found
  • Over 400 models (70M–16B params, 5–500B tokens): parameters and tokens should scale in equal proportion
  • Chinchilla (70B, 1.4T tokens) outperformed Gopher (280B), GPT-3 (175B) and Megatron-Turing NLG (530B) at the same compute as Gopher; 67.5% on MMLU
  • Presented at NeurIPS 2022 as "An empirical analysis of compute-optimal large language model training" (Outstanding Paper)
💡 Why it matters

Changed how every lab sized its models; the 20-tokens-per-parameter rule of thumb comes from here.

✨

Emergent Abilities of Large Language Models

Wei et al. (Google, Stanford, DeepMind) · TMLR 2022
📚 Journal
🧒 In plain English

Some skills seemed to be absent in small models and then switch on in large ones, like water suddenly boiling. The paper catalogued dozens of such "emergent" abilities — sparking huge interest in scaling (and, a year later, a debate over whether some were an illusion of how they were measured).

▶ An ability that "switches on" Drag scale
🔍 What they found
  • Defines an ability as emergent if not present in smaller models but present in larger ones
  • Catalogues emergent few-shot tasks and prompting strategies (e.g. chain-of-thought helps only above ~100B parameters)
💡 Why it matters

Framed scaling research for years — counterpoint in 2023's Mirage paper.

04 🧑‍🏫 Instruction Following and RLHF Become the Product Layer

Pretrained model → SFT → human preferences → RLHF → useful assistant.

🧑‍🏫

InstructGPT: Training Language Models to Follow Instructions with Human Feedback

Ouyang et al. (OpenAI) · NeurIPS 2022
📄 Accepted paper
🧒 In plain English

GPT-3 could write anything — but it didn't reliably do what you asked. OpenAI had people write good answers, rank model answers, trained a model of those rankings, and used it to steer GPT-3 with reinforcement learning. People preferred a 1.3B InstructGPT over the 175B GPT-3.

▶ From raw model to assistant Press train
🔍 What they found
  • Three stages: supervised fine-tuning on demonstrations, a reward model from human rankings, PPO against the reward
  • Labelers preferred 1.3B InstructGPT outputs to 175B GPT-3 outputs on the API prompt distribution
  • More truthful and less toxic, with small performance regressions on some benchmarks
💡 Why it matters

The recipe behind ChatGPT, released later in 2022.

🍮

Scaling Instruction-Finetuned Language Models (Flan-PaLM)

Chung et al. (Google) · arXiv Oct 2022 · JMLR 2024
📚 Journal
🧒 In plain English

Instead of teaching a model one task, teach it 1,800 different tasks, each phrased as an instruction ("Translate…", "Summarise…", "Explain step by step…"). It learns to follow instructions in general — including for tasks it never saw.

▶ More instruction tasks → better generalisation Drag the number of tasks
🔍 What they found
  • Instruction tuning scaled to ~1.8K tasks, with and without chain-of-thought data
  • Flan-PaLM 540B reached 75.2% five-shot on MMLU; Flan-T5 checkpoints released publicly
  • Gains grew with both model size and number of tasks
💡 Why it matters

Showed instruction tuning is cheap, general and complementary to RLHF.

05 💡 Reasoning Becomes Visible

Answer immediately → think step by step → explore several reasoning paths → quantitative reasoning.

🧮

Chain-of-Thought Prompting Elicits Reasoning in LLMs

Wei et al. (Google) · NeurIPS 2022
📄 Accepted paper
🧒 In plain English

Show the model a few examples where the answer comes with the working — "60 miles per hour for 3 hours is 60 × 3 = 180" — and it starts showing its own working too. For big models, that simple change makes maths word problems far more likely to be right.

▶ Same question, two prompts Toggle
🔍 What they found
  • Eight worked examples in the prompt; no fine-tuning
  • PaLM 540B with chain-of-thought reached state of the art on GSM8K, beating fine-tuned GPT-3 with a verifier
  • Helps only at large scale (~100B parameters)
💡 Why it matters

Made reasoning visible — the ancestor of every reasoning model.

🗳️

Self-Consistency Improves Chain of Thought Reasoning

Wang et al. (Google) · arXiv Mar 2022 · ICLR 2023
📄 Accepted paper
🧒 In plain English

Ask the model to reason through the same problem many times, then take a vote on the final answers. Wrong paths tend to disagree with each other; right ones tend to agree.

▶ Many paths, one vote Press sample
🔍 What they found
  • Sample diverse chain-of-thought paths, then marginalise to the most consistent answer
  • Large gains: GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2% absolute (with PaLM 540B / GPT-3)
💡 Why it matters

The first test-time-compute trick: spend more samples, get more accuracy.

🦉

Minerva: Solving Quantitative Reasoning Problems with Language Models

Lewkowycz et al. (Google) · NeurIPS 2022
📄 Accepted paper
🧒 In plain English

Take PaLM and let it read a lot of maths and science — arXiv papers and web pages with equations kept intact. Without any calculator, it began solving competition-level maths problems step by step.

▶ Reading more maths Toggle training data
🔍 What they found
  • Continued training of PaLM on 118GB of scientific papers and maths web pages
  • 50.3% on the MATH benchmark with majority voting (previous best ~6.9%); strong results on MMLU-STEM and university-level problems
💡 Why it matters

Showed domain data plus chain-of-thought could unlock quantitative reasoning.

06 🦾 Reasoning Meets Action

Chain-of-thought + external actions + observations = the first agent loop.

🔁

ReAct: Synergizing Reasoning and Acting in Language Models

Yao et al. (Princeton, Google) · arXiv Oct 2022 · ICLR 2023
📄 Accepted paper
🧒 In plain English

Instead of only thinking, or only acting, the model alternates: think about what to do, take an action (like searching Wikipedia), read what comes back, think again. Every modern agent loop descends from this.

▶ Think → act → observe Press run
🔍 What they found
  • Interleaves reasoning traces with actions and observations
  • Reduces hallucination on HotpotQA and FEVER by grounding in Wikipedia lookups
  • ALFWorld and WebShop: +34% and +10% absolute success over imitation/RL baselines with only one or two examples
💡 Why it matters

The template for agent harnesses from LangChain to today's coding agents.

🤖

SayCan: Do As I Can, Not As I Say

Ahn et al. (Google, Everyday Robots) · CoRL 2022
📄 Accepted paper
🧒 In plain English

Ask a robot "I spilled my drink, can you help?" A language model knows what sounds useful (find a sponge); the robot knows what it can actually do from where it stands. SayCan multiplies the two scores to pick each next step.

▶ Useful × possible Press plan
🔍 What they found
  • Combines LLM task-grounding scores with learned value functions (affordances)
  • 84% planning success and 74% execution success on 101 real kitchen instructions
💡 Why it matters

Grounded language models in the physical world.

🦿

RT-1: Robotics Transformer for Real-World Control at Scale

Brohan et al. (Google) · arXiv Dec 2022 · RSS 2023
📄 Accepted paper
🧒 In plain English

Apply the foundation-model recipe to robots: collect a huge dataset of real robot demonstrations and train one Transformer on all of it. The result follows hundreds of instructions and copes better with new objects and settings.

▶ One model, hundreds of skills Press run
🔍 What they found
  • 130k episodes covering 700+ tasks, collected with 13 robots over 17 months
  • 97% success on seen instructions; better generalisation to new tasks, distractors and backgrounds than baselines
💡 Why it matters

Set up RT-2's vision-language-action models in 2023.

07 💻 Code Generation Becomes Serious Problem Solving

Generate → sample massively → execute and filter → select.

🏅

AlphaCode: Competition-Level Code Generation

Li et al. (DeepMind) · Science 2022
📰 Science
🧒 In plain English

Competitive programming needs real problem-solving, not autocomplete. AlphaCode writes up to a million candidate programs, throws out those that fail the example tests, groups the rest by behaviour and submits a handful — reaching about the median human competitor.

▶ Generate → filter → cluster → submit Press run
🔍 What they found
  • Massive sampling, filtering on example tests, and clustering by program behaviour
  • Average ranking in the top 54.3% in simulated Codeforces contests
💡 Why it matters

An early version of generate-and-verify — before interactive coding harnesses.

08 🎨 The Diffusion Generative-AI Explosion

Pixel diffusion → latent diffusion → language-conditioned diffusion → consumer generative AI.

🗜️

High-Resolution Image Synthesis with Latent Diffusion Models

Rombach et al. (LMU Munich, Runway) · CVPR 2022
📄 Accepted paper
🧒 In plain English

Diffusion models create images by removing noise step by step — slow and expensive on millions of pixels. Latent diffusion first compresses the image into a much smaller code, does the denoising there, then expands it back. Much cheaper. This became Stable Diffusion.

▶ Pixel space vs latent space Toggle
🔍 What they found
  • Autoencoder compresses images; diffusion runs in its latent space
  • Cross-attention conditioning on text and other inputs
  • Large compute savings at high resolution; basis of Stable Diffusion
💡 Why it matters

Put high-quality image generation on consumer GPUs.

🖌️

DALL·E 2: Hierarchical Text-Conditional Image Generation with CLIP Latents

Ramesh et al. (OpenAI) · arXiv Apr 2022
📝 Preprint / report
🧒 In plain English

DALL·E 2 works in two steps: first turn your text into an idea of an image (a CLIP image embedding), then paint that idea with a diffusion decoder. It produced strikingly realistic and creative pictures and brought text-to-image to the public.

▶ Text → image idea → image Press generate
🔍 What they found
  • A prior maps CLIP text embeddings to CLIP image embeddings; a diffusion decoder generates the image
  • Improved diversity with comparable photorealism; enabled variations and edits
💡 Why it matters

Brought generative AI into mainstream awareness in 2022.

📷

Imagen: Photorealistic Text-to-Image Diffusion with Deep Language Understanding

Saharia et al. (Google) · NeurIPS 2022
🏆 Outstanding Paper
🧒 In plain English

Imagen's surprise: to make better pictures from words, improve the part that reads the words. A big frozen text model (T5-XXL) mattered more than a bigger image model.

▶ Which part to scale? Toggle
🔍 What they found
  • Frozen T5-XXL text encoder + cascaded diffusion models
  • Zero-shot FID of 7.27 on COCO; human raters preferred it on DrawBench
  • Scaling the text encoder improved fidelity and alignment more than scaling the diffusion model
💡 Why it matters

Linked language understanding to image quality — a NeurIPS 2022 Outstanding Paper.

09 🦩 Multimodal Foundation Models

Vision encoder + language model + interleaved image/text → few-shot multimodal intelligence.

🦩

Flamingo: a Visual Language Model for Few-Shot Learning

Alayrac et al. (DeepMind) · NeurIPS 2022
📄 Accepted paper
🧒 In plain English

Show Flamingo a couple of examples mixing pictures and text — "[cat photo] this is a cat; [dog photo] this is a dog" — and it handles new images the same way. No retraining, just a few examples, like GPT-3 but with pictures and video.

▶ Few-shot with images Press run
🔍 What they found
  • Frozen vision encoder and frozen language model bridged by a Perceiver Resampler and gated cross-attention layers
  • 80B model set few-shot state of the art on many image and video benchmarks, sometimes beating fine-tuned models
💡 Why it matters

The template for multimodal assistants like GPT-4V and Gemini.

10 ⚡ Systems Engineering Becomes a Research Frontier

Algorithms + architecture + GPU-aware engineering, not just bigger parameter counts.

⚡

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Dao, Fu, Ermon, Rudra & Ré (Stanford) · NeurIPS 2022
📄 Accepted paper
🧒 In plain English

The slow part of attention isn't the maths — it's moving data between the GPU's big slow memory and its small fast memory. FlashAttention works on small tiles that fit in fast memory and never writes the huge attention matrix out. Same exact result, much faster, far less memory.

▶ Where the time goes Toggle
🔍 What they found
  • IO-aware exact attention with tiling and recomputation; memory linear in sequence length
  • 15% faster BERT-large training vs the MLPerf 1.1 record; 3× faster GPT-2; enabled longer contexts
💡 Why it matters

Now standard in virtually every LLM training and inference stack.

11 Chinchilla's Law, Computed

Chinchilla's fitted loss formula is simple enough to explore directly. Here it is applied to Gopher's compute budget: the same FLOPs spent on a smaller model and more data give a lower predicted loss, and the optimal split keeps shifting toward more tokens as budgets grow:

"""Chinchilla in a few lines: bigger model, or more data?

Hoffmann et al. (2022) fitted final pretraining loss as
    L(N, D) = E + A / N^alpha + B / D^beta
(N = parameters, D = training tokens) with E = 1.69, A = 406.4, B = 410.7, alpha = 0.34, beta = 0.28
("Approach 3"; this fit favours smaller models than the paper's other two
methods, and a 2024 replication by Besiroglu et al. refitted it). Training compute is about C = 6 N D FLOPs. For a fixed budget we try splits of C
between model size and data and find the loss-minimising one. Standard library only.
"""
E, A, B, ALPHA, BETA = 1.69, 406.4, 410.7, 0.34, 0.28

def loss(n, d):
    return E + A / n ** ALPHA + B / d ** BETA

def best_split(c):
    best = None
    for i in range(400):
        n = 10 ** (8 + i * 0.01)                   # 1e8 .. 1e12 parameters
        d = c / (6 * n)
        l = loss(n, d)
        if best is None or l < best[2]: best = (n, d, l)
    return best

gopher = (280e9, 300e9); chinchilla = (70e9, 1.4e12)
c = 6 * gopher[0] * gopher[1]
print(f"Same compute budget: {c:.2e} FLOPs")
print(f"   Gopher      280B params,  300B tokens -> predicted loss {loss(*gopher):.3f}")
print(f"   Chinchilla   70B params, 1.4T tokens -> predicted loss {loss(*chinchilla):.3f}")
n, d, l = best_split(c)
print(f"   Loss-optimal split under this fit: {n / 1e9:.0f}B params, {d / 1e12:.1f}T tokens, loss {l:.3f}")
print(f"\n{'compute (FLOPs)':>16} {'optimal params':>15} {'optimal tokens':>15} {'tokens/param':>13}")
for c in [1e21, 1e22, 1e23, 1e24, 1e25]:
    n, d, _ = best_split(c)
    print(f"{c:>16.0e} {n / 1e9:>14.1f}B {d / 1e9:>14.0f}B {d / n:>13.0f}")
print("\nParameters and tokens should grow together: Gopher was over-sized and under-trained.")

Output:

Same compute budget: 5.04e+23 FLOPs
   Gopher      280B params,  300B tokens -> predicted loss 1.993
   Chinchilla   70B params, 1.4T tokens -> predicted loss 1.937
   Loss-optimal split under this fit: 30B params, 2.8T tokens, loss 1.936

 compute (FLOPs)  optimal params  optimal tokens  tokens/param
           1e+21            1.8B             92B            50
           1e+22            5.1B            325B            63
           1e+23           14.5B           1153B            80
           1e+24           41.7B           3998B            96
           1e+25          117.5B          14186B           121

Parameters and tokens should grow together: Gopher was over-sized and under-trained.

The exact optimum depends on which of the paper's three fitting methods you use; this one ("Approach 3") favours smaller models, and later work refitted it. The direction is robust, though: Gopher was over-sized and under-trained, and later open models such as LLaMA (2023) went further, training small models on far more tokens because a smaller model is cheaper to serve.

12 The 2021 → 2026 Roadmap

From scaling to self-improving systems. Press play:

▶ Ten stages in five years Click a stage or press play
2022Teach models to reason, follow instructions, generate rich media and act
→
2023Turn them into programmable platforms
→
2024Build richer systems around them
→
2025Give them agency and scalable reasoning
→
2026Make them persistently self-improving
AI research in 2022, in one paragraph
2022 connected scale, alignment, reasoning, generative models and action. PaLM and Chinchilla made scaling both bigger and smarter. InstructGPT and Flan made models follow instructions — the product layer that became ChatGPT that November. Chain-of-thought, self-consistency and Minerva made reasoning visible and measurable. ReAct, SayCan and RT-1 connected language models to actions and robots. AlphaCode showed generate-and-verify could solve competition problems. Latent diffusion, DALL·E 2 and Imagen triggered the generative-image explosion. Flamingo and FlashAttention pointed toward multimodal assistants and GPU-aware engineering. Five of those ideas — compute-optimal scaling, chain-of-thought, RLHF, ReAct and diffusion — explain a surprisingly large share of everything that followed.

Next: AI Research in 2021: The Year the Foundation-Model Stack Emerged →

⚠️ Notes

About the source infographic and brief: (1) most items check out against primary sources (Chinchilla's 70B vs 280B at equal compute, InstructGPT's 1.3B preferred over 175B GPT-3, Flan's ~1.8K tasks, AlphaCode's top 54.3%, FlashAttention's speed-ups); (2) venues are added: Chinchilla and Imagen were NeurIPS 2022 Outstanding Papers (Chinchilla under the title "An empirical analysis of compute-optimal large language model training"), ReAct and Self-Consistency appeared at ICLR 2023, and Latent Diffusion at CVPR 2022 after a December 2021 preprint; (3) the infographic's small emergent-abilities chart implies smooth curves; the paper's claim is that some abilities appear abruptly with scale, later partly challenged by the 2023 "Mirage" paper; (4) Minerva's 50.3% on MATH uses majority voting over 64 samples. The Chinchilla calculation uses the paper's published "Approach 3" fit.

🔗 All References