Home › Blog › AI Research in 2024
AI Research Progress Through the Years · 2024 🧭

AI Research in 2024: The Year Context, Agents and World Models Took Shape

2024 was the year AI research moved from bigger foundation models to systems built around them. Context windows grew from thousands to millions of tokens. Models began to think before answering, grade themselves and spend extra compute on hard problems. Agents got purpose-built interfaces. Alternatives to the Transformer matured. Generated images became playable worlds, and AI-for-science produced results well beyond language. This guide walks through twenty-one of the year's most important papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.

How to read this: the third article in AI Research Progress Through the Years, and the prequel to AI Research in 2025 and 2026. Awards come from the official ICLR 2024, ICML 2024 and NeurIPS 2024 announcements. Papers are grouped by theme, not ranked. The "🧒 In plain English" boxes simplify on purpose, and the animations illustrate ideas; exact claims are in "What they found" and the linked papers.

01 The 2024 Awards at a Glance

🏆 Best / Outstanding Paper🏅 Honorable Mention📄 Accepted paper📝 Preprint / report📰 Nature
ConferenceCovered hereOther award winners
ICLR 2024 (May, Vienna)Outstanding: Vision Transformers Need Registers · Protein Discovery with Discrete Walk-Jump Sampling. Honorable Mentions: Model Tells You What to Discard · Proving Test Set Contamination5 Outstanding Papers and 11 Honorable Mentions in total
ICML 2024 (July, Vienna)Best Papers: Genie · Debating with More Persuasive LLMs · Stealing Part of a Production LM · Scaling Rectified Flow TransformersAlso VideoPoet · Discrete Diffusion via Ratios (SEDD) · a position paper on private learning with public pretraining · and others
NeurIPS 2024 (December, Vancouver)Best Papers: VAR · Stochastic Taylor Derivative Estimator. Datasets & Benchmarks: PRISMRunners-up: Not All Tokens Are What You Need (Rho-1) · Guiding a Diffusion Model with a Bad Version of Itself

02 Paper Map

Twenty-one papers in eight themes. Filter by award and click to jump:

▶ 2024 paper map Click to jump

03 📜 Long Context and Frontier Foundation Models

8K → 128K → 1M+ tokens: from prompt engineering to context engineering.

📚

Gemini 1.5: Multimodal Understanding Across Millions of Tokens

Google DeepMind · arXiv Mar 2024
📝 Preprint / report
🧒 In plain English

Imagine handing a model an entire bookshelf — or hours of video — at once, then asking about one sentence buried in the middle. Gemini 1.5 could read millions of tokens in one go and still find that one sentence almost every time. Long context stopped being a limit and became a tool.

▶ Needle in a haystack Hide one fact anywhere in a huge document
🔍 What they found
  • Mixture-of-experts model handling text, video and audio at up to millions of tokens
  • Near-perfect (>99%) needle-in-a-haystack retrieval reported up to multi-million-token contexts
  • Learned to translate a low-resource language (Kalamang) from a grammar book given in context
💡 Why it matters

Made long context an architectural resource, paving the way for agents that hold whole codebases or documents in mind.

🦙

The Llama 3 Herd of Models

Meta · arXiv Jul 2024
📝 Preprint / report
🧒 In plain English

A detailed public recipe for building a frontier model: how Meta trained Llama 3 — up to a 405-billion-parameter model — on over 15 trillion tokens, then taught it to code, reason, use tools and handle long documents, and released the weights.

▶ The open frontier recipe Key numbers
🔍 What they found
  • Dense 405B Transformer trained on ~15T tokens; 128K-token context
  • Multilingual, coding, reasoning and tool use via extensive post-training (SFT, rejection sampling, DPO)
  • Competitive with leading closed models on many benchmarks; weights released
💡 Why it matters

Showed how much of the frontier recipe could be documented and released openly.

📏

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Ding et al. (Microsoft) · ICML 2024
📄 Accepted paper
🧒 In plain English

Models track word positions with a kind of internal ruler. Trained on short texts, the ruler runs out. LongRoPE stretches it — unevenly, stretching some dimensions more than others (found by search) — so an existing model can read 2 million tokens with little extra training.

▶ Stretching the position ruler Drag the target context
🔍 What they found
  • Searches non-uniform rescaling of rotary position embeddings
  • Progressive extension: fine-tune to 256K, then interpolate to 2,048K with ≤1K fine-tuning steps
  • Readjusts to recover short-context performance
💡 Why it matters

The research-side method behind million-token context windows.

04 🤔 Reasoning Before the "Reasoning Model" Era

One-pass generation → think before answering → self-evaluate → spend more compute on hard problems.

💭

Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

Zelikman et al. (Stanford) · COLM 2024
📄 Accepted paper
🧒 In plain English

People often think before they speak, even in ordinary conversation. Quiet-STaR trains a model to generate a short hidden thought before predicting each next word of ordinary text, keeping thoughts that make its predictions better. It learns to think in general — not just on maths homework.

▶ Thinking between words Hidden rationales appear between tokens
🔍 What they found
  • Generates rationales in parallel at every token, mixing predictions with and without them
  • Zero-shot gains after continued pretraining: GSM8K 5.9% → 10.9%, CommonsenseQA 36.3% → 47.2%
💡 Why it matters

A direct precursor of 2025's reasoning models: reasoning learned from general text, not only curated problems.

🏅

Self-Rewarding Language Models

Yuan et al. (Meta) · ICML 2024
📄 Accepted paper
🧒 In plain English

Normally humans (or a separate reward model) grade a model's answers. Here the model grades itself: it writes several answers, judges them with a scoring rubric, then trains on its own preferences. Round after round, it gets better at both answering and judging.

▶ Three rounds of self-rewarding Answer → self-judge → train
🔍 What they found
  • LLM-as-a-Judge prompting provides rewards; iterative DPO on self-generated preference pairs
  • Llama 2 70B after three iterations beat Claude 2, Gemini Pro and GPT-4 0613 on AlpacaEval 2.0 (at the time)
  • Judging ability improves alongside instruction following
💡 Why it matters

An early self-improvement loop — and a warning that self-judgment needs outside checks.

⚖️

Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters

Snell, Lee, Xu & Kumar · arXiv Aug 2024 · ICLR 2025 Oral
📝 Preprint / report
🧒 In plain English

You have a fixed budget. Do you buy a bigger brain (a larger model) or give a smaller one more time to think (several attempts, checked by a verifier)? The answer depends on the question: for problems the small model can sometimes solve, extra thinking wins; for problems far beyond it, you need the bigger model.

▶ Bigger model or more thinking? Same compute budget
🔍 What they found
  • Two mechanisms: search against process reward models, and revising answers sequentially
  • Compute-optimal allocation by difficulty improves efficiency by >4× over best-of-N
  • On easy and medium problems, extra test-time compute can beat a ~14× larger model at matched FLOPs; hardest problems still favour pretraining
💡 Why it matters

Turned inference-time reasoning into a scaling dimension — the idea behind 2025's reasoning models.

05 💻 Agents and Software Engineering

Same model + purpose-built interface = a much stronger agent.

🧑‍💻

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

Yang, Jimenez et al. (Princeton) · NeurIPS 2024
📄 Accepted paper
🧒 In plain English

Give a model a raw computer terminal and it flounders. Give the same model tools designed for it — a file viewer that shows 100 lines at a time, an editor that checks syntax, a search command, clear error messages — and it fixes real GitHub issues far more often. The interface matters as much as the model.

▶ Same model, different interface Pick an interface and run
🔍 What they found
  • Designed commands for navigating repos, viewing and editing files with linting, and running tests
  • State of the art at the time: 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix, far above non-interactive baselines
💡 Why it matters

A conceptual ancestor of harness engineering (see Harness & Scaffolding Research).

06 🧱 Beyond Standard Transformers

Transformer → SSM/attention duality → ultra-low precision → smarter memory.

🐍

Transformers are SSMs (Mamba-2)

Dao & Gu · ICML 2024
📄 Accepted paper
🧒 In plain English

Transformers compare every word with every other word — accurate but slow for long texts. State-space models (SSMs) instead carry a running summary — fast, but different. This paper shows the two are mathematically two views of the same thing, and uses that to build Mamba-2, a faster SSM that still competes with Transformers.

▶ Two views of the same model Drag the sequence length
🔍 What they found
  • Structured state-space duality (SSD) links SSMs and attention through structured matrices
  • Mamba-2's core layer is 2–8× faster than Mamba's while remaining competitive with Transformers in language modelling
💡 Why it matters

Helped mature attention/SSM hybrids used in later production models.

🔢

BitNet b1.58: The Era of 1-bit LLMs

Ma et al. (Microsoft) · arXiv Feb 2024
📝 Preprint / report
🧒 In plain English

Model weights are usually precise decimals. BitNet b1.58 trains models whose weights are only −1, 0 or +1. Multiplying by those is almost free — mostly additions — so the model needs far less memory and energy, yet at 3 billion parameters it matches a normal model's quality.

▶ Full precision vs ternary weights Toggle
🔍 What they found
  • Ternary weights {−1, 0, 1} trained from scratch (not post-hoc quantisation)
  • Matches full-precision LLaMA perplexity from 3B parameters, while 2.71× faster and using 3.55× less GPU memory
💡 Why it matters

A route to much cheaper, lower-energy models — changing the economics, not just shrinking a trained model.

🗜️

Model Tells You What to Discard: Adaptive KV Cache Compression

Ge et al. · ICLR 2024
🏅 Honorable Mention
🧒 In plain English

While generating, a model keeps notes on everything it has read (the KV cache), and those notes eat GPU memory. Different attention heads care about different things — some only nearby words, some only punctuation. FastGen profiles each head and keeps only the notes that head actually uses.

▶ Keep only what each head needs Press compress
🔍 What they found
  • Lightweight profiling identifies each head's attention structure
  • Per-head policies (local, special tokens, frequent tokens, full) cut KV memory with negligible quality loss and no fine-tuning
💡 Why it matters

Attacks one of the central bottlenecks of long-context inference.

07 🌍 Generative Models and World Models

Generate images → generate video → generate interactive worlds agents can act in.

🖼️

Visual Autoregressive Modeling: Next-Scale Prediction (VAR)

Tian et al. · NeurIPS 2024
🏆 Best / Outstanding Paper
🧒 In plain English

Language models write one word at a time, left to right. Doing that with image patches, row by row, is awkward. VAR paints like an artist: first a tiny blurry thumbnail, then a sharper version, then sharper again — predicting each higher resolution from the last.

▶ Coarse to fine Press paint
🔍 What they found
  • Autoregression over scales ("next-scale prediction") instead of raster-order tokens
  • GPT-style models surpass diffusion transformers on ImageNet generation and show clear scaling laws
💡 Why it matters

Brought LLM-style scaling to image generation.

🌬️

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Esser et al. (Stability AI) · ICML 2024
🏆 Best / Outstanding Paper
🧒 In plain English

Image generators turn random noise into a picture along a path. Diffusion takes a winding path; rectified flow takes an almost straight line, so it needs fewer steps. Combined with a Transformer that lets text and image talk both ways, this became the basis of Stable Diffusion 3.

▶ Noise → image: winding vs straight Drag the number of steps
🔍 What they found
  • Improved noise sampling for training rectified flows
  • MM-DiT architecture with separate weights for text and image tokens and two-way information flow
  • Predictable scaling; outperforms prior models in human preference
💡 Why it matters

The architecture behind a generation of text-to-image models.

🎮

Genie: Generative Interactive Environments

Bruce et al. (Google DeepMind) · ICML 2024
🏆 Best / Outstanding Paper
🧒 In plain English

Genie watched tens of thousands of hours of internet videos of 2D platform games — with no controller data — and learned what actions are possible. Give it a single image (even a sketch) and it becomes a playable world: press a button and it generates the next frame.

▶ A world you can play Use the buttons
🔍 What they found
  • 11B-parameter foundation world model: video tokenizer, latent action model and dynamics model
  • Learns a discrete latent action space unsupervised from unlabelled video
  • Generates controllable worlds from images, sketches or photos
💡 Why it matters

Connects generative AI to environments where future agents can act and learn.

08 🛡️ Alignment, Security and Evaluation

Better models raised a new question: can we still trust benchmarks, alignment and oversight?

🗣️

Debating with More Persuasive LLMs Leads to More Truthful Answers

Khan et al. · ICML 2024
🏆 Best / Outstanding Paper
🧒 In plain English

How can a non-expert check an expert? Let two strong models argue opposite answers while a weaker judge — a smaller model or a person without the source text — decides. Truth tends to be easier to defend, so judges pick the right answer more often.

▶ Debate vs asking one model Reported judge accuracy
🔍 What they found
  • Debate helped non-expert judges answer reading-comprehension questions: 76% (models) and 88% (humans) vs naive baselines of 48% and 60%
  • Optimising debaters for persuasiveness improved judges' accuracy further
💡 Why it matters

Early evidence for scalable oversight — supervising systems stronger than the supervisor.

🕵️

Stealing Part of a Production Language Model

Carlini et al. · ICML 2024
🏆 Best / Outstanding Paper
🧒 In plain English

Companies hide their models behind an API. But each answer's scores lie in a space whose size equals a secret internal dimension of the model. By sending many queries and analysing the outputs, the authors recovered that hidden size — and the model's final layer — for real production models.

▶ Finding the hidden dimension Press query
🔍 What they found
  • For under $20, extracted the full final projection matrix of OpenAI's Ada and Babbage models, revealing hidden dimensions of 1024 and 2048
  • Also recovered gpt-3.5-turbo's hidden dimension; providers changed their APIs in response
💡 Why it matters

Showed deployment creates new security surfaces.

🌐

The PRISM Alignment Dataset

Kirk et al. · NeurIPS 2024 Datasets & Benchmarks
🏆 Best D&B Paper
🧒 In plain English

Whose preferences should AI be aligned to? PRISM asked about 1,500 people from 75 countries to chat with 21 models and rate the answers — capturing how different people want different things, rather than averaging everyone into one "human preference".

▶ Same answer, different ratings Pick a question
🔍 What they found
  • 1,500 participants from 75 countries; 8,011 live conversations with 21 LLMs, linked to sociodemographics and stated preferences
  • Preferences vary strongly by person and topic, especially on value-laden questions
💡 Why it matters

Pushed alignment toward pluralistic, individualised preferences.

🧪

Proving Test Set Contamination in Black-Box Language Models

Oren et al. · ICLR 2024
🏅 Honorable Mention
🧒 In plain English

Did a model see a benchmark during training? A clever test: if a model memorised a dataset, it will find the dataset's original order of examples more likely than shuffled orders. Compare the original order against many shuffles — no access to training data needed.

▶ Original order vs shuffles Press test
🔍 What they found
  • Exchangeability test gives provable false-positive guarantees
  • Works on black-box models with only log-probabilities
💡 Why it matters

Benchmark trust became a research problem as competition intensified.

09 🧬 AI for Science

AI predicts structures → designs biological candidates → meets real experiments.

🧬

AlphaFold 3: Structure Prediction of Biomolecular Interactions

Abramson et al. (Google DeepMind, Isomorphic Labs) · Nature 2024
📰 Nature
🧒 In plain English

Life's machinery is molecules fitting together: proteins binding DNA, drugs slotting into proteins. AlphaFold 3 predicts the 3D shape of these whole complexes — proteins, DNA, RNA, small molecules, ions — starting from noise and sharpening it into a structure with a diffusion model.

▶ Assembling a complex Press fold
🔍 What they found
  • Diffusion-based architecture replacing much of AlphaFold 2's structure module
  • Substantially better accuracy than specialised tools for protein–ligand, protein–nucleic acid and antibody–antigen interactions
💡 Why it matters

One of the most consequential AI-for-science results; its creators shared the 2024 Chemistry Nobel for AlphaFold.

🧫

Protein Discovery with Discrete Walk-Jump Sampling

Frey et al. (Prescient Design, Genentech) · ICLR 2024
🏆 Best / Outstanding Paper
🧒 In plain English

Designing new antibodies means searching an enormous space of sequences. The trick: walk through a smoothed, noisy version of that space where moves are easy, then jump back to a clean, realistic sequence. The designs were then made and tested in a real lab.

▶ Walk in the fog, jump to solid ground Press sample
🔍 What they found
  • Trains a score model on noised data; samples with Langevin "walks" then one-step denoising "jumps"
  • In the wet lab, 97–100% of generated antibodies were expressed and purified, and 70% of functional designs bound as well as or better than known antibodies — on the first attempt
💡 Why it matters

AI generation connected to real experiments — a step toward AI-driven discovery.

📐

Stochastic Taylor Derivative Estimator (STDE)

Shi et al. · NeurIPS 2024
🏆 Best / Outstanding Paper
🧒 In plain English

Physics-informed neural networks must compute high-order derivatives (for equations like heat flow). Exact computation gets explosively expensive as dimensions and derivative order grow. STDE estimates them with cheap random samples instead, making huge problems tractable.

▶ Cost of higher-order derivatives Drag dimensions
🔍 What they found
  • Randomised Taylor-mode automatic differentiation for arbitrary differential operators
  • Solved million-dimensional PDEs in minutes on a single GPU; >1000× speed-up and >30× memory reduction reported
💡 Why it matters

Scales scientific machine learning to problems previously out of reach.

10 👁️ Vision and Multimodal Representation

Image model → multimodal → video → world representation.

🗃️

Vision Transformers Need Registers

Darcet, Oquab, Mairal & Bojanowski (Meta, Inria) · ICLR 2024
🏆 Best / Outstanding Paper
🧒 In plain English

Look at what a vision model attends to and you see bright glitches in blank areas — sky, walls. The model is using those empty patches as scratch paper. Give it a few extra blank tokens (registers) to scribble on, and the glitches vanish.

▶ Attention map of an image Toggle registers
🔍 What they found
  • High-norm artefact tokens appear in low-information regions of large ViTs (supervised and self-supervised)
  • Adding register tokens removes them, giving smoother attention maps and better dense-prediction and object-discovery results
💡 Why it matters

A tiny fix to a widely used architecture, with outsized impact.

Side note: SAM 2 (Meta, 2024) extended promptable segmentation from images to video. It was released in 2024, but its award (ICLR 2025 Honorable Mention) came the next year, so it's covered in AI Research in 2025.

11 Bigger Model or More Thinking?

Snell et al.'s question, in miniature. A fixed budget can buy one answer from a model 14× larger, or 14 answers from a small model with a verifier picking the best. Which wins depends on difficulty:

"""Spend compute on a bigger model, or on thinking longer? A toy version of Snell et al. (2024).

Same inference budget, two options:
  big model   - a model 14x larger answers once (cost 14 units)
  small + TTC - the small model samples N = 14 answers (1 unit each) and a verifier picks one;
                the verifier finds a correct answer 90% of the time when at least one exists.
Per-sample accuracies by problem difficulty are illustrative assumptions.
"""
SMALL = {"easy": 0.60, "medium": 0.25, "hard": 0.005}
BIG = {"easy": 0.80, "medium": 0.50, "hard": 0.20}
VERIFIER = 0.90

def best_of_n(p, n):
    return VERIFIER * (1 - (1 - p) ** n)

print("Accuracy at equal inference compute (14 units)\n")
print(f"   {'difficulty':<10} {'14x bigger model':>17} {'small, best-of-14':>18}   winner")
for d in SMALL:
    b, s = BIG[d], best_of_n(SMALL[d], 14)
    print(f"   {d:<10} {b:>17.0%} {s:>18.0%}   {'test-time compute' if s > b else 'bigger model'}")
print("\nHow many samples does the small model need to match the big one?")
for d in SMALL:
    n = next(n for n in range(1, 5000) if best_of_n(SMALL[d], n) >= BIG[d])
    print(f"   {d:<8} {n:>3} samples ({n / 14:.1f}x the big model's cost)")
print("\nExtra thinking wins when the model can sometimes get it right; otherwise you need a better model.")

Output:

Accuracy at equal inference compute (14 units)

   difficulty  14x bigger model  small, best-of-14   winner
   easy                     80%                90%   test-time compute
   medium                   50%                88%   test-time compute
   hard                     20%                 6%   bigger model

How many samples does the small model need to match the big one?
   easy       3 samples (0.2x the big model's cost)
   medium     3 samples (0.2x the big model's cost)
   hard      51 samples (3.6x the big model's cost)

Extra thinking wins when the model can sometimes get it right; otherwise you need a better model.

The interactive version is in the Snell et al. card above. The lesson became 2025's reasoning playbook: spend extra inference compute where the model is sometimes right, and invest in a better model where it is almost never right.

12 The 2023 → 2026 Roadmap

From foundation models to self-improving systems. Press play:

▶ Six stages in three years Click a stage or press play
2024Build the system around the model
→
2025Give the system reasoning and agency
→
2026Make the system increasingly able to improve itself
AI research in 2024, in one paragraph
2024 was the transition from foundation models to systems. Long context became practical (Gemini 1.5, LongRoPE) and open frontier recipes were published (Llama 3). Reasoning arrived before the "reasoning model" name did: models learned to think silently (Quiet-STaR), reward themselves (Self-Rewarding LMs) and spend extra inference compute wisely (Snell et al.). Agents gained purpose-built interfaces (SWE-agent). Alternatives and upgrades to the Transformer matured: state-space duality, 1.58-bit weights, smarter KV caches. Generative models climbed from images (VAR, rectified flow) to playable worlds (Genie). Better models raised new trust problems — debate for oversight, model stealing, pluralistic preferences, benchmark contamination. AI for science went well beyond language, from AlphaFold 3 to wet-lab-validated antibodies. The system was being built around the model, ready for 2025's reasoning and agency.

Next: AI Research in 2023: The Year Foundation Models Became Platforms →

⚠️ Notes

Corrections to the source infographic and brief: (1) Mamba-2 ("Transformers are SSMs") is labelled NeurIPS 2024 but was published at ICML 2024; (2) Genie is labelled ICLR 2024 but was an ICML 2024 Best Paper; (3) venues added for papers shown as "arXiv 2024": Quiet-STaR (COLM 2024), Self-Rewarding LMs (ICML 2024), LongRoPE (ICML 2024), SWE-agent (NeurIPS 2024), and Snell et al. (ICLR 2025 Oral); (4) Gemini 1.5, Llama 3 and BitNet b1.58 are technical reports or preprints rather than peer-reviewed conference papers; (5) SAM 2's award belongs to the 2025 cycle, as the brief notes. Everything else checked out: VAR, STDE and PRISM at NeurIPS 2024; Debating, Stealing, Rectified Flow and Genie as ICML 2024 Best Papers; ViT Registers and Walk-Jump as ICLR 2024 Outstanding; KV-cache compression and contamination as Honorable Mentions; AlphaFold 3 in Nature. The simulation is a toy model with made-up accuracies.

🔗 All References