AI Research in 2024: The Year Context, Agents and World Models Took Shape
2024 was the year AI research moved from bigger foundation models to systems built around them. Context windows grew from thousands to millions of tokens. Models began to think before answering, grade themselves and spend extra compute on hard problems. Agents got purpose-built interfaces. Alternatives to the Transformer matured. Generated images became playable worlds, and AI-for-science produced results well beyond language. This guide walks through twenty-one of the year's most important papers in eight research themes, each explained in plain English with an animation, followed by what it found, why it matters and its sources.
01 The 2024 Awards at a Glance
| Conference | Covered here | Other award winners |
|---|---|---|
| ICLR 2024 (May, Vienna) | Outstanding: Vision Transformers Need Registers · Protein Discovery with Discrete Walk-Jump Sampling. Honorable Mentions: Model Tells You What to Discard · Proving Test Set Contamination | 5 Outstanding Papers and 11 Honorable Mentions in total |
| ICML 2024 (July, Vienna) | Best Papers: Genie · Debating with More Persuasive LLMs · Stealing Part of a Production LM · Scaling Rectified Flow Transformers | Also VideoPoet · Discrete Diffusion via Ratios (SEDD) · a position paper on private learning with public pretraining · and others |
| NeurIPS 2024 (December, Vancouver) | Best Papers: VAR · Stochastic Taylor Derivative Estimator. Datasets & Benchmarks: PRISM | Runners-up: Not All Tokens Are What You Need (Rho-1) · Guiding a Diffusion Model with a Bad Version of Itself |
02 Paper Map
Twenty-one papers in eight themes. Filter by award and click to jump:
03 📜 Long Context and Frontier Foundation Models
8K → 128K → 1M+ tokens: from prompt engineering to context engineering.
Gemini 1.5: Multimodal Understanding Across Millions of Tokens
Google DeepMind · arXiv Mar 2024Imagine handing a model an entire bookshelf — or hours of video — at once, then asking about one sentence buried in the middle. Gemini 1.5 could read millions of tokens in one go and still find that one sentence almost every time. Long context stopped being a limit and became a tool.
- Mixture-of-experts model handling text, video and audio at up to millions of tokens
- Near-perfect (>99%) needle-in-a-haystack retrieval reported up to multi-million-token contexts
- Learned to translate a low-resource language (Kalamang) from a grammar book given in context
Made long context an architectural resource, paving the way for agents that hold whole codebases or documents in mind.
The Llama 3 Herd of Models
Meta · arXiv Jul 2024A detailed public recipe for building a frontier model: how Meta trained Llama 3 — up to a 405-billion-parameter model — on over 15 trillion tokens, then taught it to code, reason, use tools and handle long documents, and released the weights.
- Dense 405B Transformer trained on ~15T tokens; 128K-token context
- Multilingual, coding, reasoning and tool use via extensive post-training (SFT, rejection sampling, DPO)
- Competitive with leading closed models on many benchmarks; weights released
Showed how much of the frontier recipe could be documented and released openly.
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Ding et al. (Microsoft) · ICML 2024Models track word positions with a kind of internal ruler. Trained on short texts, the ruler runs out. LongRoPE stretches it — unevenly, stretching some dimensions more than others (found by search) — so an existing model can read 2 million tokens with little extra training.
- Searches non-uniform rescaling of rotary position embeddings
- Progressive extension: fine-tune to 256K, then interpolate to 2,048K with ≤1K fine-tuning steps
- Readjusts to recover short-context performance
The research-side method behind million-token context windows.
04 🤔 Reasoning Before the "Reasoning Model" Era
One-pass generation → think before answering → self-evaluate → spend more compute on hard problems.
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
Zelikman et al. (Stanford) · COLM 2024People often think before they speak, even in ordinary conversation. Quiet-STaR trains a model to generate a short hidden thought before predicting each next word of ordinary text, keeping thoughts that make its predictions better. It learns to think in general — not just on maths homework.
- Generates rationales in parallel at every token, mixing predictions with and without them
- Zero-shot gains after continued pretraining: GSM8K 5.9% → 10.9%, CommonsenseQA 36.3% → 47.2%
A direct precursor of 2025's reasoning models: reasoning learned from general text, not only curated problems.
Self-Rewarding Language Models
Yuan et al. (Meta) · ICML 2024Normally humans (or a separate reward model) grade a model's answers. Here the model grades itself: it writes several answers, judges them with a scoring rubric, then trains on its own preferences. Round after round, it gets better at both answering and judging.
- LLM-as-a-Judge prompting provides rewards; iterative DPO on self-generated preference pairs
- Llama 2 70B after three iterations beat Claude 2, Gemini Pro and GPT-4 0613 on AlpacaEval 2.0 (at the time)
- Judging ability improves alongside instruction following
An early self-improvement loop — and a warning that self-judgment needs outside checks.
Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters
Snell, Lee, Xu & Kumar · arXiv Aug 2024 · ICLR 2025 OralYou have a fixed budget. Do you buy a bigger brain (a larger model) or give a smaller one more time to think (several attempts, checked by a verifier)? The answer depends on the question: for problems the small model can sometimes solve, extra thinking wins; for problems far beyond it, you need the bigger model.
- Two mechanisms: search against process reward models, and revising answers sequentially
- Compute-optimal allocation by difficulty improves efficiency by >4× over best-of-N
- On easy and medium problems, extra test-time compute can beat a ~14× larger model at matched FLOPs; hardest problems still favour pretraining
Turned inference-time reasoning into a scaling dimension — the idea behind 2025's reasoning models.
05 💻 Agents and Software Engineering
Same model + purpose-built interface = a much stronger agent.
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, Jimenez et al. (Princeton) · NeurIPS 2024Give a model a raw computer terminal and it flounders. Give the same model tools designed for it — a file viewer that shows 100 lines at a time, an editor that checks syntax, a search command, clear error messages — and it fixes real GitHub issues far more often. The interface matters as much as the model.
- Designed commands for navigating repos, viewing and editing files with linting, and running tests
- State of the art at the time: 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix, far above non-interactive baselines
A conceptual ancestor of harness engineering (see Harness & Scaffolding Research).
06 🧱 Beyond Standard Transformers
Transformer → SSM/attention duality → ultra-low precision → smarter memory.
Transformers are SSMs (Mamba-2)
Dao & Gu · ICML 2024Transformers compare every word with every other word — accurate but slow for long texts. State-space models (SSMs) instead carry a running summary — fast, but different. This paper shows the two are mathematically two views of the same thing, and uses that to build Mamba-2, a faster SSM that still competes with Transformers.
- Structured state-space duality (SSD) links SSMs and attention through structured matrices
- Mamba-2's core layer is 2–8× faster than Mamba's while remaining competitive with Transformers in language modelling
Helped mature attention/SSM hybrids used in later production models.
BitNet b1.58: The Era of 1-bit LLMs
Ma et al. (Microsoft) · arXiv Feb 2024Model weights are usually precise decimals. BitNet b1.58 trains models whose weights are only −1, 0 or +1. Multiplying by those is almost free — mostly additions — so the model needs far less memory and energy, yet at 3 billion parameters it matches a normal model's quality.
- Ternary weights {−1, 0, 1} trained from scratch (not post-hoc quantisation)
- Matches full-precision LLaMA perplexity from 3B parameters, while 2.71× faster and using 3.55× less GPU memory
A route to much cheaper, lower-energy models — changing the economics, not just shrinking a trained model.
Model Tells You What to Discard: Adaptive KV Cache Compression
Ge et al. · ICLR 2024While generating, a model keeps notes on everything it has read (the KV cache), and those notes eat GPU memory. Different attention heads care about different things — some only nearby words, some only punctuation. FastGen profiles each head and keeps only the notes that head actually uses.
- Lightweight profiling identifies each head's attention structure
- Per-head policies (local, special tokens, frequent tokens, full) cut KV memory with negligible quality loss and no fine-tuning
Attacks one of the central bottlenecks of long-context inference.
07 🌍 Generative Models and World Models
Generate images → generate video → generate interactive worlds agents can act in.
Visual Autoregressive Modeling: Next-Scale Prediction (VAR)
Tian et al. · NeurIPS 2024Language models write one word at a time, left to right. Doing that with image patches, row by row, is awkward. VAR paints like an artist: first a tiny blurry thumbnail, then a sharper version, then sharper again — predicting each higher resolution from the last.
- Autoregression over scales ("next-scale prediction") instead of raster-order tokens
- GPT-style models surpass diffusion transformers on ImageNet generation and show clear scaling laws
Brought LLM-style scaling to image generation.
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Esser et al. (Stability AI) · ICML 2024Image generators turn random noise into a picture along a path. Diffusion takes a winding path; rectified flow takes an almost straight line, so it needs fewer steps. Combined with a Transformer that lets text and image talk both ways, this became the basis of Stable Diffusion 3.
- Improved noise sampling for training rectified flows
- MM-DiT architecture with separate weights for text and image tokens and two-way information flow
- Predictable scaling; outperforms prior models in human preference
The architecture behind a generation of text-to-image models.
Genie: Generative Interactive Environments
Bruce et al. (Google DeepMind) · ICML 2024Genie watched tens of thousands of hours of internet videos of 2D platform games — with no controller data — and learned what actions are possible. Give it a single image (even a sketch) and it becomes a playable world: press a button and it generates the next frame.
- 11B-parameter foundation world model: video tokenizer, latent action model and dynamics model
- Learns a discrete latent action space unsupervised from unlabelled video
- Generates controllable worlds from images, sketches or photos
Connects generative AI to environments where future agents can act and learn.
08 🛡️ Alignment, Security and Evaluation
Better models raised a new question: can we still trust benchmarks, alignment and oversight?
Debating with More Persuasive LLMs Leads to More Truthful Answers
Khan et al. · ICML 2024How can a non-expert check an expert? Let two strong models argue opposite answers while a weaker judge — a smaller model or a person without the source text — decides. Truth tends to be easier to defend, so judges pick the right answer more often.
- Debate helped non-expert judges answer reading-comprehension questions: 76% (models) and 88% (humans) vs naive baselines of 48% and 60%
- Optimising debaters for persuasiveness improved judges' accuracy further
Early evidence for scalable oversight — supervising systems stronger than the supervisor.
Stealing Part of a Production Language Model
Carlini et al. · ICML 2024Companies hide their models behind an API. But each answer's scores lie in a space whose size equals a secret internal dimension of the model. By sending many queries and analysing the outputs, the authors recovered that hidden size — and the model's final layer — for real production models.
- For under $20, extracted the full final projection matrix of OpenAI's Ada and Babbage models, revealing hidden dimensions of 1024 and 2048
- Also recovered gpt-3.5-turbo's hidden dimension; providers changed their APIs in response
Showed deployment creates new security surfaces.
The PRISM Alignment Dataset
Kirk et al. · NeurIPS 2024 Datasets & BenchmarksWhose preferences should AI be aligned to? PRISM asked about 1,500 people from 75 countries to chat with 21 models and rate the answers — capturing how different people want different things, rather than averaging everyone into one "human preference".
- 1,500 participants from 75 countries; 8,011 live conversations with 21 LLMs, linked to sociodemographics and stated preferences
- Preferences vary strongly by person and topic, especially on value-laden questions
Pushed alignment toward pluralistic, individualised preferences.
Proving Test Set Contamination in Black-Box Language Models
Oren et al. · ICLR 2024Did a model see a benchmark during training? A clever test: if a model memorised a dataset, it will find the dataset's original order of examples more likely than shuffled orders. Compare the original order against many shuffles — no access to training data needed.
- Exchangeability test gives provable false-positive guarantees
- Works on black-box models with only log-probabilities
Benchmark trust became a research problem as competition intensified.
09 🧬 AI for Science
AI predicts structures → designs biological candidates → meets real experiments.
AlphaFold 3: Structure Prediction of Biomolecular Interactions
Abramson et al. (Google DeepMind, Isomorphic Labs) · Nature 2024Life's machinery is molecules fitting together: proteins binding DNA, drugs slotting into proteins. AlphaFold 3 predicts the 3D shape of these whole complexes — proteins, DNA, RNA, small molecules, ions — starting from noise and sharpening it into a structure with a diffusion model.
- Diffusion-based architecture replacing much of AlphaFold 2's structure module
- Substantially better accuracy than specialised tools for protein–ligand, protein–nucleic acid and antibody–antigen interactions
One of the most consequential AI-for-science results; its creators shared the 2024 Chemistry Nobel for AlphaFold.
Protein Discovery with Discrete Walk-Jump Sampling
Frey et al. (Prescient Design, Genentech) · ICLR 2024Designing new antibodies means searching an enormous space of sequences. The trick: walk through a smoothed, noisy version of that space where moves are easy, then jump back to a clean, realistic sequence. The designs were then made and tested in a real lab.
- Trains a score model on noised data; samples with Langevin "walks" then one-step denoising "jumps"
- In the wet lab, 97–100% of generated antibodies were expressed and purified, and 70% of functional designs bound as well as or better than known antibodies — on the first attempt
AI generation connected to real experiments — a step toward AI-driven discovery.
Stochastic Taylor Derivative Estimator (STDE)
Shi et al. · NeurIPS 2024Physics-informed neural networks must compute high-order derivatives (for equations like heat flow). Exact computation gets explosively expensive as dimensions and derivative order grow. STDE estimates them with cheap random samples instead, making huge problems tractable.
- Randomised Taylor-mode automatic differentiation for arbitrary differential operators
- Solved million-dimensional PDEs in minutes on a single GPU; >1000× speed-up and >30× memory reduction reported
Scales scientific machine learning to problems previously out of reach.
10 👁️ Vision and Multimodal Representation
Image model → multimodal → video → world representation.
Vision Transformers Need Registers
Darcet, Oquab, Mairal & Bojanowski (Meta, Inria) · ICLR 2024Look at what a vision model attends to and you see bright glitches in blank areas — sky, walls. The model is using those empty patches as scratch paper. Give it a few extra blank tokens (registers) to scribble on, and the glitches vanish.
- High-norm artefact tokens appear in low-information regions of large ViTs (supervised and self-supervised)
- Adding register tokens removes them, giving smoother attention maps and better dense-prediction and object-discovery results
A tiny fix to a widely used architecture, with outsized impact.
Side note: SAM 2 (Meta, 2024) extended promptable segmentation from images to video. It was released in 2024, but its award (ICLR 2025 Honorable Mention) came the next year, so it's covered in AI Research in 2025.
11 Bigger Model or More Thinking?
Snell et al.'s question, in miniature. A fixed budget can buy one answer from a model 14× larger, or 14 answers from a small model with a verifier picking the best. Which wins depends on difficulty:
"""Spend compute on a bigger model, or on thinking longer? A toy version of Snell et al. (2024).
Same inference budget, two options:
big model - a model 14x larger answers once (cost 14 units)
small + TTC - the small model samples N = 14 answers (1 unit each) and a verifier picks one;
the verifier finds a correct answer 90% of the time when at least one exists.
Per-sample accuracies by problem difficulty are illustrative assumptions.
"""
SMALL = {"easy": 0.60, "medium": 0.25, "hard": 0.005}
BIG = {"easy": 0.80, "medium": 0.50, "hard": 0.20}
VERIFIER = 0.90
def best_of_n(p, n):
return VERIFIER * (1 - (1 - p) ** n)
print("Accuracy at equal inference compute (14 units)\n")
print(f" {'difficulty':<10} {'14x bigger model':>17} {'small, best-of-14':>18} winner")
for d in SMALL:
b, s = BIG[d], best_of_n(SMALL[d], 14)
print(f" {d:<10} {b:>17.0%} {s:>18.0%} {'test-time compute' if s > b else 'bigger model'}")
print("\nHow many samples does the small model need to match the big one?")
for d in SMALL:
n = next(n for n in range(1, 5000) if best_of_n(SMALL[d], n) >= BIG[d])
print(f" {d:<8} {n:>3} samples ({n / 14:.1f}x the big model's cost)")
print("\nExtra thinking wins when the model can sometimes get it right; otherwise you need a better model.")Output:
Accuracy at equal inference compute (14 units) difficulty 14x bigger model small, best-of-14 winner easy 80% 90% test-time compute medium 50% 88% test-time compute hard 20% 6% bigger model How many samples does the small model need to match the big one? easy 3 samples (0.2x the big model's cost) medium 3 samples (0.2x the big model's cost) hard 51 samples (3.6x the big model's cost) Extra thinking wins when the model can sometimes get it right; otherwise you need a better model.
The interactive version is in the Snell et al. card above. The lesson became 2025's reasoning playbook: spend extra inference compute where the model is sometimes right, and invest in a better model where it is almost never right.
12 The 2023 → 2026 Roadmap
From foundation models to self-improving systems. Press play:
Next: AI Research in 2023: The Year Foundation Models Became Platforms →