Standalone Deep Dive · Sequel to "The Evolution of Neural Networks" 🔭

After Transformers: What's Actually Working, What's a Dead End, and Where the Next Breakthrough Might Come From

The Evolution article ended in 2026 with the chain still moving. This one picks up exactly there: what today's dominant methods actually are and where they genuinely break, which past ideas the field tried and walked away from, where serious research money and talent are betting right now, and — most importantly — a concrete framework for judging any new idea yourself, so you're not just consuming this list but building the skill that produced it.

FrontierAGI Team

September 12, 2026 · 90 min read

Opening What's Active Dead Ends Open Bets Who's Betting Where Judging a New Idea Self-Check What's Uncertain References Videos Verdict

The Chain Doesn't Stop at "Now"

The Evolution of Neural Networks article told a chronological story ending at RLHF and ChatGPT — but treating "now" as an endpoint is exactly the mistake that article's whole pattern warns against. Every era in that story looked, to the people living through it, like the current state of the art, right up until its specific limitation became obvious enough that someone fixed it. This article is written from inside that same live moment: not as a settled history, but as an active, contested, ongoing argument about what to build next.

Three honest caveats before starting. First, "current" here means as of September 2026 — parts of this will look dated within a year or two, which is itself the whole point of the article. Second, this is necessarily more debatable than the Evolution article's history — where that piece described settled facts, this one describes live research bets, and reasonable researchers disagree about several calls made here. Third, "dead end" doesn't always mean "wrong forever" — several ideas below were abandoned for lack of compute or data at the time, and this site's own history shows that's exactly the condition that sometimes reverses.

Reading this article passively teaches you what the field currently believes. Reading it while asking "why do they believe that, and what would change their mind" teaches you how to do research.

Part 1 — What's Actually Dominant Right Now

Five Methods Doing Most of the Real Work Today

These aren't equally important, and they're not independent — modern frontier systems combine several of them at once. Each entry states the honest limitation directly, not as a footnote.

Active — the backboneDecoder-Only Transformers, Scaled Up

The architecture from the Evolution article's Era 11–12, still underneath essentially every frontier model referenced across this site. The core recipe — predict the next token, at massive scale, on massive data — remains the base layer everything else here is built on top of.

The real limitation: self-attention's compute cost grows quadratically with sequence length (covered in Article 3's linear algebra section) — meaning doubling context length roughly quadruples the attention computation, a genuine mathematical ceiling, not an engineering inconvenience. Chinchilla's 2022 finding (Article 1, Article 3) also revealed that raw scaling alone was leaving performance on the table due to data-to-parameter imbalance — "just make it bigger" was already shown to be an incomplete strategy four years ago.

Active — the alignment defaultRLHF and RLAIF (Constitutional AI)

Covered in the Evolution article's Era 13: human (or, in Anthropic's Constitutional AI, AI-assisted) preference feedback shapes model behavior after pretraining. This remains the standard way frontier labs turn a raw next-token predictor into something that behaves like a helpful assistant.

The real limitation: reward hacking — a model optimizing hard against a learned reward signal can find ways to score well on the proxy (the reward model's judgment) without actually satisfying the real underlying goal, a problem formally named years earlier in Amodei et al.'s "Concrete Problems in AI Safety" (2016). RLHF also inherits whatever biases or blind spots exist in its human raters, at scale.

Active — the scaling workaroundMixture-of-Experts (MoE)

Introduced in Article 1's Switch Transformer paper: instead of every parameter processing every input, a "router" sends each token to only a handful of specialized sub-networks ("experts") out of many available. This lets total parameter count grow far larger than compute cost, since most experts sit idle for any given token. Mixtral (Jiang et al., arXiv:2401.04088) is a widely cited open example.

The real limitation: routing instability — the router itself has to learn which expert should handle which token, and this learned assignment can collapse (a few experts get overused while others go untrained) or behave unpredictably near training-data distribution shifts, an active engineering headache distinct from the core modeling problem.

Active — patches the knowledge ceilingRetrieval-Augmented Generation (RAG)

Lewis et al.'s 2020 RAG paper: instead of relying purely on facts memorized during pretraining, the model retrieves relevant documents from an external source at inference time and conditions its answer on them — directly addressing the "knowledge cutoff" and hallucination-on-obscure-facts problems.

The real limitation: RAG is only as good as the retrieval step — if the wrong documents are retrieved, or the query itself is ambiguous, the model confidently generates from irrelevant context, which is a different failure mode than hallucination but not obviously a better one from a user's perspective.

Active — the newest entrantTest-Time Compute / Reasoning Models

Rather than producing an answer in one forward pass, models like OpenAI's o1-family generate an extended internal chain of reasoning before answering, effectively trading more compute at inference time for better accuracy on hard problems — a genuinely new axis of scaling beyond "bigger model" or "more training data." Lightman et al.'s "Let's Verify Step by Step" (2023) is a key process-reward-modeling paper underpinning this direction.

The real limitation: cost-per-query rises substantially, since generating and evaluating a long reasoning chain is expensive — meaning this approach is currently reserved for problems that justify the cost, not a free upgrade to every query, and its benefits are least clear on tasks that don't actually require multi-step reasoning.

Part 2 — The Graveyard: Ideas the Field Walked Away From

Real Dead Ends — Treated Honestly, Not Dismissively

Calling something a "dead end" requires care: some of these were wrong ideas, and some were right ideas at the wrong moment (echoing the Evolution article's Eras 4–7). Both categories are useful to know as a researcher, for different reasons.

Largely abandonedHand-Built Symbolic Knowledge Bases (Pre-Neural GOFAI)

Projects like Cyc, built on manually encoding millions of common-sense facts and logical rules by hand, represented an entire pre-2012 research paradigm competing with neural approaches for "general intelligence." This approach has been almost entirely superseded by learning-based methods for general-purpose reasoning, though some hybrid neuro-symbolic research continues in narrower domains.

Why it stalled: the sheer scale of manually encoding common-sense knowledge doesn't scale the way learning from data does — every new fact requires human labor, while a neural network's "knowledge" scales with available text data and compute, which grew exponentially while human labor did not.

Superseded, not disprovenExplicit Memory Modules (Neural Turing Machines)

Graves et al.'s Neural Turing Machines (2014) proposed giving a neural network an explicit, addressable external memory bank it could read from and write to, directly analogous to a computer's RAM — a genuinely elegant idea for giving networks long-term, structured memory.

Why it stalled: in practice, the field found a simpler alternative worked better at scale — just make the context window bigger, and/or use retrieval (Part 1 above) instead of a learned read/write memory controller. The explicit-memory idea wasn't proven wrong so much as out-competed by blunter, more scalable alternatives — worth remembering, since "the more scalable approach won" doesn't always mean "the more elegant idea was flawed."

Didn't achieve broad adoptionCapsule Networks

Sabour, Frosst & Hinton's "Dynamic Routing Between Capsules" (2017) — proposed by one of deep learning's own pioneers as a fix for a real limitation of CNNs (their pooling operations lose precise spatial relationships between features). It's a particularly instructive example precisely because Hinton himself, who co-invented backpropagation's popularization (Evolution article, Era 4), proposed it.

Why it stalled: the routing mechanism was computationally expensive and difficult to scale efficiently on existing hardware, and subsequent architectures (Vision Transformers, Article 4) addressed similar spatial-relationship weaknesses through a completely different mechanism (attention) that scaled better in practice. The lesson: a technically sound idea from a top researcher still needs the era's hardware and tooling to cooperate.

A necessary caveat on this whole section: "dead end" is a claim about the current mainstream research consensus, not a permanent verdict — the Evolution article's own history (Eras 4–8) shows ideas can sit dormant for decades before the right data or compute revives them. Treat this section as "currently out of favor and why," not "provably impossible."

Part 3 — Where the Field Is Actively Betting

Open Research Directions With Real, Current Momentum

Open betContinual Learning (Solving Catastrophic Forgetting)

Today's models are essentially frozen after training — they don't genuinely learn from new experience the way a human researcher does day to day (a gap this site's Day in the Life article touches on from the human-researcher side). The core obstacle, catastrophic forgetting, was formally named decades ago in McCloskey & Cohen's classic 1989 paper on the sequential learning problem: training a network on new information tends to overwrite what it previously learned, unlike human memory's much more graceful degradation.

This is a genuinely unsolved, actively pursued problem — not a matter of scaling existing methods further, but of finding qualitatively different training approaches.

Open betWorld Models

Yann LeCun's "A Path Towards Autonomous Machine Intelligence" (2022) position paper argues that predicting the next token in text is a fundamentally limited proxy for genuine understanding, and that systems need internal, predictive models of how the physical and causal world behaves — closer to how animals and humans build mental models of cause and effect — to reach more general intelligence.

This remains a genuinely contested bet: it's a minority position relative to "scale the current recipe further," championed seriously by a researcher with deep credibility (LeCun co-invented CNNs — Evolution article, Era 5), which is exactly the kind of disagreement a researcher should learn to sit with rather than resolve prematurely. "The AGI Debate: Scaling vs. New Architectures" steelmans both sides of this exact disagreement in full.

Open betMechanistic Interpretability

Understanding what a trained model is actually doing internally, rather than treating it as a black box, is one of the fastest-growing subfields covered elsewhere on this site (the Interpretability Gap article, and Neel Nanda's career profile in the Field Guide). Anthropic's foundational "Toy Models of Superposition" (2022) is a key entry point into why this is hard: models pack far more concepts into their internal representations than they have individual neurons, using overlapping, entangled patterns instead of one-concept-per-neuron.

Open betLong-Horizon Autonomous Agents

Current models are strong at single-turn or short-exchange tasks but degrade noticeably on tasks requiring sustained, correct action over long time horizons — planning, executing, checking, and correcting across many steps without a human in the loop, a gap directly relevant to the "AgentMode" perspective in this site's interactive learning tool. Research organizations like METR focus specifically on rigorously measuring how this "autonomous task-length" capability is progressing over time, rather than assuming it scales automatically with general capability.

Open betMemory Beyond the Context Window

A middle path between the abandoned Neural Turing Machine approach (Part 2) and pure brute-force context scaling: Google's Infini-attention (2024) is one recent example of research aiming at effectively unbounded context through compressive memory mechanisms, rather than either extreme.

Part 4 — Who's Marching Where

A Quick Map of Bets to Labs

This is a compressed summary — the full comparative analysis, with sourcing for each lab's actual research portfolio, lives in this site's Research Frontier Map, which this table intentionally doesn't try to replace.

Open Bet (Part 3)	Labs/Researchers Most Associated With It
World models	Meta FAIR / Yann LeCun; broader robotics-adjacent research at Google DeepMind
Mechanistic interpretability	Anthropic (dedicated interpretability team); independent researchers like Neel Nanda; academic labs pursuing "ARENA"/"MATS"-style programs (per this site's Field Guide)
Long-horizon agents	OpenAI and Anthropic's agentic-product research; METR as an independent measurement organization
Continual learning & generalization theory	Academic ML theory groups; explicitly central to Safe Superintelligence Inc.'s stated research thesis (per this site's SSI investigation)
Reasoning / test-time compute	OpenAI (o1-family); rapidly adopted across most frontier labs since

Notice the pattern: no single lab owns every open bet, and several labs are explicitly betting on different, competing answers to "what's actually missing for AGI" — which is itself the clearest evidence that this question is genuinely unresolved, not just under-marketed.

Part 5 — The Actual Skill: Judging a New Idea Yourself

A Researcher's Framework for Evaluating Any New Idea

Everything above is a snapshot that will partially age. What doesn't age is the method the Evolution article's whole pattern implies: every real breakthrough addressed a specific, nameable wall using math, data, or compute that had just become available. Here's that pattern turned into a checklist you can run against any new paper or idea you encounter, starting today.

What specific wall does this claim to break? If you can't state it in one plain sentence — "this fixes X, which was broken because of Y" — be suspicious of the claim, or of your own understanding of it yet.

Is that wall real and current, or already solved by something else? Part 2 of this article exists precisely because some "walls" get solved by a completely different, simpler idea before the elegant fix matures (Neural Turing Machines vs. bigger context windows).

Is the fix load-bearing, or cosmetic? Does removing this specific idea make the whole system fail, or does the system work almost as well without it? RMSNorm (Article 4) is load-bearing-ish but replaceable; the Transformer's attention mechanism is not.

What would falsify this idea? A claim with no stated failure condition — no experiment that could, in principle, prove it wrong — is much closer to a belief than a scientific result, regardless of how compelling the writing is.

What does it cost relative to what it buys? Test-time reasoning (Part 1) is a real capability gain that also has a real, quantifiable compute cost — a mature evaluation asks about both sides, not just the headline benchmark number.

Has this been tried before under a different name? Checking Part 2's graveyard, and the broader literature, before assuming an idea is new is a basic form of intellectual honesty — and frequently the answer is "yes, partially, in 20XX, and here's specifically what's different this time."

This checklist is, itself, just the Evolution article's four-part pattern (Idea / Why Now / What It Unlocks / What It Breaks) turned around and pointed at a paper you're evaluating instead of a historical breakthrough you're reading about. That symmetry is not a coincidence — it's the actual transferable skill this whole two-article pair is trying to teach.

Want to see this framework actually run against real papers, question by question, before trying it yourself? "How to Read a Paper Like a Researcher" does exactly that — three full worked examples, including one still-unsettled case.

Self-Check: Try the Framework on Something Real

Pick one paper from this article's reference list you haven't read closely, and run all six framework questions against it before reading its own conclusion section — then compare your answers to the authors' own framing.

Pick one item from Part 3's open bets and write, in your own words, what specific evidence would convince you it's not going to pan out — a real falsification condition, not a vague "if it doesn't work out."

⚠️ What's Genuinely Uncertain in This Article

This article contains more genuine, live disagreement than most others on this site. The "dead ends" in Part 2 reflect current mainstream consensus, which some serious researchers dispute in specific cases. The "open bets" in Part 4 are this site's characterization of where visible research effort is concentrated, not a claim about which bet will actually pay off — reasonable, informed people currently disagree, sometimes sharply, about several of these calls (world models vs. pure scaling being the most visible current split). Treat every claim in Parts 1–4 as dated to September 2026 and worth re-checking against current sources before relying on it.

🔗 Reference Links

🎥 Recommended Videos

But What Is a Neural Network?

3Blue1Brown — foundational context for this whole article

Attention in Transformers, Visually Explained

3Blue1Brown — the backbone method from Part 1

🧭 Closing — You're Reading This at a Genuinely Open Moment

🎯 The Bottom Line

Every method in Part 1 has a real, citable limitation; every idea in Part 2 was reasonable when tried and lost to something else, not to nothing; every bet in Part 4 has serious, credible researchers behind it and no guaranteed winner yet. That's not a discouraging place to start a research career — it's the only kind of moment a research career can meaningfully start in. The framework in Part 5 is the actual takeaway: apply it consistently, to this article included, and you'll be doing the same thing every researcher profiled across this site's Field Guide was doing before anyone called them a researcher.