Standalone Deep Dive · Fifth in the Evolution / After Transformers Arc🧵
Two Threads, One Chain: Architecture vs. Learning Technique, and How They Forked Into Today's AGI Debate
"The Evolution of Neural Networks" told one merged story. This article pulls it apart into two separate lineages — the network's structure (architecture) and how it improves (learning technique) — plus the scaffolding tricks that bridge them, and shows exactly how these two threads, sometimes racing together and sometimes badly out of sync, produced today's scaling-vs-architecture fork.
The Evolution of Neural Networks article walked through 83 years of history as one chain, era by era. That's the right way to tell it once, but it hides something important: "what shape is the network" and "how does the network get better" are two genuinely separate questions, with two separate histories, that don't always advance together. Sometimes an architecture idea sat unused for years because the training technique to make it work hadn't been invented yet. Sometimes a training technique (like reinforcement learning from human feedback) got bolted onto an architecture that hadn't changed in five years. Seeing these as two threads — plus a third, connective thread of scaffolding tricks — makes visible exactly where today's central disagreement, covered in full in "The AGI Debate", actually comes from.
A definitional note before starting, since precision matters here: Architecture means the network's structure — what's connected to what, in what shape (a perceptron's single layer, a CNN's convolutions, a Transformer's attention). Learning technique means how the network's weights get adjusted based on data (a fixed threshold, the perceptron rule, backpropagation, RLHF). Scaffolding means the auxiliary tricks that don't change what the network computes, but change whether it can be trained successfully at all (normalization, initialization, residual connections).
A brilliant architecture with no way to train it is a diagram. A powerful training technique with the wrong architecture underneath it has nothing to improve. Progress happens where the two threads meet — and the gaps between them tell you exactly what to work on next.
Part 1 — Thread A: Architecture (The Network's Shape)
Tracing Architecture Alone, Ignoring How Any of It Was Trained
Read this section pretending training doesn't exist yet — just ask, at each point, "what shape is being proposed?"
1943The Threshold Neuron
McCulloch & Pitts propose a single unit: sum weighted inputs, fire if the sum clears a threshold. Pure structure — the weights are fixed by hand, so there's no learning thread yet at all (Evolution article, Era 1).
1958The Perceptron — Still One Layer
Rosenblatt's perceptron is architecturally almost identical to the McCulloch-Pitts neuron — one layer, one linear decision boundary. Its real innovation (Part 2 below) is entirely on the learning side, not the architecture side — worth noticing, since it's tempting to credit 1958 with more structural novelty than it actually contained.
1989–1998Convolutional Structure (LeNet)
LeCun's CNNs are a genuine architectural leap: weight-sharing filters slid across an image, encoding the assumption that the same visual feature can appear anywhere. This is a structural idea, independent of how it gets trained — LeNet used backpropagation (Part 3), but the convolutional structure itself is a separate contribution.
1997Recurrent Structure With Gated Memory (LSTM)
Hochreiter & Schmidhuber's LSTM adds an architectural feature — a protected "cell state" with learned gates — specifically to solve a problem that shows up when you try to train a plain recurrent structure over long sequences. This is a rare case where the architecture change was designed in direct response to a learning-technique failure (the vanishing gradient problem), which is exactly the kind of cross-thread interaction this article is trying to make visible.
2017The Transformer — Structure Without Recurrence
Vaswani et al.'s Transformer removes recurrence entirely, replacing it with self-attention: every position directly connects to every other position, computed in parallel. This is the architecture thread's biggest single leap since the Perceptron — and notably, it was enabled as much by a hardware fact (GPUs reward parallel computation) as by a structural insight, a theme Part 4 returns to.
2020–2021Transformer Variants: ViT and Mixture-of-Experts
Dosovitskiy et al.'s Vision Transformer (arXiv:2010.11929) applies the same attention structure to image patches instead of words. Fedus, Zoph & Shazeer's Switch Transformer (arXiv:2101.03961) changes the structure again — routing each input to a subset of specialized "expert" sub-networks rather than processing it through the whole model. Both are structural variations on the same 2017 core, not replacements of it — the architecture thread has been comparatively quiet since 2017 relative to how much the learning-technique thread has moved, a gap Part 4 examines directly.
Part 2 — Thread B: Learning Technique (How the Network Improves)
Tracing Learning Technique Alone, Ignoring What Shape the Network Was
Now the opposite exercise: ignore architecture entirely, and ask only "how are the weights being adjusted, based on what signal?"
1949Hebbian Learning — A Rule With No Machine Yet
Donald Hebb's principle ("neurons that fire together, wire together") is a learning rule proposed years before any of the McCulloch-Pitts-style architectures existed to apply it to — a rare case of the learning-technique thread getting ahead of the architecture thread, even if only as a biological hypothesis rather than a computational one.
1958The Perceptron Learning Rule
Rosenblatt's real contribution: adjust each weight in proportion to the error and the input, after every mistake. This is the learning thread's first true leap — self-correction from data — running on an architecture (Part 1) that had barely changed from 1943.
(Backpropagation, 1986, is covered in Part 3 as scaffolding rather than here — see the note in that section for why.)
2014Adam — Smarter Gradient Descent
Kingma & Ba's Adam optimizer (arXiv:1412.6980, Article 3) refines how a gradient signal is turned into a weight update — adaptive, per-parameter learning rates plus momentum — without changing what architecture it's applied to. This became the near-universal default optimizer across CNNs, LSTMs, and later Transformers alike, a genuinely architecture-agnostic learning-technique advance.
2018Unsupervised Pretraining at Scale
BERT and GPT-1 (Evolution article, Era 12) introduce a learning-technique shift, not an architecture one: train on massive unlabeled text with a simple, task-agnostic objective (predict a missing or next word), before any task-specific fine-tuning. The Transformer architecture underneath was already a year old by this point — this is squarely a learning-technique leap riding on an unchanged structure.
2022RLHF at Scale
Ouyang et al.'s InstructGPT ("How to Read a Paper Like a Researcher") adds a further learning-technique layer on top of pretraining: human-preference-based reinforcement learning, shaping behavior rather than raw next-token prediction. Zero architectural change — the same decoder-only Transformer from 2018 is still underneath.
2023–2024Test-Time Compute / Process Reward Models
Lightman et al.'s process-reward-model work ("How to Read a Paper Like a Researcher") and the reasoning-model wave it feeds into ("After Transformers") represent the learning-technique thread's newest axis: spend more compute at inference time generating and evaluating reasoning steps, rather than only improving the weights during training. Once again, the underlying architecture is unchanged — this is a learning/inference-technique innovation riding on a seven-year-old structure.
Notice the pattern across Thread B alone: 2018, 2022, and 2023–2024 are three separate major learning-technique leaps — pretraining, RLHF, test-time compute — all applied to the same 2017 Transformer architecture. The learning-technique thread has been sprinting; the architecture thread has been comparatively still.
Part 3 — The Scaffolding Thread: What Made Architectures Trainable at All
Neither Structure Nor Learning Signal — the Bridge Between Them
Scaffolding techniques don't propose a new network shape (Thread A), and they don't propose a new source of learning signal (Thread B) — they solve the practical problem of getting a specific architecture and a specific learning technique to actually work together, at depth, without diverging or collapsing.
1986Backpropagation — Why It's Scaffolding, Not Purely "Learning Technique"
This is a deliberately debatable placement, worth arguing with rather than accepting passively: backpropagation is often filed under "learning technique," and that's defensible — but it can equally be seen as scaffolding, because its actual contribution is making any multi-layer architecture trainable via the chain rule, regardless of which specific learning signal (perceptron-rule-style correction, RLHF, anything else) eventually rides on top of it. This article files it here specifically to highlight that some ideas genuinely straddle the boundary between threads — the categories are a useful lens, not a rigid taxonomy.
2015Batch Normalization
Ioffe & Szegedy's technique (Article 4) doesn't change a CNN's structure or its learning signal — it stabilizes the numerical behavior of activations during training, which is precisely what let CNN architectures go meaningfully deeper than the 1990s LeNet-era depth without training collapsing.
2015He Initialization
He et al.'s initialization scheme (Article 4) — scaling starting weights based on layer size — is pure scaffolding: it doesn't touch the architecture or the learning rule, it just makes the starting point of training numerically survivable for deep ReLU-based networks.
2016–2019Layer Normalization and RMSNorm
Ba, Kiros & Hinton's LayerNorm, and later Zhang & Sennrich's RMSNorm (both Article 4), are the scaffolding that specifically made the Transformer architecture (Part 1, Era A5) trainable at depth — worth noting explicitly: the Transformer paper's own architecture depends on scaffolding invented in 2016, a year before the paper itself. The architecture thread and scaffolding thread had to be in the right order for 2017 to happen when it did.
Part 4 — Sync and Lag: Where the Threads Met, and Where They Didn't
A Direct Comparison: Lockstep Moments vs. Lag Moments
Laying the three threads side by side makes a pattern visible that the single-timeline version of this history doesn't show as clearly: some breakthroughs required all three threads to arrive together, and some breakthroughs happened because one thread raced ahead while the others stood still.
Moment
Architecture Thread
Learning-Technique Thread
Scaffolding Thread
In Sync?
1958, Perceptron
Unchanged since 1943
Major leap (self-correcting rule)
None yet
Learning ahead of architecture
1986, Backpropagation
Multi-layer nets already proposed conceptually
Enabling breakthrough
Is the breakthrough itself
All three converge
2012, AlexNet
Deeper CNN, not fundamentally new
Unchanged (backprop + SGD)
ReLU + Dropout arrive together with data (ImageNet) + compute (GPUs)
Learning technique sprinting far ahead of architecture
That last row is the whole article's destination. Since 2017, the learning-technique thread has produced three distinct major leaps on an architecture that has barely changed. That imbalance — not a neutral fact, but a specific, visible gap between two threads — is exactly what the scaling-vs-architecture debate is arguing about.
Part 5 — The Fork
How This Exact Gap Produced Today's Two Camps
The same visible gap between the two threads' pace since 2017 is read completely differently by the two camps — as a temporary lag still worth exploiting further, or as a sign the architecture thread itself needs its own next leap.
1
The scaling camp's reading of the gap: the learning-technique thread's three leaps (pretraining, RLHF, test-time compute) each produced large, real capability gains on an unchanged architecture — proof that the current structure has plenty of headroom left, and the smart bet is squeezing more out of Thread B rather than risking a costly, unproven Thread A leap.
2
The new-architecture camp's reading of the same gap: precisely because Thread B has already tried three consecutive major interventions on the same structure, and specific capabilities (continual learning, sample-efficient world modeling — "After Transformers," Part 3) remain unsolved regardless, the honest read is that Thread A itself is the actual bottleneck, and no further learning-technique cleverness will substitute for a genuine structural change.
3
Both readings use the exact same historical facts laid out in Parts 1–4 of this article — which is a useful thing to notice: this isn't a disagreement about what happened, it's a disagreement about what the pattern predicts happens next, which is precisely the kind of question the "How to Read a Paper" framework's falsifiability question is built to probe.
The full steelman of both readings — with sourcing for each camp's strongest evidence, where they actually agree, and what would resolve the disagreement — is already built out in "The AGI Debate: Scaling vs. New Architectures." This article's job was narrower and, hopefully, clarifying: showing you the exact structural gap in the historical record that the debate is actually about, rather than treating it as an abstract disagreement between two schools of thought. For the actual mechanics underneath every era referenced above — what was literally happening during training, not just which lane each idea belonged to — see "How Each Network Architecture Actually Learned."
🧪 Exercise: Extend the Sync Table Yourself
Pick one item from "After Transformers"' open bets (world models, mechanistic interpretability, continual learning, long-horizon agents, or memory beyond context) and classify it: is it primarily a Thread A (architecture) proposal, a Thread B (learning-technique) proposal, or does it require new scaffolding? Write two sentences justifying your classification — this is the same skill Part 4's table required, applied to something still unresolved rather than historical.
⚠️ What's Uncertain in This Article
The three-thread classification is a teaching lens, not a rigid scientific taxonomy. Backpropagation's placement under scaffolding rather than learning technique is explicitly debatable, as stated in Part 3, and several other ideas across this article's history could reasonably be filed differently (RLHF, for instance, has an architectural footprint too, in that it typically involves a separate reward model). Treat the three lanes as a useful way to see the historical pattern, not as the one correct way to categorize every idea.
🧭 Closing — The Gap Is Real; What It Means Is the Argument
🎯 The Bottom Line
Pulling architecture and learning technique apart into separate threads shows something the merged timeline obscures: since 2017, the learning-technique thread has produced three major leaps on a structurally unchanged Transformer, while the architecture thread has been comparatively quiet. That specific, visible gap — not an abstract philosophical difference — is the actual empirical fact both camps in the AGI debate are interpreting. Seeing the gap clearly doesn't tell you who's right; it tells you exactly what evidence would settle it, which is the more useful place for a researcher to stand.