Home › Blog › How Each Architecture Actually Learned
Standalone Deep Dive · Sixth in the Evolution / After Transformers Arc ⚙️

How Each Network Architecture Actually Learned: Training Mechanics, Limits, and Fixes, Era by Era

"Evolution" told the chain. "Two Threads" separated structure from learning technique. This article goes a level deeper than both: for each major architecture, a real, mechanical walkthrough of what was literally happening during training, exactly where it hit a wall, what specific fix (architectural, learning-technique, or scaffolding) got it past that wall — ending at what training itself might have to look like for the next real breakthrough.

FL
FrontierAGI Team
September 14, 2026 · 110 min read
Opening 1. Perceptron 2. MLP + Backprop 3. CNN 4. RNN / LSTM 5. Attention-RNN 6. Transformer 7. Pretraining 8. RLHF 9. Reasoning Models What's Anticipated Next Your Turn What's Uncertain References Videos Verdict

What Makes This Different From the Last Two Articles

"The Evolution of Neural Networks" told this history as one narrative chain. "Two Threads, One Chain" pulled architecture and learning technique apart into separate lineages to reveal a pattern. This article does something different from both: it picks up each architecture generation and stays there, going as deep as plain language allows into exactly what was happening, mechanically, during training — what numbers were being computed, what specifically broke, and what specific change (in the architecture, the learning technique, or the scaffolding around them — using the same three-way vocabulary from "Two Threads") got it working.

One promise up front: every era below follows the same three-part structure — Training Mechanics (how did this thing actually learn, step by step), The Limitation (what specifically broke, and why), The Fix (what change, and from which of the three lanes, solved it). This repetition is deliberate — by the ninth era, the shape should feel completely familiar, because that shape is the actual skill this whole arc has been teaching.

Every architecture in this article eventually hit a wall that had nothing to do with the cleverness of its designers and everything to do with a specific, nameable mechanical bottleneck in how it was trained. Once you can spot that bottleneck yourself, you can predict roughly where the next one will be.
Era 1 — The Single-Layer Perceptron
1958–1969The Perceptron
Rosenblatt, 1958 — one layer, one decision boundary
Training Mechanics
Picture a single unit with a handful of input wires, each with a weight (a number saying how important that input is), feeding into one output. Training happens example by example: show it an input, let it make a guess, compare the guess to the correct answer, and if it's wrong, nudge every weight slightly in the direction that would have made that specific example more likely to be answered correctly. Repeat this thousands of times across many examples. No calculus is involved yet — the nudge rule is simple arithmetic: new_weight = old_weight + (correct − guess) × input × small_step_size
Plain terms: if you were wrong, and a particular input was "on" when you got it wrong, turn that input's importance up or down a little, depending on which way would have helped.
The Limitation
A single perceptron can only draw one straight line (or, with more inputs, one flat plane) to separate two groups of examples. This isn't a training bug that more nudging fixes — it's a structural ceiling. XOR (output 1 only when exactly one of two inputs is on) is provably impossible to solve with one straight line, no matter how long you train or how carefully you tune it — Minsky and Papert's 1969 proof (Evolution article, Era 3) made this mathematically airtight, which is exactly why it stalled the field for a decade rather than just slowing it down.
The Fix, and Which Lane It Came From
This is an architecture fix waiting on a learning-technique fix: stacking multiple layers of perceptrons (an architecture change) can represent XOR and much more, but nobody yet had a working way to train the weights of the hidden, in-between layers — only the final layer's error was directly observable. That specific gap is exactly what Era 2 closes.
Era 2 — Multi-Layer Perceptrons and Backpropagation
1970s–1986Backpropagation Unlocks Depth
Rumelhart, Hinton & Williams, 1986 — multiple layers, trainable end to end
Training Mechanics
Now picture three or four layers stacked: input → hidden layer → hidden layer → output. A "forward pass" pushes an example through all the layers to produce a guess. Then the actual mechanical trick happens: compute how wrong the final guess was, and use the chain rule (a calculus fact about how a change early in a chain of calculations ripples through to the end) to work out, for every single weight in every layer — including the hidden ones deep inside — exactly how much that weight contributed to the final error. Every weight then gets nudged proportionally to its share of the blame. This "backward pass" is why it's called backpropagation: the error signal literally flows backward through the network, layer by layer, assigning blame as it goes. contribution_of_this_weight = (how much the error changes per unit of this weight's output) × (how much this weight's output changes per unit of the weight itself)
The Limitation
Two separate limitations stacked on top of each other here, and it's worth telling them apart. First, a hardware one: 1980s computers were desperately slow at the sheer volume of arithmetic this backward pass requires across many layers and many examples — the algorithm was correct, the machines weren't fast enough yet. Second, a data one: there simply wasn't enough labeled example data lying around for multi-layer networks to show a real advantage over simpler methods (this is the gap Support Vector Machines exploited through the 1990s, per "After Transformers"' graveyard section).
The Fix, and Which Lane It Came From
This is a case where no single fix arrived — it took roughly 25 years of scaffolding and infrastructure catching up (faster CPUs, then GPUs; the slow accumulation of digitized datasets) rather than one clean algorithmic breakthrough. That's an important, easy-to-miss lesson: sometimes the "fix" for a limitation isn't a new idea at all, it's just the world catching up to an idea that was already correct.
Era 3 — Convolutional Networks
1989–2012CNNs, From LeNet to AlexNet
LeCun, 1989–1998; Krizhevsky, Sutskever & Hinton, 2012
Training Mechanics
Same backpropagation engine as Era 2, but applied to a structurally different network: instead of every input pixel connecting independently to the next layer, a small filter (imagine a 3×3 or 5×5 grid of weights, like a tiny stencil) slides across the entire image, using the exact same weights at every position. During training, backprop still computes how each weight should change — but because the filter is reused everywhere, the error signal from every position where that filter was applied gets combined into one shared update. This is the architectural trick (weight sharing) directly changing what the learning-technique has to compute — a good early example of how Thread A and Thread B interact rather than sitting in isolation.
The Limitation
LeNet-5 (1998) worked, and was genuinely deployed reading handwritten bank checks (Evolution article, Era 5) — but going deeper than a handful of convolutional layers ran into the same vanishing-gradient problem haunting Era 2's backward pass, made worse by depth, plus the same data/compute ceiling: there simply wasn't a dataset or a machine in the 1990s or early 2000s large enough to prove a much deeper CNN was worth the trouble.
The Fix, and Which Lane It Came From
AlexNet (2012) is the clean three-lane convergence story: a deeper CNN (architecture, though not radically new), trained with ReLU activations and Dropout (scaffolding — Article 4), on ImageNet's 14 million labeled images running on GPUs repurposed for parallel matrix math (data + compute, the enabling condition rather than a lane itself). No single lane solved it alone — all three had to be ready at once, which is exactly why 2012, not 1998, is when the field actually switched over.
Era 4 — RNNs and LSTM
1990s–1997Recurrent Networks and the Memory Problem
Hochreiter & Schmidhuber, 1997 — solving "backprop through time"
Training Mechanics
A recurrent network processes a sequence (like a sentence) one element at a time, carrying a "hidden state" forward — a running summary of everything seen so far — and feeding it back in as input to the next step. Training this uses a variant of backpropagation called "backpropagation through time": unroll the network across every time step (treat step 1, step 2, step 3... as if they were separate layers stacked in a chain), then run the exact same backward-pass blame-assignment as Era 2, except now the "depth" being backpropagated through is the length of the sequence, not just the number of layers.
The Limitation
This is where the vanishing gradient problem shows up in its sharpest form yet: for a long sequence (say, 50 words), the error signal has to travel backward through 50 sequential multiplications to reach the beginning. Just as with very deep networks (Era 2/3), these repeated multiplications tend to shrink toward zero (or occasionally explode) — meaning a plain RNN effectively "forgets" anything more than a few steps back, no matter how important that early information was to the final answer.
The Fix, and Which Lane It Came From
This is a genuine architecture fix, not a scaffolding patch: LSTM adds a separate "cell state" — a protected pathway that information can flow through relatively unchanged — controlled by three small learned gates (forget, input, output) that decide what to keep, add, or discard at each step. Mechanically, this gives the backward pass a much more direct route back through time for important information, sidestepping the repeated-multiplication problem rather than just training around it.
Era 5 — Attention Bolted Onto Recurrence
2014Attention-Augmented RNNs — the Bridge Era
Bahdanau, Cho & Bengio, 2014 — attention as a learned "lookup," still riding on an LSTM
Training Mechanics
On top of an LSTM's usual step-by-step processing, attention adds a mechanism computed at every output step: compare the current output position against every earlier input position, producing a relevance score for each (how useful is this earlier word, right now?), turn those scores into weights that sum to 1, and use them to build a weighted blend of all the earlier positions' information — rather than relying on a single fixed-size summary carried forward through the whole sequence. These relevance-score calculations are themselves just more weights, trained by the exact same backpropagation engine as everything before them.
The Limitation
Attention fixed the specific bottleneck it targeted (translation quality on long sentences), but the LSTM underneath was still processing the sequence strictly one step at a time — step 50 mechanically cannot start until step 49 finishes. Training remained sequential and slow, and — the detail that turns out to matter most in hindsight — this sequential dependency is precisely the kind of computation GPUs are bad at accelerating, since GPUs are built to do many independent things simultaneously, not one long dependent chain.
The Fix, and Which Lane It Came From
The eventual fix is the most radical architecture move in this entire article: delete the recurrence entirely, and keep only the attention mechanism, applied to every position simultaneously instead of one at a time. That's Era 6.
Era 6 — The Transformer
2017–2018Self-Attention, Trained in Parallel
Vaswani et al., 2017 — "Attention Is All You Need"
Training Mechanics
Every word in the input is turned into three separate learned representations: a Query (what am I looking for?), a Key (what do I have to offer?), and a Value (what information do I actually carry?). Every word's Query is compared against every other word's Key to produce relevance scores (this comparison is just a dot product — multiply matching numbers and add them up), those scores are turned into weights, and each word's output becomes a weighted blend of every other word's Value. Crucially, this entire computation for every word happens simultaneously — there's no step-by-step dependency like an RNN — which means the whole thing can be expressed as a small number of large matrix multiplications, exactly the operation GPUs are fastest at. Backpropagation still trains all of this the same way as every prior era, but now the forward and backward passes can be massively parallelized.
The Limitation
Two limitations showed up almost immediately. First, a scaffolding gap: stacking many self-attention layers deep enough to be useful turned out to be numerically unstable without normalization — which is exactly why LayerNorm (2016, "Two Threads") had to exist a year before this paper for it to actually work at depth. Second, a structural cost: comparing every word against every other word means the computation grows quadratically with sequence length — double the sequence, and the core computation roughly quadruples, a real mathematical ceiling still unresolved today ("After Transformers," Part 1).
The Fix, and Which Lane It Came From
The numerical-instability half was already solved by scaffolding that existed just in time (LayerNorm). The quadratic-cost half has never been fully "fixed" — only worked around, largely through learning-technique and engineering patches (RAG's retrieval instead of attending to everything, MoE's selective computation) rather than a structural repeal of the quadratic cost itself. Worth sitting with: this is a limitation from 2017 that the field has spent nine years working around, not solving outright.
Era 7 — Pretraining at Scale
2018–2020BERT, GPT, and the Scaling Discovery
Devlin et al. (BERT), Radford et al. (GPT), Kaplan et al. and Hoffmann et al. (scaling laws)
Training Mechanics
The architecture (Era 6) stops changing here — what changes is the training signal. Instead of training on a small labeled dataset for one narrow task, the network is trained on a simple, free, task-agnostic goal applied to enormous amounts of raw text: predict a word that's been hidden (BERT's masking) or predict the next word in a sequence (GPT's approach). Because the "correct answer" for this task is just the next word in existing text, there's effectively unlimited free training data — no human labeling required. Backpropagation is, again, the same engine as every prior era; what's new is that the loss being minimized is this simple prediction task, applied at a scale (billions of words) that was previously unthinkable.
The Limitation
Early scaling attempts (through GPT-3, 2020) mostly grew the parameter count while keeping the amount of training data roughly fixed, following an earlier scaling-law estimate (Kaplan et al., 2020) about how to best trade off model size against data. Chinchilla (Hoffmann et al., 2022, Article 1 and 3) directly demonstrated this allocation was wrong: for a fixed compute budget, a smaller model trained on proportionally more data outperformed a much larger model trained the old way — meaning a real, measurable share of the compute spent on the largest pre-2022 models had been mechanically misallocated.
The Fix, and Which Lane It Came From
This is a pure learning-technique correction — no architecture change, no new scaffolding, just a better understanding of how to allocate the same training compute between model size and data quantity, directly changing how every subsequent large training run was planned.
Era 8 — RLHF-Aligned Models
2022RLHF — Training on Human Preference Instead of Text Prediction
Ouyang et al. (InstructGPT), building on Christiano, Leike et al., 2017
Training Mechanics
Three stages, mechanically distinct from anything before: first, the pretrained model (Era 7) generates multiple candidate responses to a prompt; second, human raters rank these responses by preference, and this ranking data trains a separate small network (the "reward model") to predict which response a human would prefer; third, the original model is further trained — using reinforcement learning, not plain backpropagation-on-labels — to produce more outputs that the reward model scores highly. This third stage is mechanically different from everything in Eras 1–7: instead of a direct "correct answer" to backpropagate against, the model is nudged toward behaviors that indirectly maximize a learned, imperfect proxy for human preference.
The Limitation
Optimizing against a learned proxy (the reward model) instead of ground truth introduces a mechanical vulnerability that supervised training (Eras 1–7) doesn't have in the same way: reward hacking — the model can find ways to score well according to the reward model's imperfect judgment without actually producing what a human genuinely wants, a failure mode named years earlier in Amodei et al.'s "Concrete Problems in AI Safety" (2016, "After Transformers"). This isn't a bug to patch once; it's a structural property of training against any imperfect, learned proxy.
The Fix, and Which Lane It Came From
Mostly ongoing learning-technique refinement rather than a single clean fix: Constitutional AI (Article 1) uses AI-generated feedback guided by written principles instead of purely human labels, aiming to reduce some sources of rater inconsistency; more careful reward-model training and monitoring for reward-hacking symptoms are standard practice, but the underlying vulnerability described above remains an open, acknowledged limitation rather than a solved one.
Era 9 — Reasoning and Test-Time Compute
2023–2026Process Reward Models and Test-Time Reasoning
Lightman et al., 2023 ("Let's Verify Step by Step"); the o1-style reasoning-model wave
Training Mechanics
A further refinement to Era 8's reward-modeling idea: instead of a reward model that only judges a final answer as right or wrong, a "process" reward model judges each individual step of a multi-step reasoning chain — meaning a model can be trained to prefer reasoning paths that are step-by-step sound, not just paths that stumble into a correct final answer for the wrong reasons. At inference time (after training is complete), the model can then generate a much longer internal chain of reasoning before answering, effectively spending more computation at the moment of use — a genuinely different lever from every previous era, all of which only ever spent more compute during training.
The Limitation
Cost, mechanically and directly: generating and evaluating a long reasoning chain for every query is far more expensive per response than a single forward pass (Eras 1–8), meaning this technique is currently reserved for problems that justify the expense, not a universal upgrade. There's also a more fundamental open question, honestly unresolved as of this writing: whether process-level supervision is fixing the actual limitation of the underlying Transformer architecture (Era 6's quadratic-cost, single-forward-pass ceiling), or just working around it more cleverly — precisely the question "The AGI Debate" article's two camps read completely differently.
The Fix, and Which Lane It Came From
As of September 2026, this is squarely a learning-technique innovation (new training signal, new inference-time procedure) riding on the same 2017 architecture — which is exactly the empirical pattern "Two Threads, One Chain" traced across this whole period: three consecutive learning-technique leaps (pretraining, RLHF, test-time compute) on a structurally unchanged Transformer.
Where the Next Innovation Is Anticipated

What Training Might Have to Look Like for the Next Real Leap

Everything in this section is reasoned speculation, clearly labeled as such — extending the mechanical pattern from Eras 1–9 forward onto the open bets already covered in "After Transformers," not new information or a forecast this site can verify. Treat it as a way to practice the same pattern-recognition skill this whole article has been teaching, not as a prediction to rely on.
SpeculativeIf Continual Learning Is Next
Every era above trains a model, then freezes it — inference and learning are mechanically separate phases. Genuine continual learning (catastrophic forgetting, "After Transformers") would likely require a training mechanic that doesn't exist in any era above: updating weights safely from a single new experience, without the network overwriting unrelated prior knowledge in the same weights — mechanically closer to how backpropagation currently updates weights during training, but running continuously, safely, during actual use.
SpeculativeIf World Models Are Next
LeCun's world-model proposal ("The AGI Debate") implies a different training signal than next-token prediction (Era 7) entirely: training a network to predict the consequences of actions in a learned internal representation of the world, rather than predicting the next token of text — mechanically closer to how a physics simulator or a video-prediction model is trained than to Era 7's text-prediction objective, though the specific training mechanics remain genuinely unsettled and contested.
SpeculativeIf Memory-Beyond-Context Is Next
Infini-attention and similar approaches ("After Transformers," Part 3) suggest a training mechanic partway between Era 6's pure attention and the largely-abandoned explicit-memory approach (Neural Turing Machines, "After Transformers" graveyard): a compressive memory that's updated incrementally as a sequence is processed, trained end to end with backpropagation like everything else in this article, but explicitly designed to avoid the quadratic cost named in Era 6's limitation.
🧪 Exercise: Apply the Three-Part Pattern Yourself

Pick one item from "After Transformers"' open bets not covered above (mechanistic interpretability or long-horizon agents) and write your own Training Mechanics / Limitation / Fix breakdown for it, exactly matching this article's three-part structure — even though, for an open bet, the "fix" section will necessarily be more speculative than historical. This is the same skill "How to Read a Paper Like a Researcher"'s exercise asked for, applied to architecture-and-training reasoning instead of paper evaluation.

⚠️ What's Uncertain in This Article

The "What's Anticipated Next" section is explicitly speculative and should not be read with the same confidence as Eras 1–9, which describe documented, historical training mechanics. Additionally, some mechanical descriptions above are necessarily simplified for plain-language clarity (the actual matrix mathematics of attention, backpropagation, and RLHF's policy-gradient updates involve more detail than fits a layman explanation) — readers wanting the full mathematical treatment should follow this article's references to the original papers and to Article 3 of this site's Foundations series.

🎥 Recommended Videos

The Spelled-Out Intro to Neural Networks and Backpropagation
Andrej Karpathy — Era 2's mechanics, in code
Attention in Transformers, Visually Explained
3Blue1Brown — Era 6's mechanics, visually
Let's Build GPT: From Scratch, in Code, Spelled Out
Andrej Karpathy — Era 7's pretraining, hands-on

🧭 Closing — Nine Eras, One Repeating Mechanical Pattern

🎯 The Bottom Line
Every era in this article hit a specific, mechanically nameable wall — a straight line that can't represent XOR, a gradient that vanishes across depth or time, a computation that can't parallelize, a proxy signal that can be gamed — and every fix came from precisely one of three places: a new shape, a new training signal, or new scaffolding to make the shape and the signal actually work together. That's the whole pattern, nine times over — no era required a different kind of explanation than the others. Applying that same three-part lens to whatever open bet turns out to matter next (Part "What's Anticipated Next") is, at this point, a skill you now have real practice with, not a return to theory.