Standalone Deep Dive · Companion to AGI Researcher Foundations
The Evolution of Neural Networks: From a 1943 Toy Model to Transformers
Written for someone who just finished engineering and wants to actually understand the chain of ideas — not just memorize that "Transformers came after RNNs." Every breakthrough below was a direct, forced response to a specific limitation of the one before it, and every limitation was a mix of missing math, missing data, or missing compute. This article walks that chain, in order, in plain language, with light equations for anyone who wants them.
Why This Article Exists, and How to Read It
Most explanations of deep learning start at the Transformer and work outward — which makes sense if you already have a job doing this, but is a strange way to build actual understanding if you're starting fresh. It leaves you able to use the tools without knowing why they exist in this particular form, and it makes every idea feel like it appeared by magic. This article instead starts at 1943, before "compute" or "training data" meant anything close to what they mean now, and walks forward one forced step at a time.
Every section below follows the same four-part shape, because the pattern itself is the lesson: an idea appears → something about the moment (math, data, or hardware) makes it possible right then → it unlocks something real → it also runs into a wall, which becomes the reason the next idea gets invented. Understanding that repeating shape is, in a real sense, more valuable to an aspiring researcher than any single architecture — it's the pattern you'll be living inside for the rest of your career, since the field hasn't stopped moving through it.
No idea in this article was invented because someone was simply smarter than everyone before them. Every single one was invented because the previous idea hit a wall, and someone had — for the first time — either the math, the data, or the compute to get past it.
Part 1 — Before Computers Could Really Compute
1943The Artificial Neuron — A Model With No Way to Learn Yet
Warren McCulloch (a neuroscientist) and Walter Pitts (a logician) — "A Logical Calculus of the Ideas Immanent in Nervous Activity"
Every input is added up; if the total clears a fixed threshold, the neuron "fires" (outputs 1), otherwise it stays silent (outputs 0). Nothing here adjusts itself yet.
The IdeaReal neurons in a brain either fire an electrical pulse or they don't — there's no "half a pulse." McCulloch and Pitts asked: what's the simplest possible mathematical object that behaves the same way? Their answer: add up a set of inputs, and fire (output 1) only if the sum clears some fixed threshold, otherwise stay silent (output 0).
output = 1 if (x₁+x₂+x₃+...) ≥ threshold, else 0
In plain terms: it's a yes/no decision made by adding things up and comparing to a cutoff — nothing more exotic than that.
Why Now (1943)This wasn't a computer-science paper — there was barely such a field yet. It came from logic and neuroscience meeting: McCulloch studied real brains, Pitts was a mathematical logician, and both were trying to answer "can something built from simple on/off units perform any logical computation?" The answer (yes, in principle) mattered more as a proof about what's mathematically possible than as a working machine — there was no way to run one of these except on paper.
What It UnlockedIt proved that networks of simple threshold units could, in principle, compute anything a digital circuit could compute. That's a big deal philosophically: it's the first formal argument that "thinking," reduced to its logical bones, might be buildable out of simple parts.
What It Couldn't DoThe thresholds and weights in this model were fixed by hand — chosen by the person building it, not learned from examples. There was no concept yet of a machine adjusting itself based on whether it got something right or wrong. That missing piece — learning — is exactly what the next era adds.
↓ Fixed weights aren't enough — the network needs to learn its own weights from examples ↓
1958The Perceptron — The First Machine That Learns From Mistakes
Frank Rosenblatt, Cornell Aeronautical Laboratory — funded by the U.S. Office of Naval Research
The perceptron starts with a random line separating the two classes, then nudges it — a little at a time, after every mistake — until it correctly separates them.
The IdeaRosenblatt took the McCulloch-Pitts neuron and added exactly one new thing: a learning rule. Start with random weights. Show the machine an example. If it gets the answer wrong, nudge each weight slightly in the direction that would have made it more likely to get that example right. Repeat, thousands of times, over many examples.
wᵢ ← wᵢ + learning_rate × (correct_answer − prediction) × xᵢ
Plain terms: "if you were wrong, adjust every input's importance a little bit toward being right next time — bigger adjustment for inputs that were more responsible."
Why Now (1958)Digital computers had just become real, physical machines you could actually run a loop on — the Mark I Perceptron was custom-built hardware, not software on a general computer, precisely because general computers were still so limited. This was the first moment "run an adjustment rule over thousands of examples automatically" was even physically feasible, however slowly.
What It UnlockedA machine that improves with experience, with no human hand-tuning individual weights. The New York Times, reporting on it in 1958, described a machine the Navy expected to eventually "walk, talk, see, write, reproduce itself and be conscious of its existence" — wildly overhyped even at the time, but it shows how genuinely new the idea of a self-adjusting machine felt.
What It Couldn't DoA single perceptron can only draw one straight line to separate two classes. Some very simple problems — famously XOR ("output 1 if exactly one of two inputs is 1, not both or neither") — cannot be solved by any single straight line, no matter how it's positioned. This specific, provable limitation is what the next era is built around.
↓ A formal proof of this exact limitation nearly ends the field for a decade ↓
Part 2 — The First AI Winter
1969"Perceptrons" — The Book That Nearly Ended the Field
Marvin Minsky & Seymour Papert, MIT — the book "Perceptrons: An Introduction to Computational Geometry"
XOR's four points: no matter how you rotate or shift one straight line, it's mathematically impossible to put both green points on one side and both red points on the other.
The IdeaMinsky and Papert didn't invent anything new here — they proved something. They rigorously showed that a single-layer perceptron mathematically cannot represent functions like XOR, no matter how it's trained, because such functions aren't "linearly separable" (can't be split by one straight line, or in higher dimensions, one flat plane).
Why Now (1969)By the late 1960s, perceptron research had attracted heavy U.S. government funding riding on Rosenblatt-era hype. Minsky and Papert, both at MIT, believed that hype had outrun the actual mathematics — and had the rigor to prove exactly where the technique broke, rather than just expressing skepticism.
What It Unlocked (Indirectly)The book's proof was correct and important: it showed the field precisely where the limitation was, and — significantly — Minsky and Papert themselves speculated that multi-layer networks might overcome it, though they were pessimistic that anyone could train one. That speculation planted the target the next two decades quietly worked toward.
What It BrokeThe book's chilling effect on funding was disproportionate to its actual claim (it disproved single-layer perceptrons, not all neural networks) — but nuance rarely survives contact with funding committees. Research funding for neural networks collapsed through most of the 1970s, a period now called the First AI Winter. The missing piece wasn't a new idea so much as a workable training method for multi-layer networks — which existed in rough form but hadn't yet been popularized or proven practical.
↓ A general, practical way to train multi-layer networks finally gets popularized ↓
Part 3 — Learning Returns, Deeper
1986Backpropagation — Teaching Every Layer, Not Just the Last One
David Rumelhart, Geoffrey Hinton & Ronald Williams — "Learning representations by back-propagating errors," Nature
Forward: the network makes a guess. Backward: the error is traced back through every layer using the chain rule, so every single weight learns exactly how much it contributed to the mistake.
The IdeaIf a network has multiple layers, how do you know which weights, deep inside, deserve blame for a wrong final answer? Backpropagation's answer is the chain rule from calculus — you covered this in Article 3 of this series' Foundations sequence — applied layer by layer, backward from the output: figure out how much the final error changes if you nudge the last layer's weights, then use that to figure out how much it changes if you nudge the layer before that, and so on, all the way back to the first layer.
∂Error/∂weight = (∂Error/∂output) × (∂output/∂weight)
Plain terms: "how much does this specific weight matter to the final mistake?" is answered by multiplying together a chain of 'how much does A affect B' steps — exactly the chain rule.
Why Now (1986)The chain rule itself is centuries old, and versions of backpropagation had actually been described earlier (including by Paul Werbos in a 1974 PhD thesis) without much notice. What the 1986 Rumelhart-Hinton-Williams paper did differently was demonstrate it clearly, on real problems, with strong enough results and clear enough writing that the wider research community actually adopted it — a reminder that "who gets credit" in science is often about who makes an idea usable and visible, not strictly who's first.
What It UnlockedMulti-layer ("deep") networks became trainable at all — directly answering Minsky and Papert's XOR problem from Era 3, since a network with even one hidden layer, trained this way, can represent XOR and much more complex functions.
What It Couldn't Do (Yet)Backpropagation worked in principle, but 1980s computers were desperately slow, and there wasn't nearly enough labeled training data lying around for most real-world problems. The idea was ready; the world's data and compute were not. This gap is exactly why the next 25 years look, from the outside, like the field "went quiet."
↓ With training solved in principle, the next question becomes: what should the network's *structure* look like for a specific kind of data, like images? ↓
1989–1998Convolutional Networks — Building the Structure of Vision Into the Network
Yann LeCun and collaborators, Bell Labs — LeNet, reading handwritten digits on real bank checks
A small filter (say, a 3×3 patch of weights that detects an edge) slides across the whole image, reusing the exact same weights at every position — instead of learning separate weights for every pixel location.
The IdeaAn edge detector that works in the top-left corner of an image should work exactly the same way in the bottom-right corner — a cat's ear looks like a cat's ear no matter where in the photo it appears. LeCun's convolutional layers bake this assumption directly into the network's structure: instead of every pixel connecting independently to the next layer (as a plain backprop network would do), a small filter is reused, unchanged, across every position in the image.
Why Now (1989–1998)This is the first place in the story where a genuinely new mathematical idea (parameter sharing, borrowed loosely from earlier biological vision research by Hubel and Wiesel, and Fukushima's 1980 Neocognitron) combined with a real, practical dataset — the U.S. Postal Service's need to read handwritten zip codes, and later, actual bank check digits — to produce something used in the real world, not just a lab demo.
What It UnlockedLeNet-5, LeCun's 1998 network, was genuinely deployed to read a meaningful share of handwritten checks processed by U.S. banks — one of the earliest deep learning systems with real, measurable, everyday-world impact, years before the field's more famous breakthroughs.
What It Couldn't DoCNNs solved vision structure, but a completely different problem remained wide open: sequences, where order and memory matter — language, speech, time series. A sliding filter doesn't naturally capture "what happened earlier in this sentence." That's a structurally different problem, tackled by a parallel line of research running at almost the same time.
↓ Handling sequences requires a network with memory — and memory over long sequences turns out to be its own hard problem ↓
1997LSTM — Giving Networks a Way to Remember, and Forget, on Purpose
Sepp Hochreiter & Jürgen Schmidhuber — "Long Short-Term Memory," Neural Computation
A dedicated "cell state" runs through the whole sequence like a conveyor belt. Three small gates decide, at each step: what to forget, what new information to add, and what to output — all learned, not hand-coded.
The IdeaA plain recurrent network (one that feeds its own previous output back in as it processes a sequence) has a serious flaw: information from many steps ago gets repeatedly multiplied and reprocessed until it either explodes or fades to nothing — the "vanishing/exploding gradient" problem, again a chain-rule issue, just like the one Article 3 and Article 4 of this series cover for deep networks generally, except here it happens across time steps instead of layers. LSTM's fix is a separate "cell state" that information can flow through mostly unchanged, protected by learned gates that decide what to keep, add, or discard.
Why Now (1997)The vanishing gradient problem in recurrent networks had been identified and analyzed rigorously — including in Hochreiter's own 1991 diploma thesis — years before the fix arrived. The 1997 paper is the payoff of that earlier diagnostic work: understanding precisely why something breaks is often most of the work of fixing it.
What It UnlockedNetworks that could, for the first time, reliably carry information across dozens or hundreds of sequence steps — a genuine prerequisite for anything resembling real language understanding, years before anyone had the data or compute to fully exploit it.
What It Couldn't DoLSTMs still process a sequence one step at a time, in strict order — step 50 can't start until step 49 finishes. That's an inherent speed ceiling that no amount of clever gating fixes, and it becomes the central bottleneck the Transformer (Era 11) is built specifically to remove.
↓ Meanwhile, the ideas from Eras 4–6 all sit mostly unused at real scale for a simple reason: the era's computers and datasets are both far too small ↓
Part 4 — The Quiet Decades
1990s–2000sThe Quiet Winter — When Simpler Math Won, Temporarily
Support Vector Machines and other "classical" machine learning methods dominate mainstream results
The IdeaThis era doesn't have one iconic diagram because its real story is an absence, not an invention. Support Vector Machines (SVMs) and similar methods, resting on cleaner, better-understood mathematics (convex optimization — briefly touched on in Article 3), reliably out-performed neural networks on the small, modest datasets available at the time, and were far cheaper to train.
Why This HappenedEvery idea from Eras 4–6 (backpropagation, CNNs, LSTMs) was mathematically sound and already published — nothing was "wrong" with them. What was missing was scale: the datasets of the era (thousands, not millions, of examples) and the CPUs of the era (no practical GPU training yet) simply couldn't show off what deep, multi-layer networks were actually capable of. On small data, simpler methods with stronger theoretical guarantees, like SVMs, genuinely did better — this wasn't hype or bias, it was a fair, honest empirical result at that scale.
What Kept the Field AliveA small number of researchers — Geoffrey Hinton, Yann LeCun, and Yoshua Bengio prominent among them — kept working on neural networks through this period specifically because they believed the limitation was data and compute, not the ideas themselves. Their bet is exactly what pays off in the next era.
What Was Still MissingThree things needed to arrive together before deep networks could show their real advantage: (1) datasets large enough that a network's greater flexibility actually mattered, (2) hardware fast enough to train on that much data in reasonable time, and (3) a handful of practical training tricks to make very deep networks stable. None of the three alone was enough.
↓ The first of the three pieces — a genuinely practical training trick for deep networks — arrives in 2006 ↓
2006"Deep Learning" Gets Its Name and a Practical Training Recipe
Geoffrey Hinton, Simon Osindero & Yee-Whye Teh — "A Fast Learning Algorithm for Deep Belief Nets," Neural Computation
The IdeaHinton's team showed that a very deep network could be trained more reliably by first training it one layer at a time in an unsupervised way (each layer learning to reconstruct its own input, without needing labeled examples), then fine-tuning the whole stack together afterward — "layer-wise pretraining." This sidestepped some of the instability that made very deep networks hard to train directly at the time.
Why Now (2006)By the mid-2000s, computers (even without GPUs yet in wide ML use) were meaningfully faster than in 1986, and a decade of accumulated smaller results gave Hinton's team the confidence and evidence to push depth further than most researchers were currently attempting. The paper's own title — emphasizing "fast" — signals its practical, not just theoretical, ambition.
What It UnlockedThis paper is widely credited with re-popularizing the term "deep learning" itself and re-energizing serious interest in deep, multi-layer networks after the quiet 1990s — directly setting up the community and momentum that the 2012 breakthrough would build on.
What Was Still MissingLayer-wise pretraining turned out to be a useful but ultimately transitional technique — later architectural and training improvements (many covered in Article 4 of the Foundations series: better initialization, normalization, ReLU activations) would make it largely unnecessary. What this era really contributed wasn't the final technique so much as proof that renewed effort on deep networks was worthwhile — momentum, not just method.
↓ The second missing piece — genuinely massive labeled data — and the third — GPU compute — finally arrive together ↓
Part 5 — The Convergence: Data Meets Compute
2012AlexNet — The Moment Everything Finally Lined Up
Alex Krizhevsky, Ilya Sutskever & Geoffrey Hinton, University of Toronto — the ImageNet Large Scale Visual Recognition Challenge
Three separate, individually-existing ingredients — a big enough dataset, fast enough hardware, and the right small training tricks — finally converged in one project.
The IdeaAlexNet wasn't a fundamentally new architecture — it was, at its core, a deeper, bigger version of LeCun's 1998 CNN (Era 5), trained with a handful of practical refinements: the ReLU activation function (faster and more stable than older options), and Dropout (Srivastava et al., cited in Article 4) to reduce overfitting. Individually, none of these three pieces (bigger CNN, ReLU, dropout) was radically new by 2012.
Why Now (2012)This is the cleanest "why now" in the whole story. Fei-Fei Li's ImageNet dataset (started 2009) had, by 2012, assembled roughly 14 million labeled images — orders of magnitude beyond anything available in the 1990s. Meanwhile, gaming GPUs, built for rendering graphics, turned out to be extremely good at the exact kind of parallel matrix math neural networks need, and by 2012 were cheap and programmable enough (via NVIDIA's CUDA) for a graduate student's project to use two of them. Data and compute, missing since the 1990s, were both suddenly, genuinely available.
What It UnlockedAlexNet won the 2012 ImageNet competition by a stunning margin — roughly 10.8 percentage points ahead of the second-place entry, an unprecedented gap in that competition's history. It didn't just win; it made the entire computer vision field switch to deep learning within about two years, essentially overnight by academic-field standards.
What It Couldn't DoAlexNet was built for images — a grid of fixed size, with the "same detector, every position" logic of convolution (Era 5). Language is fundamentally different: variable length, and meaning that depends heavily on relationships between words that can be very far apart in a sentence. Solving vision at scale didn't solve language, and the field's next major fight would be exactly that.
↓ The LSTM's step-by-step memory (Era 6) still isn't enough for long sentences — a more direct way to relate distant words is needed ↓
Part 6 — Language Finds Its Breakthrough
2014Attention Is Born — Letting a Model Look Directly at Any Word It Needs
Dzmitry Bahdanau, Kyunghyun Cho & Yoshua Bengio — "Neural Machine Translation by Jointly Learning to Align and Translate"
Instead of squeezing an entire sentence through one fixed-size memory (as older sequence models did), attention lets the model look back directly at every earlier word, weighting each by relevance — thicker line means more relevance.
The IdeaOlder translation systems compressed an entire input sentence into one fixed-size vector, then tried to generate the output from that single compressed summary — a serious bottleneck for long sentences, similar in spirit to why a single perceptron's one straight line (Era 2) couldn't capture everything. Bahdanau's "attention" mechanism instead lets the model, at each step of generating output, look back and directly weigh every word in the input, learning which ones matter most for the word it's producing right now.
attention_weight(word) = how relevant is this word, right now?
Plain terms: instead of trying to remember the whole sentence at once, the model is allowed to "look back and check" — like re-reading a sentence while translating a specific word, rather than translating purely from memory.
Why Now (2014)By 2014, LSTMs (Era 6) were mature and widely used in production translation systems, so their specific weaknesses — the fixed-size bottleneck for long sentences — were well understood from direct, practical experience, not theoretical speculation. Attention was a targeted fix to a concretely observed, painful problem.
What It UnlockedA measurable, immediate improvement in machine translation quality, especially on long sentences — and, more importantly for this story, a completely new mechanism (learned relevance-weighting across a sequence) that would turn out to be far more powerful and general than anyone translating sentences in 2014 realized.
What It Still KeptBahdanau's attention was bolted onto an LSTM — the sequence was still processed one step at a time underneath the attention mechanism. The truly radical next step, three years later, was asking: what if we removed the recurrent LSTM part entirely, and used attention as the only mechanism?
↓ Remove the sequential bottleneck entirely — process every word of a sentence at the same time ↓
Part 7 — The Transformer
2017"Attention Is All You Need" — Deleting Recurrence Entirely
Ashish Vaswani et al., Google Brain & Google Research — introduced the Transformer architecture
An LSTM must finish word 1 before starting word 2 — inherently sequential. A Transformer looks at all words simultaneously, with every word directly attending to every other word — inherently parallel, which is exactly what a GPU (built for doing many things at once) wants.
The IdeaVaswani and colleagues asked a genuinely bold question: what if attention (Era 10) isn't just a helper mechanism bolted onto an LSTM, but is the entire mechanism — with the recurrent, one-step-at-a-time part removed completely? The resulting Transformer processes every word in a sequence at the same time, using "self-attention" to let every word directly weigh its relevance to every other word, all in one parallel computation rather than a long chain of sequential steps.
Why Now (2017)This is the point where Era 9's lesson (GPUs reward parallel computation) and Era 10's tool (attention) combine directly. An LSTM's sequential nature is precisely the kind of workload GPUs are bad at accelerating (GPUs excel at doing the same operation on many pieces of data simultaneously, not at long chains of dependent steps). A Transformer, by removing the sequential dependency entirely, is exactly the kind of workload GPUs are extremely good at — the architecture and the hardware era finally matched each other perfectly.
What It UnlockedTraining on far larger sequences, far faster, because the parallelizable design finally let researchers actually use the full parallel power of GPU (and later TPU) hardware, rather than being bottlenecked by sequential processing. Every major language model referenced throughout this site's other articles — GPT, Claude, Gemini — descends directly from this architecture.
What Was Still MissingThe Transformer paper itself was about machine translation — a useful but narrow task. The next leap wasn't architectural at all: it was the realization that this same architecture, trained not on a narrow task but on massive amounts of general text, would learn far more than translation alone.
↓ Instead of training for one narrow task, train on everything — and let the model learn general-purpose language understanding first ↓
Part 8 — Pretraining and Scale
2018–2020Pretrain Everything — BERT, GPT, and the Scaling Discovery
Google (BERT, 2018), OpenAI (GPT-1/2/3, 2018–2020), and Kaplan et al.'s scaling laws (2020)
| Model | Year | What Changed |
|---|---|---|
| BERT (Devlin et al., Google) | 2018 | Pretrains a Transformer on massive unlabeled text by having it fill in masked-out words — then fine-tunes cheaply for specific tasks |
| GPT-1 (OpenAI) | 2018 | Pretrains a Transformer to simply predict the next word, at scale — the "decoder-only" design behind nearly every modern LLM in this series |
| GPT-2 (OpenAI) | 2019 | Same idea, 10x the parameters — quality improved smoothly and predictably with scale, a hint of what was coming |
| GPT-3 (OpenAI, cited in Article 1) | 2020 | 175 billion parameters — emergent "few-shot learning": the model could perform new tasks from just a few examples in its prompt, with no retraining |
The IdeaInstead of designing a model architecture specifically for each task (translation, summarization, question-answering), pretrain one large Transformer on an enormous amount of general text using a simple, task-agnostic goal — predict the next word, or fill in a missing word — and let it absorb grammar, facts, and reasoning patterns as a side effect of getting good at that simple goal.
Why Now (2018–2020)The Transformer's parallel-friendly design (Era 11) meant that, for the first time, training on truly enormous text datasets was computationally realistic. As labs pushed model size up through GPT-1, GPT-2, and GPT-3, Kaplan et al.'s 2020 scaling laws paper (cited in Article 1) formalized what was becoming visible empirically: performance improved smoothly and predictably as you scaled up model size, data, and compute together — turning "make it bigger" from a hopeful guess into an evidence-backed research strategy.
What It UnlockedGPT-3's "few-shot learning" — performing a task correctly after seeing just a couple of examples in its prompt, without any retraining — was a genuinely emergent capability nobody had explicitly trained for. This is the moment "scale up a simple recipe" started looking like a real path toward increasingly general capability, not just better performance on narrow benchmarks.
What It Still Got WrongA model trained purely to predict the next word isn't the same as a model trained to be helpful, honest, and safe when talking to a person — raw GPT-3 could be confidently wrong, and had no built-in mechanism for following instructions the way it wasn't literally trained on. That gap between "predicts text well" and "is a good assistant" becomes the next era's whole focus.
↓ Teach the model what humans actually want, using human feedback itself as the training signal ↓
Part 9 — From Predicting Text to Being an Assistant
2022–2026RLHF, ChatGPT, and the Current Frontier
Ouyang et al.'s InstructGPT (2022), building on Christiano, Leike et al.'s 2017 RLHF foundations — and OpenAI's ChatGPT launch, November 2022
The model's answers are ranked by humans; those rankings train a separate "reward model" to predict human preference; the original model is then updated to produce more of what the reward model scores highly — a closed loop, repeated many times.
The IdeaRather than only training a model to predict text, have humans compare pairs of the model's outputs and say which one they prefer. Use those preferences to train a separate "reward model" that learns to predict what humans will like. Then use that reward model to further train the original model — via reinforcement learning — to produce more of what humans actually prefer, not just what's statistically likely text.
Why Now (2022)The theoretical foundation — Christiano, Leike, and colleagues' 2017 RLHF paper, cited in Article 1 — existed years before it was applied at LLM scale. What changed by 2022 was having a base model (GPT-3-class) capable enough that refining its behavior, rather than teaching it new capabilities from scratch, was the actual bottleneck to usefulness. The idea had been sitting ready; the base model finally caught up to where it mattered.
What It UnlockedInstructGPT (Ouyang et al., 2022) demonstrated that a much smaller RLHF-tuned model could be preferred by human raters over a far larger non-tuned model — behavior alignment mattered more than raw scale at this stage. ChatGPT's November 2022 public launch, built on this same approach, became the fastest-growing consumer application in history at the time, converting a research technique into the moment the general public first understood what these models could do.
Where the Chain ContinuesEverything since — Constitutional AI (Article 1), Chinchilla's compute-optimal training (Articles 1 and 3), reasoning models with test-time compute, agentic systems — is this same repeating pattern continuing: a real limitation of the current best approach, met by an idea whose math already existed, made possible by the compute or data of that specific moment. This site's FrontierAI Timeline and Research Frontier Map pick up this exact thread for what's happening right now, in 2026 — the companion piece "After Transformers" picks the chain up exactly here, covering what's actively working today, which past ideas turned out to be dead ends, where research is betting next, and a framework for judging any new idea yourself — and "Two Threads, One Chain" goes back and re-traces this entire history by separating architecture from learning technique, showing exactly how the gap between them produced today's central debate — while "How Each Network Architecture Actually Learned" goes deeper still, into the literal training mechanics of every era.
Read backward, the whole chain is one repeating sentence, said thirteen different ways: "here is a wall we hit, here is the math that was quietly ready, and here is what changed in the world that finally let us use it."
Part 10 — Wrapping Up
What This Means If You're Starting Out Right Now
If you're aiming to become a researcher, the practical lesson from this entire history isn't "memorize thirteen dates." It's that breakthroughs in this field have never come from nowhere — they've come from someone who understood a specific limitation deeply enough to notice exactly what was missing, and who had (or built) the math, data, or compute to fix that one specific thing. That's a learnable skill, not a lightning bolt of genius, and it's exactly the mindset the rest of this series' Technical Stack, Self-Assessment, Mathematical Foundations, and Core Deep Learning Concepts articles are built to help you practice.
A note on historical nuance: the "credit" for several ideas in this article is genuinely contested or shared more widely than a single named paper suggests — backpropagation had earlier, less-noticed precursors (including Werbos, 1974); convolutional ideas trace back further to Fukushima's Neocognitron (1980); and attribution disputes around deep learning's history are an active, sometimes contentious topic among the field's own pioneers. This article presents the most commonly cited version of each story, not necessarily the complete or uncontested one.
🔗 Reference Links, In Chronological Order
- McCulloch & Pitts — "A Logical Calculus of the Ideas Immanent in Nervous Activity" (1943)
- Rosenblatt — "The Perceptron" (1958)
- Hochreiter & Schmidhuber — "Long Short-Term Memory" (1997)
- Hinton, Osindero & Teh — "A Fast Learning Algorithm for Deep Belief Nets" (2006)
- Krizhevsky, Sutskever & Hinton — AlexNet (2012)
- Bahdanau, Cho & Bengio — "Neural Machine Translation by Jointly Learning to Align and Translate" (arXiv:1409.0473)
- Vaswani et al. — "Attention Is All You Need" (arXiv:1706.03762)
- Devlin et al. — "BERT" (arXiv:1810.04805)
- Brown et al. — "GPT-3" (arXiv:2005.14165)
- Ouyang et al. — "Training Language Models to Follow Instructions" — InstructGPT (arXiv:2203.02155)
- This site — AGI Researcher Foundations: The Technical Stack
- This site — AGI Researcher Foundations: The Mathematical Foundations
- This site — AGI Researcher Foundations: Core Deep Learning Concepts
- This site — FrontierAI Timeline
🎥 Recommended Videos
But What Is a Neural Network?
Attention in Transformers, Visually Explained
The Spelled-Out Intro to Neural Networks and Backpropagation
Let's Build GPT: From Scratch, in Code, Spelled Out
🧭 Closing — You're Not Late, and the Pattern Isn't Finished
🎯 The Bottom Line
Every era in this 83-year story followed the same shape: a real limitation, math that was often already sitting quietly in a paper or a thesis, and a specific moment where data or compute finally caught up enough to matter. Nobody in this story had to be the smartest person alive — they had to notice one wall clearly, and have (or build) exactly what was needed to get past it. That pattern is still running today, visibly, in this site's ongoing coverage of frontier labs and current research — which means the next link in this chain hasn't been written yet, and there's no structural reason it can't be written by someone reading this article for the first time.