AGI Researcher Foundations · Article 4 of 11 🧠

Core Deep Learning Concepts: Architectures, Normalization, and Why Training Actually Works

Beyond "it's a Transformer": the architectural building blocks (normalization, initialization, residual connections) that make deep networks trainable at all, the training dynamics that explain generalization and its failures, and how these pieces combine in the models this series has already covered.

FrontierAGI Team

September 12, 2026 · 75 min read

Opening Beyond the Transformer Normalization & Init Training Dynamics Regularization Key Papers Courses to Complete Real Scenarios Self-Assessment What's Missing Where This Series Goes Videos References Verdict

Why "It's a Transformer" Isn't a Complete Answer

Article 3 established the math underneath deep learning; this article covers the architectural and training concepts built on top of it. It would be easy to treat "the Transformer" (Article 1's Vaswani et al. paper) as the whole story of modern deep learning architecture, but the Transformer alone doesn't train — it needs normalization to keep activations stable across dozens of layers, careful initialization so gradients don't vanish or explode from the first step, residual connections so gradients can flow through very deep stacks at all, and regularization so the resulting model generalizes past its training set rather than memorizing it. These pieces are individually simple and collectively responsible for most of why deep learning works in practice.

This article also goes slightly further back than the Transformer, because understanding what came before it — and specifically why it had real limitations — is what makes the Transformer's design choices legible rather than arbitrary.

1997 Year the LSTM paper was published — the dominant sequence architecture before Transformers

2015 Year Batch Normalization was introduced, dramatically speeding up deep network training

RMSNorm The normalization variant used by LLaMA and many modern open-weight LLMs

2019 Year "Deep Double Descent" formally documented a training-size phenomenon that contradicts classical statistics

Part 1 — Architectures Beyond the Transformer

What Came Before, and What the Transformer Actually Changed

Before Transformers, sequence modeling (text, speech, time series) was dominated by Recurrent Neural Networks and specifically Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997), which process a sequence one step at a time, carrying a hidden state forward. This works, but has two structural problems the Transformer's attention mechanism (Article 1) was specifically designed to solve: sequential processing can't be parallelized across time steps (slow to train at scale), and information from early in a long sequence has to survive being carried through every intermediate step (prone to being diluted or lost — the "vanishing gradient" problem in its sequence-length form).

Architecture Family	Core Mechanism	Where It's Still Used / Relevant
CNNs (Convolutional Networks)	Local, weight-shared filters slid across spatial data	Vision backbones, though Vision Transformers (below) now compete directly in many settings
RNNs / LSTMs	Sequential hidden-state carry-forward	Largely superseded by Transformers for language; still relevant in some low-latency or streaming contexts
Transformer (encoder-decoder)	Self-attention across the full sequence, in parallel	Original machine-translation design in Vaswani et al. (Article 1)
Transformer (decoder-only)	Causal self-attention, predicting the next token	The architecture behind GPT-family and most modern LLMs referenced throughout this series
Vision Transformer (ViT)	Treats image patches as a token sequence, applies the same attention mechanism	Introduced by Dosovitskiy et al., 2020; now a mainstream vision architecture family

The practical takeaway for a newcomer: you don't need deep historical mastery of RNNs to work with modern LLMs, but knowing that the Transformer's core innovation was trading sequential processing for parallelizable attention — at the cost of the O(n²) compute-in-sequence-length noted in Article 3's linear algebra section — explains both why Transformers scaled so well and why long-context is still an active engineering challenge.

Part 2 — Normalization & Initialization

Normalization and Initialization: Keeping Deep Networks Trainable

Article 3's calculus section covered how gradients can explode or vanish through the chain rule across many layers. Normalization and initialization are the two architectural tools that keep this under control from the start, rather than fixing it after the fact with gradient clipping alone.

Technique	What It Does	Where It's Used
Batch Normalization (Ioffe & Szegedy, 2015)	Normalizes activations across a mini-batch, stabilizing and speeding up training	Standard in CNN architectures; less common in Transformers, which batch differently across sequence length
Layer Normalization (Ba, Kiros & Hinton, 2016)	Normalizes across features within a single example rather than across the batch	Used in the original Transformer (Article 1) and most Transformer variants since
RMSNorm (Zhang & Sennrich, 2019)	A simplified, cheaper variant of Layer Normalization that skips re-centering	Used in LLaMA and many modern open-weight LLM architectures for its lower compute cost
He Initialization (He et al., 2015)	Scales initial weights based on layer size to keep activation variance stable at the start of training	Standard default for networks using ReLU-family activations

Residual / skip connections (He et al., ResNet — Article 1) Pre-norm vs. post-norm Transformer variants

Residual connections — already cited in Article 1's ResNet paper — deserve a second mention here specifically as a training-dynamics tool: by adding a layer's input directly to its output, gradients have a direct path backward through the network that doesn't depend entirely on passing through every intervening transformation, which is a large part of why networks with dozens or hundreds of layers became trainable at all.

Part 3 — Training Dynamics

Training Dynamics: Loss Landscapes, Generalization, and Double Descent

Getting a model to train (loss going down) is necessary but not sufficient — the actual goal, per Article 3's probability section, is a model that performs well on data it hasn't seen. The gap between training performance and held-out performance is the generalization gap, and understanding its behavior is one of the more counterintuitive parts of modern deep learning.

Loss landscapes are non-convex (per Article 3's optimization section) — full of local structure, saddle points, and flat regions — yet first-order methods like Adam reliably find good solutions in practice, which is still an active area of theoretical research rather than something fully explained.

Overfitting — when a model's training loss keeps improving while its validation loss gets worse — is the classical failure mode, and remains a real, common problem, especially with small datasets.

Deep double descent (Nakkiran et al., 2019) documented something classical statistics doesn't predict: test error can rise, then fall again, as model size or training time increases past the point where the model can exactly fit its training data — meaning "bigger model, worse generalization" is sometimes only a temporary, not a permanent, relationship.

Double descent is a genuinely strange empirical result — it means some of the intuitions from classical statistics about model complexity and overfitting don't transfer cleanly to the heavily overparameterized regime modern deep learning operates in.

Part 4 — Regularization

Regularization: Fighting Overfitting Directly

Dropout (Srivastava et al., 2014) — randomly zeroes out a fraction of activations during training, forcing the network to not rely too heavily on any single unit; less commonly used in modern large Transformers than in the CNN era, but still relevant in smaller models and fine-tuning setups.

Weight decay — an L2 penalty on weight magnitude (tying back to Article 3's linear algebra norms), discouraging unnecessarily large weights and a standard default in most training configurations, including Adam's decoupled variant, AdamW.

Data augmentation and early stopping — simpler, non-architectural regularization: expanding effective training data variety, or simply stopping training before the validation loss starts to rise, using the experiment-tracking discipline from Article 2's self-assessment item 6.

Part 5 — Papers & Courses

Key Papers to Read First

Paper	Why It's Foundational	Link
Hochreiter & Schmidhuber — Long Short-Term Memory (1997)	The dominant sequence architecture before Transformers; explains what attention was designed to replace	PDF
Ioffe & Szegedy — Batch Normalization (2015)	Established normalization as a core training-stability tool	arXiv:1502.03167
Ba, Kiros & Hinton — Layer Normalization (2016)	The normalization variant used in the original Transformer	arXiv:1607.06450
Srivastava et al. — Dropout (2014)	The canonical regularization technique for neural networks	JMLR
He et al. — Delving Deep into Rectifiers (2015)	Introduces He initialization, standard for ReLU-family networks	arXiv:1502.01852
Zhang & Sennrich — Root Mean Square Layer Normalization (2019)	RMSNorm, used in LLaMA and many modern open-weight LLMs	arXiv:1910.07467
Dosovitskiy et al. — An Image Is Worth 16x16 Words (ViT, 2020)	Extends the Transformer's attention mechanism to images	arXiv:2010.11929
Nakkiran et al. — Deep Double Descent (2019)	Documents a training-dynamics phenomenon that contradicts classical bias-variance intuition	arXiv:1912.02292

Courses to Complete

Stanford CS231n — Deep Learning for Computer Vision

Free notes · Previously cited in Article 1

Covers CNNs, normalization, and initialization in depth — the most direct course match for Parts 1 and 2 of this article.

Karpathy — Neural Networks: Zero to Hero

Free · YouTube series · Previously cited

Its "Building makemore" episodes build up normalization and initialization concepts from scratch, directly complementing Part 2.

fast.ai — Practical Deep Learning for Coders

Free · Previously cited in Article 1

Covers regularization and generalization pragmatically, with a strong emphasis on diagnosing overfitting in real projects.

This article intentionally reuses several courses already cited in Articles 1 and 3 rather than introducing new ones for their own sake — CS231n, fast.ai, and Karpathy's series each cover a meaningful share of this article's content directly, and re-pointing to the same trusted sources is more useful than diluting the list.

Part 6 — Real Scenarios

Real Scenario Walkthroughs

📉Scenario A — A Deep Network That Won't Train At All

Before Batch Normalization and residual connections became standard, simply stacking many layers often made a network harder to train, not easier — a symptom of vanishing/exploding activations and gradients, per Article 3's calculus section. The historical fix wasn't a smarter optimizer; it was architectural: normalize activations layer by layer (Ioffe & Szegedy), initialize weights to preserve variance (He et al.), and add residual connections so gradients have a direct path backward (He et al.'s ResNet, Article 1). Recognizing "this network won't train past a few layers" as an architectural problem, not just a hyperparameter problem, is exactly what this article's Part 2 is for.

The lesson: normalization and initialization aren't optional refinements — historically, they were the difference between deep networks working at all and not working.

🦙Scenario B — Why LLaMA Uses RMSNorm Instead of LayerNorm

A newcomer reading an open-weight LLM's architecture paper for the first time will often see "RMSNorm" and assume it's an unfamiliar, exotic technique. In practice, it's a direct, well-documented simplification of LayerNorm (Zhang & Sennrich's paper explicitly frames it this way): by skipping the re-centering step and only rescaling by the root-mean-square of activations, it's computationally cheaper at the scale of billions of parameters and trillions of training tokens, with negligible quality cost — the kind of practical, compute-driven engineering decision that recurs constantly at frontier-lab scale (per this series' Field Guide and Lane 1 team coverage).

The lesson: architectural choices in production LLM papers are frequently compute-driven refinements of ideas from Articles 1–4 of this series, not unrelated new inventions.

🎯Scenario C — Diagnosing Overfitting on a Small Reproduction Project

Working on one of Article 2's reproduction exercises (like a nanoGPT fine-tune on a small custom dataset), a common real outcome is training loss dropping steadily while validation loss stops improving or gets worse — the textbook definition of overfitting from this article's Part 3. Applying Article 2's experiment-tracking skill (item 6) is exactly what makes this diagnosable at all — without logging both training and validation loss over time, this pattern is invisible until the model is already deployed and underperforming on new data.

The lesson: the generalization gap isn't an abstract concept — it's directly visible as two diverging lines on a Weights & Biases chart, if you bothered to log both of them.

Every one of the eight "core concepts" in this article exists to solve one of two problems: making a deep network trainable in the first place, or making a trainable network generalize once it is.

Self-Assessment Checklist

Can you explain why LSTMs process sequences slower than Transformers, in terms of parallelization, not just "attention is better"?

Can you explain what problem residual (skip) connections solve, and why very deep networks struggled before they existed?

Do you know the practical difference between Batch Normalization and Layer Normalization, and why Transformers use the latter?

Have you ever plotted training loss against validation loss on your own project and used the gap between them to diagnose overfitting?

Can you explain, at a high level, what "deep double descent" showed that classical statistics wouldn't have predicted?

⚠️ What's Missing or Scoped Out of This Article

This article does not cover: attention-mechanism variants beyond the original Transformer (e.g., sparse or linear attention schemes), mixture-of-experts routing details (already introduced via the Switch Transformer paper in Article 1, but not re-derived here), or a full theoretical treatment of why non-convex optimization works as well as it does in practice — the last of these remains a genuinely open research question, not a settled one, and readers should treat any confident-sounding explanation of it elsewhere with appropriate skepticism.

Where This Series Goes Next

Article 5 moves from these architectural and training-dynamics building blocks into generalization and learning theory proper — formal treatments of why models generalize, PAC-learning-style frameworks, and the bias-variance tradeoff's relationship to the double-descent phenomenon introduced in this article. From there, the series continues through RL foundations, interpretability, alignment, world models, systems, and research methodology, before a capstone article ties everything back into one unified map.

🔗 Additional Reference Links

🎥 Recommended Videos

Neural Networks: Zero to Hero

Andrej Karpathy — "makemore" episodes cover normalization/init directly

Attention in Transformers, Visually Explained

3Blue1Brown — the architecture this article contextualizes

The Spelled-Out Intro to Neural Networks and Backpropagation

Andrej Karpathy — gradient flow fundamentals underlying Part 3

🧭 Closing — Two Problems, One Toolkit

🎯 The Bottom Line

Every concept in this article — LSTMs' limitations, normalization, initialization, residual connections, dropout, double descent — exists to solve one of exactly two problems: making a deep network trainable at all, or making a trainable network generalize once it is. None of these ideas require re-deriving from first principles to use correctly; they require recognizing which problem you're facing (a network that won't train at all points to Part 2; a network that trains but doesn't generalize points to Parts 3 and 4). The next article in this series formalizes the generalization side of that split — why models generalize at all, and what double descent implies for how we think about model capacity.