The Mathematical Foundations: What You Actually Use, Versus What's Textbook-Only
Linear algebra, probability, calculus and optimization, and information theory — scoped deliberately: not a full math curriculum, but the specific slice each area contributes to daily research work, with the line drawn explicitly between "you'll use this every week" and "you'll rarely touch this past a lecture."
Why This Article Draws a Line Instead of Listing a Curriculum
Ask ten AI researchers what math you "need" and you'll get ten overlapping but different answers, usually padded with everything they personally studied in a math or physics degree. That's not useful to someone deciding what to actually spend the next three months on. This article takes a narrower, more falsifiable approach for each of four areas — linear algebra, probability and statistics, calculus and optimization, and information theory — separating what shows up in daily research work (reading papers, writing training code, debugging a run) from what's genuinely textbook-only for most ML practitioners, and citing exactly where each claim comes from.
This isn't a claim that deeper math is useless — for the research-scientist end of the Field Guide's career paths, more theoretical depth compounds over a career. It's a claim about sequencing: the "daily use" slice below is what unblocks Articles 1 and 2 of this series right now: the reproduction, debugging, and paper-reading skills. The rest can be added later, as specific research directions demand it.
Linear Algebra: Tensors, Rank, and Norms
Every model in this series — from nanoGPT (Article 2) to the frontier-scale systems in the Field Guide — is, mechanically, a sequence of matrix multiplications. Linear algebra isn't a prerequisite you clear once; it's the language the rest of this article is written in.
| Concept | Daily Use or Textbook-Only? | Where It Actually Shows Up |
|---|---|---|
| Vectors & matrices as tensors | Daily | Every tensor in PyTorch/JAX code from Article 1 is this, directly |
| Matrix multiplication & its cost | Daily | Understanding why attention is O(n²) in sequence length, why batching helps GPU utilization |
| Norms (L1, L2) | Daily | Weight decay, gradient clipping, regularization terms in nearly every loss function |
| Eigenvalues & SVD, rank | Regularly, not daily | Understanding why LoRA (Article 1's Hu et al. paper) works: it assumes weight updates during fine-tuning are low-rank |
| Formal vector space axioms, full proofs of decompositions | Textbook-only | Rarely needed to read or write ML papers; useful mainly for a pure theory research track |
Probability & Statistics: Distributions, Bayes, and Divergence
Nearly every model discussed in this series outputs a probability distribution, not a single answer — an LLM's next-token prediction is a distribution over the vocabulary, sampled from at generation time. Probability is how you reason about what a model is actually doing, not just what it outputs.
| Concept | Daily Use or Textbook-Only? | Where It Actually Shows Up |
|---|---|---|
| Distributions (Gaussian, categorical/softmax) | Daily | Weight initialization, the softmax output layer of every classifier and LLM |
| Expectation & variance | Daily | Loss functions are expectations over a data distribution; variance shows up in gradient noise, batch size tradeoffs |
| KL divergence, cross-entropy | Daily | The standard training loss for classifiers and language models; the theoretical basis of RLHF's KL penalty term |
| Bayes' rule (intuition-level) | Regularly, not daily | Reasoning about uncertainty, priors in Bayesian-flavored papers — useful as intuition even outside formal Bayesian ML |
| Measure theory, most classical hypothesis testing | Textbook-only | Rarely required to read or implement mainstream deep learning papers |
Calculus & Optimization: Gradients, the Chain Rule, and Why Training Works at All
Article 2's first self-assessment item asked whether you can implement a training loop's forward pass, loss, backward pass, and optimizer step without copying a tutorial. Calculus is the "why" underneath that "how" — specifically, the chain rule is the entire mathematical justification for backpropagation.
| Concept | Daily Use or Textbook-Only? | Where It Actually Shows Up |
|---|---|---|
| Gradients & the chain rule | Daily | The entire mechanism behind loss.backward() in Article 2's training-loop skill |
| SGD / Adam mechanics | Daily | Nearly every training run in this series uses one of these two optimizer families |
| Convexity (intuition-level) | Regularly, not daily | Understanding why loss landscapes are hard (non-convex) and why that's normal, not a bug |
| Learning rate schedules (warmup, decay) | Daily | Present in essentially every training script and paper's hyperparameter table |
| Full convex optimization theory, second-order methods (e.g. full Newton's method at scale) | Textbook-only | Rarely used directly in large-scale deep learning; first-order methods (SGD/Adam variants) dominate in practice |
NaN is one of the most common real debugging scenarios tied directly to Article 2's training-loop skill — and diagnosing it requires calculus intuition, not just code-reading. The usual cause is exploding gradients: through repeated application of the chain rule across many layers, small per-layer gradient magnitudes multiply into enormous ones, especially in deep or poorly-initialized networks. The standard fixes — gradient clipping (a linear-algebra norm operation, tying back to Part 1) and lower learning rates — only make sense once you understand gradients are the thing being clipped and scaled in the first place.Information Theory: Entropy, Perplexity, and Why LLMs Are Measured This Way
Information theory is the smallest of the four areas in daily surface area, but it directly explains one number that appears constantly in this series' coverage of language models: perplexity.
| Concept | Daily Use or Textbook-Only? | Where It Actually Shows Up |
|---|---|---|
| Entropy, cross-entropy | Daily | Directly the loss function used to train nearly every LLM in this series |
| Perplexity | Regularly, not daily | A standard reported metric in papers — literally 2 raised to the cross-entropy (in bits), i.e., a re-expression of the training loss as an interpretable number |
| KL / JS divergence | Regularly, not daily | Already covered under probability above — the same concept, viewed through an information-theoretic lens |
| Channel capacity, coding theory proofs | Textbook-only | Foundational to the field of information theory itself, but rarely touched in mainstream deep learning research |
The field traces to a single source: Claude Shannon's 1948 paper "A Mathematical Theory of Communication", which introduced entropy as a measure of information content. Every cross-entropy loss function used across this entire series traces its name, and its mathematical justification, directly back to that paper.
Key Papers & Resources
Unlike Article 1's key-papers list, most of the foundational sources here are textbooks and classic papers rather than recent arXiv preprints — the math underneath deep learning has moved far more slowly than the architectures built on top of it.
| Source | Why It's Foundational | Link |
|---|---|---|
| Shannon — A Mathematical Theory of Communication (1948) | Introduces entropy; the direct ancestor of every cross-entropy loss function in this series | |
| Kingma & Ba — Adam: A Method for Stochastic Optimization (2014) | The optimizer used by default across the vast majority of training code referenced throughout this series | arXiv:1412.6980 |
| Goodfellow, Bengio & Courville — Deep Learning (2016) | Free online textbook; Chapters 2–4 map almost exactly onto this article's four sections | deeplearningbook.org |
| Hu et al. — LoRA (2021, previously cited in Article 1) | The applied linear-algebra example used in Part 1 of this article | arXiv:2106.09685 |
Courses to Complete
Self-Assessment Checklist
AB ≠ BA in general) and why that's relevant to how layers compose in a neural network?⚠️ What's Missing or Scoped Out of This Article
Where This Series Goes Next
Article 4 moves from mathematical foundations to core deep learning concepts built directly on top of them — architectures beyond the Transformer, normalization techniques, initialization schemes, and the practical training dynamics (loss landscapes, generalization gaps) that connect back to the calculus and optimization concepts in Part 3 of this article. From there, the series continues through generalization and learning theory, RL foundations, interpretability, alignment, world models, systems, and research methodology, before a capstone article ties everything back into one unified map.
- Shannon — "A Mathematical Theory of Communication" (1948)
- Kingma & Ba — "Adam: A Method for Stochastic Optimization" (arXiv:1412.6980)
- Goodfellow, Bengio & Courville — Deep Learning Book
- Hu et al. — "LoRA: Low-Rank Adaptation" (arXiv:2106.09685)
- MIT 18.06 — Linear Algebra (Gilbert Strang)
- Harvard Stat 110 — Probability
- Hugging Face — Perplexity of Fixed-Length Models (documentation)
- This site — AGI Researcher Foundations: The Technical Stack (Article 1)
- This site — AGI Researcher Foundations: The Self-Assessment Deep Dive (Article 2)