AGI Researcher Foundations · Article 7 of 11 🔬

Interpretability: Looking Inside a Trained Model to See What It's Actually Doing

Every prior article treated a trained network as a black box you evaluate by its outputs. This article covers the tools researchers actually use to open that box — from early saliency maps to modern mechanistic interpretability, circuits, superposition, and sparse autoencoders — building directly on this site's Interpretability Gap article and Anthropic's superposition research already referenced in "After Transformers."

FrontierAGI Team

September 14, 2026 · 75 min read

Opening Early Methods Mechanistic Interpretability Superposition & SAEs Causal Tracing Key Papers Courses to Complete Real Scenarios Self-Assessment What's Missing Where This Series Goes Videos References Verdict

Why "It Works" Isn't the Same as "I Understand Why It Works"

Every article in this series has, up to now, evaluated models by what they output — accuracy, loss curves, human preference scores (Article 6). Interpretability asks a different question entirely: what is actually happening inside the network to produce that output? This site's Interpretability Gap article already covers why this matters at a strategic level — this article goes into the actual technical toolkit, from the earliest, cruder methods through the field's current mechanistic frontier.

A framing worth holding onto throughout: interpretability methods split roughly into two generations. Earlier methods (Part 1) mostly ask "which input pixels/words mattered for this output?" — useful, but shallow. Mechanistic interpretability (Parts 2–4) asks a much harder question: "what specific internal computation is the network performing, and can I identify and even edit the exact piece responsible?"

2013 Year the earliest gradient-based saliency maps were introduced for CNNs

2022 Year Anthropic's "Toy Models of Superposition" explained why one neuron can represent many unrelated concepts

SAEs Sparse autoencoders — the leading current tool for decomposing superposed features into interpretable ones

2 Interpretability generations covered: correlational (early) and mechanistic/causal (modern)

Part 1 — Early Methods: Correlational Interpretability

Saliency Maps, Feature Visualization, and Probing

Saliency maps (Simonyan et al., 2013) — compute the gradient of a model's output with respect to each input pixel, highlighting which pixels most influenced the prediction. Simple and intuitive, but only correlational: a highlighted pixel tells you it mattered, not why or through what internal computation.

Feature visualization (Olah et al., 2017, Distill) — synthesize an artificial input image that maximally activates a specific neuron, revealing what pattern that neuron seems to "want" to see. This produced some of the first genuinely intuitive pictures of what's happening inside a CNN.

Network Dissection (Bau et al., 2017) — systematically checks whether individual neurons align with human-understandable concepts (a "dog detector" neuron, a "texture" neuron), quantifying how much of a network is made of neatly interpretable units versus messier, entangled ones.

Probing classifiers (Alain & Bengio, 2016) — train a small separate classifier on a network's internal activations to check whether specific information (like grammatical structure, in an LLM) is linearly recoverable from that layer, even if the main network was never explicitly trained to represent it that way.

All four of these methods share a limitation Part 2 addresses directly: they're observational, not causal. Knowing a neuron "seems to represent dogs" doesn't tell you whether the network's actual dog-classification computation runs through that neuron, or whether intervening on it would change the output in the way you'd expect.

Part 2 — Mechanistic Interpretability: Circuits

Finding the Actual Circuit, Not Just a Correlated Neuron

Olah et al.'s "Zoom In" (2020, Distill) proposed a more ambitious research program: treat a trained network the way a reverse-engineer treats an unfamiliar circuit board — find the actual, specific pathways of neurons and connections ("circuits") that implement a particular computation, verified rigorously rather than just visualized suggestively.

For Transformers specifically, Anthropic's "A Mathematical Framework for Transformer Circuits" (2021) did for attention-based models what "Zoom In" did for CNNs — and its most famous discovery is the induction head: a specific, identifiable circuit (spanning two attention layers) that lets a model complete a repeated pattern (if "A B" appeared earlier in a sequence and "A" appears again, predict "B" next) — a genuine, mechanistically verified building block of in-context learning, not just a suggestive visualization.

Circuits (verified computational pathways) Induction heads Attention head analysis TransformerLens (open-source tooling)

An induction head isn't a metaphor for how the model might be completing patterns — it's a specific, located, mathematically verified piece of the network doing exactly that, the same way an electronics engineer can point to the specific resistor doing a specific job in a circuit.

Part 3 — Superposition and Sparse Autoencoders

Why One Neuron Can Mean Many Different Things

Anthropic's "Toy Models of Superposition" (2022, previously cited in "After Transformers") explains a discovery that complicates Part 2's circuit-hunting program: networks often represent far more distinct concepts than they have individual neurons, by packing multiple, mostly-unrelated features into overlapping combinations of neurons — a phenomenon called superposition. A single neuron might fire for "this text mentions a dog" and "this text is in French," for reasons that have nothing to do with dogs and French being related — an efficient use of limited neurons, but a serious obstacle to Network Dissection-style "one neuron, one concept" interpretation (Part 1).

The current leading tool for undoing this entanglement is the sparse autoencoder (SAE): a separate, small network trained to re-express a layer's activations in a much larger space where, ideally, each dimension corresponds to one genuinely distinct, human-interpretable feature — trading the efficiency of superposition for interpretability, at the cost of extra compute. Anthropic's "Towards Monosemanticity" (2023) demonstrated this approach successfully extracting thousands of cleanly interpretable features from a real language model, a significant practical step past the toy examples in the 2022 superposition paper.

Part 4 — Causal Tracing: Proving a Circuit Actually Matters

Activation Patching and Editing What a Model "Knows"

Meng et al.'s ROME paper ("Locating and Editing Factual Associations in GPT," 2022) demonstrates the causal, interventionist endpoint of this whole progression: not just observing that a specific internal component correlates with a factual claim (e.g., "the Eiffel Tower is in Paris"), but locating the specific layer and location responsible via "causal tracing" (systematically corrupting and restoring activations to see which ones causally matter), then directly editing that location to change the model's stated belief (e.g., making it say the Eiffel Tower is in Rome) — a level of surgical precision far beyond Part 1's correlational methods.

Activation patching — run the model twice (once on a normal input, once on a corrupted variant), then selectively copy ("patch") activations from one run into the other at specific locations to test whether that location is causally responsible for a specific behavior.

Causal tracing — a systematic version of activation patching used to localize exactly where in a network a specific fact or behavior is stored, as demonstrated in ROME.

Model editing — once a behavior is causally localized, directly modifying that location's weights to change the model's behavior, without full retraining — both a research tool for verifying understanding and, more controversially, a potential practical technique in its own right.

Part 5 — Papers & Courses

Key Papers to Read First

Paper	Why It's Foundational	Link
Olah et al. — "Zoom In: An Introduction to Circuits" (2020)	Establishes the circuits research program this article's Part 2 is built around	Distill
Anthropic — "A Mathematical Framework for Transformer Circuits" (2021)	Extends circuits research to Transformers; introduces induction heads	transformer-circuits.pub
Anthropic — "Toy Models of Superposition" (2022, previously cited in "After Transformers")	Explains why individual neurons resist clean, one-concept interpretation	transformer-circuits.pub
Anthropic — "Towards Monosemanticity" (2023)	Demonstrates sparse autoencoders extracting real, interpretable features at scale	transformer-circuits.pub
Meng et al. — "Locating and Editing Factual Associations in GPT" — ROME (2022)	The clearest demonstration of causal tracing and direct model editing	arXiv:2202.05262

Courses to Complete

Transformer Circuits Thread

Free · Anthropic's ongoing interpretability research publication

Not a course in the traditional sense, but the single richest, most current source for mechanistic interpretability research — the papers cited throughout this article all live here.

Neel Nanda — Getting Started in Mechanistic Interpretability

Free · Practical guide, previously referenced in this site's Field Guide

A hands-on, practitioner-written entry point covering TransformerLens and how to actually start doing this kind of research, not just read about it.

This article's course list is intentionally short compared to earlier articles — interpretability research moves quickly enough that the Transformer Circuits Thread itself, updated continuously, is a more reliable current source than any fixed course curriculum. Check it directly for the latest work rather than relying solely on this article's paper list.

Part 6 — Real Scenarios

Real Scenario Walkthroughs

🔍Scenario A — Debugging Reward Hacking With Interpretability Tools

Article 6 described reward hacking as an RL failure mode (exploiting a flawed reward signal) but offered no way to actually diagnose it beyond output inspection. Activation patching (Part 4) offers a concrete path: if a model's response scores unexpectedly high on a reward model, causal tracing can help identify whether that score comes from genuinely satisfying the reward model's intended criteria, or from an unrelated, exploitable pattern the policy has learned to trigger — turning "the model seems to be gaming something" into a specific, checkable technical claim.

The lesson: interpretability isn't just an academic curiosity about how models work — it's a genuine debugging tool for problems (like reward hacking) that this series has previously only been able to describe from the outside.

🧩Scenario B — Why Superposition Complicates "Just Look at the Neurons"

A newcomer's first instinct, on hearing "we can visualize what neurons do" (Part 1), is often to assume interpretability is close to solved. Superposition (Part 3) is the direct, humbling correction: most individual neurons in a real trained network don't cleanly correspond to one human concept, which is exactly why sparse autoencoders — an entire additional trained model just to make the original model's activations interpretable — became necessary rather than optional.

The lesson: interpretability's hardest problem often isn't building a method to look inside a model — it's that what you find inside resists clean human categories by default.

Self-Assessment Checklist

Can you explain the difference between a correlational method (saliency maps) and a causal method (activation patching), and why the distinction matters?

Can you explain what an induction head is and why finding one counts as a verified circuit, not just a visualization?

Can you explain superposition to someone who assumes "one neuron, one concept" is how neural networks naturally organize information?

Do you understand, at a high level, what a sparse autoencoder is doing and why it needs to be a separate trained model rather than a simple analysis script?

Can you describe what ROME's causal tracing actually demonstrated, beyond "researchers edited a model's knowledge"?

⚠️ What's Missing or Uncertain in This Article

Interpretability is one of this series' fastest-moving subfields — the Transformer Circuits Thread and related research groups publish new findings continuously, and this article's snapshot (September 2026) should be treated as a foundation for reading current work, not a complete or final account. Additionally, whether mechanistic interpretability can scale to fully explain frontier-scale models (rather than the smaller models most published circuits research examines) remains a genuinely open, actively debated question this article does not resolve.

Where This Series Goes Next

Article 8 moves from interpretability to alignment — the broader problem of ensuring a model's goals and behavior match human intent, building on Article 6's RLHF theory and this article's tools for verifying, rather than just hoping, that alignment techniques are working as intended. From there, the series continues through world models, systems, and research methodology, before a capstone article ties every foundational concept back into one unified map.

🔗 Additional Reference Links

🎥 Recommended Videos

But What Is a Neural Network?

3Blue1Brown — foundational context for internal-representation intuitions

Attention in Transformers, Visually Explained

3Blue1Brown — the architecture induction heads (Part 2) live inside

🧭 Closing — From Looking to Proving

🎯 The Bottom Line

Interpretability's history is a story of increasing rigor: saliency maps and feature visualization (Part 1) show you something suggestive; circuits and induction heads (Part 2) verify a specific computation exists; causal tracing and model editing (Part 4) prove a location is responsible and let you change it directly. Each generation of methods didn't replace the last so much as raise the bar for what counts as actually understanding a model, rather than merely visualizing it. Superposition (Part 3) is the honest reminder that this bar keeps rising because the underlying problem — models that pack far more concepts than they have clean, separable neurons — is genuinely hard, not because researchers haven't tried.