Language Modeling from Scratch, Part 1: Course Primer & Overview

What This Course Actually Is

The course's own description, quoted directly from its site, states its purpose plainly:

That operating-systems comparison is doing real work, not just flavor text. An OS-from-scratch course doesn't stop at "here's how a kernel works conceptually" — it has you write a scheduler, a memory allocator, a filesystem, things that either function or don't. CS336 applies the same standard to language models: rather than treating the transformer, the tokenizer, or the training loop as black boxes you import, you build each one, and it either trains a working model or it doesn't. The course's own site does not use a phrase like "efficiency frontier" or name any single unifying philosophy beyond this — worth stating plainly rather than inventing a tidier narrative than what's actually there.

One structural note that matters immediately: the course is officially listed as a 5-unit class, and the site adds a direct warning about workload — "this is a very implementation-heavy class, so please allocate enough time for it." That's not throat-clearing. The prerequisites section backs it up with specifics.

Prerequisites

If you're following this series without the Stanford enrollment behind it, none of this is disqualifying — but it does mean the earlier parts of this series (this one included) will occasionally point you toward a prerequisite concept rather than re-deriving it from first principles, the same way the course itself would.

The Five Assignments

The course's coursework is organized around five large public assignments, each with its own GitHub repository, and each corresponding to a cluster of lectures later in this series. Their official one-line descriptions, quoted directly:

Two competitive leaderboard repos also exist publicly — assignment1-basics-leaderboard and assignment2-systems-leaderboard — if you want to benchmark your own implementation against other public submissions once you get there.

Instructors and Offerings

CS336 is taught by Percy Liang and Tatsunori Hashimoto, both Stanford faculty (Liang leads the Stanford NLP Group and Stanford CRFM; both are listed as instructors on every public offering of the course). The course has run at least three times — Spring 2024, Spring 2025 (now archived), and Spring 2026 (the current offering this series follows by default) — and the syllabus genuinely differs between years, not just in dates. Spring 2025, for instance, taught Kernels/Triton as its own dedicated lecture by Hashimoto; Spring 2026 moved that same topic to a different lecture slot taught by Liang, and added TPU coverage alongside GPUs. This series will follow the Spring 2026 structure as its primary source, and flag it explicitly whenever a prior year handled a topic differently in a way that's worth knowing.

The Nineteen Lectures

The official Spring 2026 schedule, in order:

Official lecture recordings are published on YouTube, linked directly from the course's own schedule page. Video recordings and lecture materials (executable code traces and PDF slide decks, committed to the course's public lectures repository) are the two primary sources this series draws on lecture by lecture, alongside the actual assignment repos.

Thinking Like the Course, Day One

Before Part 2 gets into tokenization properly, it's worth sitting with one idea the course leads with even in its overview: nothing here is free, and the course wants you counting the cost from the start. A simple way to get into that mindset immediately — before writing a single line of the actual assignment code — is to ask a question the course's own Lecture 2 makes central: given a model's shape, how many parameters does it have, and roughly how much compute does one training step cost?

Here's a small, original Python script that answers exactly that for a toy decoder-only Transformer — the same kind of back-of-envelope resource accounting the course treats as a foundational skill, not an afterthought:

# A toy resource-accounting script, in the spirit of CS336's own framing:
# "how big is this model, and how much does one training step cost?"
# This is original example code for this series, not copied from any
# CS336 assignment repo.

def transformer_param_count(vocab_size, d_model, n_layers, d_ff, n_heads):
    # Embedding + output projection (often tied, counted once here)
    embedding = vocab_size * d_model

    # Per-layer: attention (Q, K, V, O projections) + feed-forward (2 matrices)
    attn_per_layer = 4 * d_model * d_model
    ffn_per_layer = 2 * d_model * d_ff
    per_layer = attn_per_layer + ffn_per_layer

    total = embedding + n_layers * per_layer
    return total

def training_step_flops(n_params, batch_size, seq_len):
    # Rule of thumb used throughout the scaling-laws literature:
    # ~6 FLOPs per parameter per token for a full forward+backward pass
    tokens_per_step = batch_size * seq_len
    return 6 * n_params * tokens_per_step

# A GPT-2-small-ish configuration
config = {
    "vocab_size": 50257,
    "d_model": 768,
    "n_layers": 12,
    "d_ff": 3072,
    "n_heads": 12,
}

n_params = transformer_param_count(**config)
flops = training_step_flops(n_params, batch_size=32, seq_len=1024)

print(f"Parameters: {n_params:,} (~{n_params/1e6:.1f}M)")
print(f"FLOPs per training step: {flops:.2e}")

What's Coming in This Series

Part 2 picks up the other half of Lecture 1 — tokenization: byte-pair encoding, why it exists, and a from-scratch implementation walkthrough grounded in what assignment1-basics actually asks you to build. From there, this series follows the Spring 2026 lecture order one part per lecture: architectures and attention alternatives, the systems stack (GPUs/TPUs, kernels, parallelism), scaling laws and inference, data and evaluation, and post-training/alignment — nineteen parts in total, each with its own real citations, code walkthroughs, and hands-on directions tied to the actual public assignment repos.

⚠️ Confidence Notes and Gaps

🔗 Full Reference List