GPT-1: How the First GPT Was Actually Built

One Paper, One Small Team, One Real Recipe

"GPT" stands for Generative Pre-trained Transformer, and the model that first earned that name was documented in a single 2018 OpenAI paper, "Improving Language Understanding by Generative Pre-Training" by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. It introduced no new architecture of its own — the Transformer already existed — and it trained on a dataset that, by today's standards, is almost comically small. What it introduced was a recipe: pretrain one large language model on raw, unlabeled text, then fine-tune that same model, with minimal changes, on a wide range of specific tasks. That recipe, more than any single number in the paper, is the direct ancestor of every GPT model since.

Why NLP Lagged Behind Vision

By 2017, computer vision had ImageNet-pretrained models that could be fine-tuned onto almost any new visual task with a small amount of labeled data — a transfer-learning recipe that made vision progress compound quickly. Natural language processing had no equivalent. Most NLP systems were still trained largely from scratch, per task, on relatively small labeled datasets, because unlike images, there was no obvious, universally useful "pretraining task" for text that everyone had agreed worked well. This series' own Generalization & Learning Theory article covers exactly why this mattered: a model trained on a small labeled dataset generalizes only as well as that dataset's coverage of the underlying task — and most labeled NLP datasets in 2017 were small enough that this was a real, binding constraint on how good any single-task model could get.

A Semi-Supervised Bet, Not a New Architecture

GPT-1's core idea was not, in isolation, novel — it built directly on two real 2018 precursors: ELMo (Peters et al.), which learned contextual word representations from a bidirectional LSTM language model and fed them into task-specific architectures, and ULMFiT (Howard & Ruder), which pretrained and fine-tuned a full language model for text classification specifically. GPT-1's bet was to go further in one specific direction: use a single Transformer-based language model, pretrained once on unlabeled text, and fine-tune that entire model — not just its output representations — on each downstream task with only a minimal, task-specific input transformation and output layer. This is the same idea this site's Day One Onboarding article describes at a smaller scale — a real project idea starting as a hypothesis worth testing cheaply, not a guaranteed result.

A Roughly Four-Person Effort

The GPT-1 paper credits exactly four authors: Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. That is a strikingly small credited team by the standards of frontier model releases today, where training a flagship model routinely involves credit lists running into the hundreds once infrastructure, data, safety, and evaluation contributors are all named. This isn't just a historical curiosity — it directly shapes what this series' Other Teams' Code article calls Conway's Law in action: a four-person team can iterate on one shared codebase with almost no cross-team coordination overhead, which is part of why a small team could move from idea to published result as quickly as GPT-1's timeline suggests. That same small-team structure would become impossible once training runs required the compute-request processes and cross-team scheduler integration this series' Compute Economics article describes.

Why a Decoder, Not an LSTM

Vaswani et al.'s "Attention Is All You Need" (2017) had introduced the Transformer only a year before GPT-1, originally for machine translation with a full encoder-decoder structure. GPT-1's authors made a specific, consequential choice: use only the decoder half, with masked self-attention that only looks backward at previous tokens — the natural fit for a left-to-right language model, and the architectural seed of every GPT model since. Choosing the Transformer over the LSTMs that ELMo used mattered specifically because self-attention parallelizes across the full sequence during training in a way recurrent architectures cannot, directly connecting to this series' Mathematical Foundations and Systems articles' treatment of why architecture choices are inseparable from what hardware can actually execute efficiently.

BookCorpus: Small, Specific, and Deliberately Chosen

GPT-1 was pretrained on the BooksCorpus dataset, originally assembled by Zhu et al. (2015) for a different purpose (aligning books and movies) — roughly 7,000 unique unpublished books across a range of genres. The GPT-1 paper explicitly explains the choice: books contain long stretches of contiguous text, which lets the model learn to model long-range dependencies in a way that shuffled, sentence-level datasets like those used for some other language-modeling benchmarks at the time could not. This is the pretraining-era equivalent of this series' Production Training Code article's data-pipeline lessons — the specific properties of a dataset, not just its size, shape what a model can actually learn.

117 Million Parameters, By Deliberate Design

The GPT-1 architecture, as specified in the paper: a 12-layer decoder-only Transformer, 768-dimensional hidden states, 12 attention heads per layer, and a 3,072-dimensional position-wise feed-forward network inside each block — roughly 117 million parameters in total. Text was tokenized using byte-pair encoding (BPE) with a 40,000-merge vocabulary, a subword scheme that lets the model handle rare and out-of-vocabulary words gracefully. Every one of these numbers was a real constraint-driven decision, not an arbitrary round figure — matching what this series' Custom Kernels article calls the compute-bound vs. memory-bound tradeoff: a 2018-era single machine with a handful of GPUs simply could not have trained a much larger model in a reasonable amount of time, so the architecture was sized to what the available hardware could actually execute.

A Month, Eight GPUs

The paper states the pretraining run took roughly one month on 8 GPUs — a real, verifiable number that is almost quaint next to the multi-thousand-GPU, multi-month runs this series' Compute Economics article describes as standard for frontier models today. The training objective was straightforward next-token prediction (standard language modeling), optimized with Adam at a max learning rate of 2.5e-4, over 100 epochs on the BooksCorpus data with minibatches of 64 randomly sampled contiguous 512-token sequences. "State of the art hardware" in 2018 meant a single small GPU cluster that a four-person team could plausibly manage without any of the priority-quota-preemption scheduling infrastructure this series' Compute Economics article describes as necessary once hundreds of teams compete for the same GPUs.

One Model, Twelve Tasks

The paper's central experiment is adapting the same pretrained model to 12 different NLP datasets spanning four task types: natural language inference, question answering, semantic similarity, and text classification. For each task, the input text was reformatted into a single ordered sequence the pretrained Transformer could consume (for example, a premise and hypothesis for entailment tasks joined with a delimiter token), with only a small linear output layer added on top — deliberately minimizing task-specific architecture, in contrast to the custom architectures common before this paper. This mirrors this series' Idea to Merged PR article's principle almost exactly: one shared underlying system, validated against many specific tests, rather than a forked implementation per test.

State of the Art on 9 of 12 Benchmarks — Honestly Reported

The paper reports new state-of-the-art results on 9 of the 12 datasets it evaluated against, covering benchmarks like MultiNLI, SciTail, QNLI, RACE, and the Story Cloze Test, while being explicit about where results were more mixed. This kind of honest self-assessment — reporting real wins without overstating generality — is exactly the discipline this series' Self-Assessment Deep Dive article covers, and it directly parallels the "shipping scrutiny" this series' On-Call Reality article describes: a result only counts once it survives honest, specific evaluation, not just a single favorable number.

A Paper, a Blog Post, and a GitHub Repo — No API, No Safety Review

GPT-1's release, in June 2018, consisted of the paper itself, an OpenAI blog post, and code and weights published to a public GitHub repository — a release process almost unrecognizable next to how frontier labs release models today. There was no API gatekeeping usage, no staged rollout, and none of the formal safety review processes that later became standard as models grew more capable — a difference this series' Alignment article covers directly: release norms in AI have tightened specifically because capability grew faster than the field's shared understanding of what could go wrong, and GPT-1's era predates almost all of that infrastructure.

The Recipe Survived; the Scale Didn't

GPT-2 (2019) and GPT-3 (2020) scaled up almost every number in this article — parameters, data, compute, team size — while keeping GPT-1's core recipe intact: a decoder-only Transformer, pretrained on raw text via next-token prediction, adapted to downstream use with minimal task-specific machinery. This is the same "scale, don't reinvent" pattern this site's After Transformers and Evolution of Neural Networks articles trace across the field's whole history — and it's the direct throughline from a four-person team's one-month, 8-GPU training run to the frontier labs and multi-thousand-GPU training jobs this site's entire Engineering Practicum series exists to explain.

Readiness Checklist

⚠️ What's Missing or Uncertain

Where This Case Study Goes Next

GPT-2 scales this exact recipe up by roughly 13x in parameter count, replaces the curated BooksCorpus with a much larger and messier web-scraped dataset, and pushes the fine-tuning-free idea one step further into genuine zero-shot task transfer — while also becoming the field's first major staged-release controversy, a direct preview of the release-norms tightening this article's "What Changed Since" box gestures toward.

🔗 Reference Links

🎥 Recommended Videos

🧭 Closing — A Small Team's Bet That Defined a Decade