GPT-3: How the Third GPT Was Actually Built

The Paper That Made "Prompting" a Real Word

GPT-3 was documented in the 2020 paper "Language Models are Few-Shot Learners" (Brown et al., NeurIPS 2020) — a single paper credited to 31 authors at OpenAI. Where GPT-2 tested zero-shot task transfer with mixed results, GPT-3's central experiment was in-context few-shot learning: show the model a small number of input-output examples directly inside the prompt, with no weight updates whatsoever, and let it infer the task from those examples alone. At 175 billion parameters, this was also, by a wide margin, the largest language model trained up to that point.

Zero-Shot Was Promising but Unreliable

GPT-2's zero-shot results were genuinely mixed — strong on some benchmarks, clearly behind dedicated systems on others, with no reliable way to steer the model toward a specific task beyond hoping the prompt phrasing happened to work. GPT-3's authors framed this directly: humans don't need thousands of labeled examples to learn a new task from a few demonstrations, so why should a language model require full fine-tuning to adapt? This reframes this series' Generalization & Learning Theory article's core question — how much a model can generalize from limited examples — as something to test directly inside a single forward pass, not through additional training at all.

In-Context Learning: No Gradient Updates, Just Examples

GPT-3's core idea: at sufficient scale, a language model can perform "in-context learning" — inferring a new task purely from examples placed in its input context, with the model's weights completely frozen. The paper systematically compares three conditions on the same model: zero-shot (task description only), one-shot (one example), and few-shot (a handful of examples) — and shows performance improving substantially from zero-shot to few-shot, with the gap widening as model size increases. This is the direct escalation of GPT-1's fine-tuning and GPT-2's zero-shot bets: each generation removed one more requirement (task-specific training, then any gradient updates at all) while adding scale to compensate.

From Six Authors to Thirty-One

GPT-3's paper credits 31 authors — a genuine step-change from GPT-2's 6, and a scale of collaboration where this series' Other Teams' Code article's Conway's Law dynamics become unavoidable rather than negligible. A team this size training a single 175B-parameter model almost certainly involved distinct sub-groups for data pipeline construction, distributed training infrastructure, and evaluation — the kind of internal specialization this series' Engineering Practicum describes as standard practice once a project outgrows what a handful of generalists can hold in their heads at once. Ilya Sutskever, credited on both GPT-1 and GPT-2, appears again here, alongside a much larger group of newly-credited collaborators.

Eight Sizes, Alternating Attention Patterns

GPT-3 kept the same decoder-only Transformer lineage from GPT-1 and GPT-2, but scaled it across eight sizes from 125M up to 175B parameters, with the context window doubled again to 2,048 tokens. A notable architectural detail: GPT-3 alternates dense and locally-banded sparse self-attention patterns across its layers, drawing on the Sparse Transformer work (Child et al., 2019) to make attention more tractable at this scale — a direct real-world instance of this series' Custom Kernels article's compute-bound-vs-memory-bound tradeoff, since full dense attention at 2,048 tokens across 96 layers would have been considerably more expensive without this pattern.

A Filtered, Weighted Mixture — Not Just "More Web Text"

GPT-3's training data mixed a filtered and deduplicated version of Common Crawl with an expanded WebText2, two internet-based books corpora (Books1 and Books2), and Wikipedia. The Common Crawl portion was filtered using a classifier trained to distinguish high-quality, WebText-like documents from the raw crawl, then fuzzily deduplicated — and critically, the mixture was not sampled in proportion to raw dataset size: cleaner datasets like Wikipedia were sampled more frequently relative to their size than the much larger but noisier Common Crawl. This is a direct, larger-scale extension of this series' Production Training Code article's data-pipeline lesson — how a dataset is weighted and filtered shapes the model as much as how much raw text goes in.

A Real, Disclosed Compute Figure — Unlike GPT-2

In a return to GPT-1-style transparency (and a departure from GPT-2's undisclosed figure), the GPT-3 paper explicitly reports training compute for every model size in the family, with the full 175B model requiring approximately 3.14×10²³ floating-point operations — a number the paper itself uses to study how performance scales with compute, directly connecting to the scaling-law thinking this series' Systems article covers. This scale of training run is exactly the regime this series' Compute Economics article describes as requiring dedicated scheduling infrastructure, quota systems, and multi-week-to-multi-month training windows — a different world entirely from GPT-1's "one month, 8 GPUs."

Emergent Arithmetic, Honest Weaknesses

GPT-3's few-shot results were, in aggregate, a genuine step forward: strong performance across translation, question answering, and cloze-style completion tasks, and a widely-noted result on simple arithmetic — few-shot GPT-3 could perform 2- and 3-digit addition and subtraction with meaningfully higher accuracy than smaller models in the same family, a capability the paper describes as emerging with scale rather than being explicitly trained for. The paper is also candid about weaknesses: performance on tasks requiring comparing two pieces of text (like the Winograd-style WIC benchmark) remained comparatively weak, and the paper includes a dedicated "Broader Impacts" section discussing misuse potential and bias — the same self-assessment discipline this series' Self-Assessment Deep Dive article covers, now applied at a scale where the stakes of getting it wrong were considerably higher than for GPT-1 or GPT-2.

No Open Weights This Time

GPT-3's release marks the sharpest break yet from GPT-1's full open release and GPT-2's eventually-full staged release: OpenAI never publicly released GPT-3's weights at all. Instead, access was provided exclusively through a commercial API, first opened as a private beta in June 2020. This was as much a business-model decision as a safety one — GPT-3 became the foundation of OpenAI's first real commercial product line — and it permanently changed the default expectation for how a frontier lab distributes its most capable model, extending the release-norms trajectory this series' GPT-2 article traces from "staged but eventually open" to "API-only, indefinitely."

The Base Model That Became ChatGPT's Ancestor

GPT-3 itself was a raw, pretrained base model — capable but not yet tuned to reliably follow instructions or converse naturally. That gap was closed by InstructGPT (2022), which applied reinforcement learning from human feedback (RLHF) on top of a GPT-3-family model to make it dramatically better at following instructions and less prone to unwanted outputs — a direct real-world application of this series' Reinforcement Learning Foundations article. That RLHF-tuned lineage is the direct ancestor of ChatGPT, making GPT-3 the pivotal base model on which the entire modern conversational-AI product category was eventually built.

Readiness Checklist

⚠️ What's Missing or Uncertain

Where This Case Study Goes Next

GPT-4 moves from a raw base model to a genuinely productized, multimodal system with a far more involved safety and evaluation process before release — the point where this case-study series' running theme of tightening release norms and growing team size becomes most visible.

🔗 Reference Links

🎥 Recommended Videos

🧭 Closing — Scale Made a New Capability Appear