GPT-4: How the Fourth GPT Was Actually Built

The Report That Explains Why It Won't Explain

GPT-4 was documented in OpenAI's 2023 "GPT-4 Technical Report", released alongside a much larger reference document, the GPT-4 System Card. The technical report contains a sentence that makes this case study fundamentally different from the previous three: it states directly that the report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, or training method, citing "the competitive landscape and the safety implications of large-scale models." This article treats that non-disclosure as the central fact of GPT-4's story, not a gap to paper over.

A Base Model Wasn't the Product Anymore

GPT-3 was a raw base model later turned into a usable assistant via InstructGPT's RLHF fine-tuning. ChatGPT, launched on that lineage in November 2022, became a mass-market product almost overnight — which changed the problem GPT-4 needed to solve. It was no longer "can we build a more capable base model," but "can we build a materially more capable, more reliable, more steerable model, safely enough to put in front of hundreds of millions of people." That shift in problem statement — from a research artifact to a product millions depend on — is the direct throughline into this series' On-Call Reality and Alignment articles: the scrutiny a release needs to survive scales with who's actually using it.

Scale, Multimodality, and Verifiable Safety Testing — Together

GPT-4's stated advances, per the technical report: substantially improved performance on a wide range of professional and academic benchmarks, the ability to accept image inputs alongside text (multimodal input, though the initial public release was text-only), and a training and evaluation process that explicitly incorporated adversarial safety testing before release, rather than after. This is a different kind of "idea" than GPT-1's or GPT-2's architectural or training bets — it's as much a bet about process (test extensively before shipping) as about the model itself.

From an Author List to a Contributor List

GPT-4's technical report breaks the pattern of every prior case study in this series. GPT-1 had 4 authors, GPT-2 had 6, GPT-3 had 31 — each a clean, citable number. GPT-4's report does not list a comparable numbered author roster at all; instead, it credits contributions across many roles (pretraining, long-context, vision, reinforcement learning, evaluation, safety, and more) and closes with an extensive acknowledgments section naming a very large number of contributors, including external red-teamers. This is itself a real, verifiable data point about how far team scale and organizational complexity had grown — exactly the territory this series' Other Teams' Code article describes as the point where Conway's Law stops being an interesting observation and becomes an unavoidable organizational reality.

What the Report Actually Confirms — and What It Doesn't

The technical report confirms GPT-4 is a Transformer-based model pretrained to predict the next token, fine-tuned using reinforcement learning from human feedback, and capable of accepting both text and image inputs. It does not confirm parameter count, layer count, attention pattern, or any other architectural specific — a sharp reversal from GPT-1's 117M, GPT-2's four disclosed sizes, and GPT-3's 175B and full model family table. Various parameter-count figures have circulated publicly since 2023, but none originate from an official OpenAI disclosure — this article treats all of them as unverified rumor, not fact, and reports only what the primary source actually states.

A Real, Verifiable Methodological Contribution

Despite withholding most training specifics, the report does disclose one genuinely notable methodological result: OpenAI built infrastructure that let them accurately predict aspects of GPT-4's final performance — including its final training loss — by extrapolating from much smaller models trained with a small fraction (the report states as little as 1/1,000th to 1/10,000th) of GPT-4's compute. This "predictable scaling" approach is a direct engineering answer to a real problem this series' Systems article raises: at this scale, you cannot afford to discover a training run was misconfigured only after spending the full compute budget, so predicting outcomes from cheap, small-scale proxies becomes essential risk management, not just research curiosity.

Professional Exams, Academic Benchmarks, and Adversarial Red-Teaming

The technical report evaluates GPT-4 on a wide range of professional and academic exams — including a simulated bar exam, on which the report states GPT-4 scored around the 90th percentile of human test-takers — alongside standard academic NLP benchmarks. Separately, the System Card documents a distinct kind of evaluation: adversarial red-teaming for dangerous emergent capabilities, including work with external evaluators such as the Alignment Research Center (ARC), which tested whether the model could autonomously acquire resources or replicate itself — the kind of test this series' Self-Assessment Deep Dive article treats as essential precisely because it's evaluating for a failure mode the developers hope doesn't exist, not just measuring a capability they want to showcase.

A Release Process, Not Just a Release Date

GPT-4 launched on March 14, 2023, initially available through ChatGPT Plus and a waitlisted API — itself another step past GPT-2's staged rollout and GPT-3's API-only model, this time with the staging applied to access tiers and capabilities (text-only at launch, with image input rolled out later) rather than model size. The System Card documents specific mitigations applied before release — including refusal behaviors for certain categories of harmful requests — developed in response to red-teaming findings, a direct real-world instance of this series' On-Call Reality article's discipline of treating a known risk as something to mitigate proactively, not react to after an incident.

The Template for Every Frontier Release Since

GPT-4's pairing of a technical report with a separate, dedicated safety-evaluation document became the template other frontier labs would go on to adopt for their own major model releases — a genuine, lasting process contribution distinct from any specific number in the paper. Its withheld architectural details also set a norm that later OpenAI releases (and, to varying degrees, competitors) have largely continued, making GPT-4 the clearest dividing line in this entire case-study series between the "open research paper" era and the "product release with an accompanying safety document" era.

Readiness Checklist

⚠️ What's Missing or Uncertain

Where This Case Study Goes Next

This series has now traced the GPT lineage from a 4-person, fully open, 117M-parameter research paper to a closed, multimodal, safety-documented product built by an organization too large to credit with a simple author list — the same underlying recipe (a Transformer, pretrained on next-token prediction) carried across five years of compounding scale, team growth, and tightening release norms. GPT-5 continues that trajectory two years later, replacing "one model, one size" with a routed system spanning multiple internal models chosen automatically per query.

🔗 Reference Links

🎥 Recommended Videos

🧭 Closing — The Story Became About What Isn't Said