The GPT Lineage & the ChatGPT Moment
💬 The GPT Lineage & the ChatGPT Moment
🏢 OpenAI
OpenAI, founded in 2015, built the model lineage that would eventually become ChatGPT directly on top of Chapter 6's Transformer architecture — specifically, a stack of Transformer decoder layers trained to predict the next word in a sequence, given everything that came before it.
📈 GPT-1 to GPT-3 — Scaling the Same Idea
GPT-1 (2018) demonstrated that unsupervised pre-training on a large body of text, followed by task-specific fine-tuning, worked remarkably well across a range of language tasks — a relatively modest 117-million-parameter model by later standards. GPT-2 (2019) scaled that same core idea up to 1.5 billion parameters and showed a noticeably more fluent, coherent writing style.
OpenAI initially declined to release GPT-2's full model weights, citing concern that the model was fluent enough to generate convincing misinformation or spam at scale — a genuinely unusual moment of a research lab publicly withholding its own work over anticipated misuse, rather than releasing it and letting the field sort out consequences afterward. The full model was eventually released in stages over the following months, but the decision itself sparked real debate about how AI labs should weigh openness against potential harm — a debate this course's final chapter picks back up.
GPT-3 (2020) represented a genuine scale leap: 175 billion parameters, more than a hundred times larger than GPT-2. It demonstrated few-shot learning — the ability to perform a new task correctly from just a handful of examples given directly in the prompt, with no additional training required. Access was limited to an API rather than a public release or open weights, reflecting both the model's scale and OpenAI's shift toward a commercial product.
📐 Scaling Laws
In 2020, OpenAI researchers including Jared Kaplan published empirical scaling laws: model performance improved smoothly and predictably as model size, dataset size, and compute were all increased together, with no sign of the improvements running out at the scales tested. This finding is, in effect, the formal, quantified version of the "data and compute" throughline running through this entire course since Chapter 3 — and it gave the industry a genuine, evidence-based justification for investing enormous sums into training ever-larger models.
🎯 RLHF — Teaching a Model to Be Helpful
A raw GPT-3-style model is trained to predict plausible next words — it isn't inherently trained to be helpful, honest, or safe to talk to. InstructGPT, published in early 2022, closed that gap using Reinforcement Learning from Human Feedback (RLHF): human raters ranked different model outputs by quality, and those rankings were used as a reward signal to fine-tune the model's behavior toward what people actually found useful. This is Chapter 5's reinforcement learning showing up again, in a very different role — not learning to win a board game through self-play, but learning to align with human preferences through human judgment.
🚀 November 30, 2022
OpenAI released ChatGPT — a free, public, conversational web interface built on a GPT-3.5 model refined with RLHF along the InstructGPT lineage. It reached one million users within five days and roughly 100 million monthly active users within about two months, making it, at the time, the fastest-growing consumer application in history.
Here's the detail worth sitting with: GPT-3, a model with broadly comparable underlying capability, had already existed for two full years before ChatGPT's release. What actually changed on November 30, 2022 wasn't primarily a research breakthrough — it was RLHF's conversational polish combined with a free, simple, chat-style interface that required no technical knowledge or API access to use. The capability was largely already there. What ChatGPT actually delivered was access — and the gap between "a lab has a capable model" and "hundreds of millions of people can feel what that capability is like" turned out to matter enormously, echoing this course's recurring theme (the Mechanical Turk, Deep Blue) about how much public perception depends on a convincing, accessible performance rather than on the underlying capability alone.
📜 Two Years, Same Core Capability
| GPT-3 (2020) | ChatGPT (2022) |
|---|---|
| Comparable core language capability | Same lineage, refined with RLHF for conversational behavior |
| Developer-only access via a paid API | Free, public, web-based chat interface |
| Relatively little mainstream public awareness | ~100 million monthly users within roughly two months |
ChatGPT's public visibility triggered an industry-wide scramble that this course has seen the shape of before: Google rushed out its own competing chatbot, Microsoft invested billions of dollars into OpenAI and rebuilt Bing around the same underlying technology, and Anthropic — founded in 2021 by former OpenAI researchers — released its own model, Claude. It's a genuinely bigger, faster, more capital-intensive race than Course 2's expert-systems boom or Japan's Fifth Generation Project ever were. Whether this round follows the same recurring hype-cycle shape Course 2 named — and if so, what a correction would even look like at this scale — is exactly where this course's final chapter picks up.
🤔 Questions to Sit With
GPT-3's capability existed two years before ChatGPT made it famous. If a comparably capable model appeared today but only through an obscure developer API, do you think it could stay largely unnoticed by the public the way GPT-3 initially did — or has that gap between "capable" and "known" gotten permanently smaller?
OpenAI initially withheld GPT-2 over misuse concerns, then released later, larger, more capable models far more openly. What do you make of that shift — does it suggest the initial caution was overblown, or that competitive and commercial pressure can erode safety caution over time regardless of whether the underlying concern was valid?
RLHF uses human preference judgments to shape model behavior — a very different role for reinforcement learning than Chapter 5's self-play, where the "reward" was an objective win or loss. What are the risks of optimizing a system toward what human raters prefer, as opposed to what's objectively correct or true?
🎯 What's Next
Next chapter — the final chapter of this course: Where We Are Now — diffusion models, multimodal AI, and a light-touch look at the AGI and alignment debate, covering ground that, unlike everything before it in this course, is still actively unfolding.