Fine-Tuning & RLHF: From GPT-3 to ChatGPT

LLMs

Chapter 9 · Fine-Tuning & RLHF: From GPT-3 to ChatGPT

llm1-7 ended with an honest note: pretraining alone produces a base model — a raw next-token predictor with no notion of following an instruction or refusing a harmful request. This chapter covers the real, three-step mechanism that turns a base model into an assistant — the actual process behind historyai3-7's own GPT-3-to-ChatGPT narrative.

Resolving nlp1-9's Own Distinction, at Full Scale

Exactly the shape nlp1-9 predicted
nlp1-9 named frozen-vs-fine-tuned for a lookup table of word embeddings, and explicitly said this shape mirrors "an entire pretrained network" being either used as-is or further adapted — "only the scale differs." This chapter is where that promise is delivered: everything below is fine-tuning, applied not to an embedding table but to the entire pretrained Transformer from llm1-5, using the identical underlying logic nlp1-9 already introduced — start from a broadly-trained representation, then adapt it further using a smaller, more targeted signal.

Step 1: Supervised Fine-Tuning (SFT)

Humans write high-quality examples of what a good response to an instruction looks like — a much smaller, carefully curated dataset than llm1-7's own massive raw pretraining corpus. The base model continues training on these prompt/response pairs, using the identical next-token-prediction objective from llm1-7 — nothing about the architecture changes (llm1-6's own flexibility point resurfaces here), only the data being trained on. 

# same objective as llm1-7, different data
# pretraining data:     raw internet text, books, code
# SFT data:              curated (instruction, ideal_response) pairs

SFT shifts the model's own behavior toward instruction-following, but it only teaches imitation of the specific examples shown — it can't, by itself, capture the subtler, harder-to-write-down human preferences about what makes one otherwise-reasonable response genuinely better than another.

Step 2: Reward Modeling

To capture those preferences, a separate model is trained. The SFT model generates several different responses to the same prompt; humans rank them by preference; a reward model is trained to predict that ranking — given a prompt and a response, output a single scalar score estimating how much a human would prefer it. 

responses = [sft_model.generate(prompt) for _ in range(k)]   # several candidate responses
ranking = human_rank(responses)                                # humans order them by preference
reward_model.train(prompt, responses, ranking)                  # learn to predict the ranking

Step 3: Reinforcement Learning from Human Feedback (RLHF)

The SFT model is fine-tuned further, this time via reinforcement learning: it generates a response, the trained reward model scores it, and the model's own parameters are updated (via Proximal Policy Optimization, PPO) to increase the likelihood of generating higher-reward responses in the future.

A real safeguard against gaming the reward model
A KL-divergence penalty against the original SFT model is typically included in the RLHF objective — it discourages the model from drifting too far from its own SFT starting point purely to exploit quirks in the reward model rather than genuinely satisfying human preference. Without it, a model can learn to produce responses that score artificially well on the reward model without actually being better in any way a human would recognize.

The Actual Mechanism Behind the ChatGPT Moment

historyai3-7 already named the real, historical November 2022 ChatGPT launch and its RLHF namecheck. This three-step pipeline — SFT, then reward modeling, then RLHF/PPO — is that mechanism, made concrete: GPT-3 (a base model, exactly per llm1-7's own honest note) became ChatGPT through this pipeline, not through additional pretraining or additional scale. llm1-8's own scaling laws describe how pretraining loss improves with scale — a completely separate axis from this chapter's own instruction-following and preference-alignment transformation.

An Honest Limitation

Reward hacking — a real, documented risk
RLHF optimizes a model toward whatever the reward model rewards — and the reward model is itself only an imperfect proxy for genuine human preference, trained on a finite set of human rankings. A model can learn to exploit specific quirks or blind spots in the reward model (a real instance of Goodhart's Law — when a measure becomes a target, it stops being a good measure) rather than genuinely improving in the way human raters actually intended. RLHF measurably improves helpfulness and reduces harmful outputs in practice, but it is not a guarantee of perfect alignment with human intent.

Hands-On Exercises

Exercise 1

Explain precisely why this chapter's own SFT/reward-model/RLHF pipeline counts as "fine-tuning applied to an entire pretrained network," directly connecting this to nlp1-9's own frozen-vs-fine-tuned distinction for embeddings.

📄 View solution
Exercise 2

Explain why supervised fine-tuning alone is insufficient to capture human preferences between two otherwise-reasonable responses, and explain specifically what the reward model adds that SFT cannot provide on its own.

📄 View solution
Exercise 3

Explain what reward hacking is, why it's a real risk specific to RLHF's own design, and explain how the KL-divergence penalty against the original SFT model helps mitigate it.

📄 View solution

Chapter 9 Quick Reference

  • SFT — continue training the base model on curated (instruction, ideal response) pairs, same objective as llm1-7, different data
  • Reward modeling — a separate model trained to predict human preference rankings between candidate responses
  • RLHF/PPO — fine-tune the SFT model to maximize the reward model's own score, with a KL penalty against drifting too far from the SFT starting point
  • This three-step pipeline is the real mechanism behind historyai3-7's own GPT-3-to-ChatGPT narrative — not more pretraining, not more scale (llm1-8)
  • Resolves nlp1-9's own frozen-vs-fine-tuned distinction at full LLM scale, exactly as that chapter predicted
  • Honest limitation: reward hacking — the reward model is an imperfect proxy for genuine human preference, a real instance of Goodhart's Law
  • Next chapter: Context Windows, Quadratic Attention & Honest Limitations