Scaling Laws

LLMs

Chapter 8 · Scaling Laws

llm1-7 gave real numbers for what "scale" means — but not why more of it helps, or by how much. This chapter closes that gap: scale isn't a vague intuition, it's a measured, predictable relationship.

Loss Improves as a Power Law, Not a Guess

As model size (parameters, N), dataset size (tokens, D), and training compute (C) each increase, the cross-entropy loss from llm1-7's own training objective decreases smoothly, following a power-law relationship:

Loss(N) ≈ (N_c / N) ^ alpha_N

# plotted on a log-log axis, this relationship is a straight line —
# loss keeps falling predictably as N grows, across many orders of magnitude

This is what "scaling laws" literally means: laws, discovered empirically (Kaplan et al., 2020), that govern how scale relates to performance. Not a diminishing-returns cliff, not a random walk — a genuinely predictable curve.

Why Predictability Is the Practically Important Part

Extrapolating before spending millions

Because the relationship is a smooth, measurable curve, labs can run many small, cheap training runs at modest scale, fit the power-law curve to those results, and predict with real accuracy how a proposed much larger — and far more expensive — training run will perform, before committing the compute budget to actually run it. This is the load-bearing, practical use of scaling laws, not an academic curiosity.

Kaplan (2020) vs. Chinchilla (2022) — A Real Correction

Kaplan et al.'s original 2020 scaling laws suggested that, for a fixed compute budget, model size should be scaled up much faster than dataset size — leading to very large models trained on comparatively modest amounts of data. GPT-3 is a real example of this era's approach. DeepMind's 2022 Chinchilla paper (Hoffmann et al.) found this was actually suboptimal.

Model	Parameters	Approach	Result
Gopher	280B	Kaplan-style — large model, comparatively less data	Undertrained relative to its own size
Chinchilla	70B	Model size and dataset size scaled roughly equally, same compute budget as Gopher	Outperformed the far larger Gopher

The real finding: for a given compute budget, there's an optimal ratio of model size to dataset size — not "bigger model always wins," and not "more data always wins," but a genuine balance. Many large models from the Kaplan-influenced era were significantly undertrained relative to their own parameter count.

Completing nlp1-10's "Scale" Claim

nlp1-10 asserted that scale matters; llm1-7 gave it real numbers; this chapter supplies the actual mathematical relationship — a measured power law, with a real, historically-corrected understanding of how to spend a compute budget wisely across model size and data size, rather than "throw more of everything at it."

An Honest Debate: Emergent Abilities

Some capabilities — multi-step arithmetic, certain reasoning tasks — appear to show up suddenly at a scale threshold, rather than improving gradually. This was initially presented as evidence that scale produces genuinely new, qualitatively different capabilities in a discontinuous way.

A genuine, unresolved debate — both sides stated honestly

Later work (Schaeffer et al., 2023, "Are Emergent Abilities a Mirage?") showed that at least some apparent emergence is a measurement artifact. If a discontinuous, all-or-nothing metric is used (like exact-match accuracy on a multi-step problem, which scores 0 unless every step is correct), performance can look like it jumps suddenly at some threshold. Switching to a smooth metric (like token-level accuracy or log-likelihood) on the exact same models often reveals the underlying improvement was gradual all along — the "emergence" was partly an artifact of how the capability was measured, not necessarily a real discontinuity in the model itself. This doesn't settle the debate entirely — some researchers maintain certain abilities do show genuine discontinuities — but it's a real, important caveat that shouldn't be skipped.

What Scaling Laws Don't Cover

Scaling laws describe how pretraining loss improves with scale — next-token-prediction accuracy, nothing more. They say nothing directly about usefulness as an assistant, safety, or alignment with human intent. A model with excellent pretraining loss is still, per llm1-7's own honest note, only a base model. Turning predictable loss improvement into a genuinely useful, well-behaved assistant is llm1-9's own job.

Hands-On Exercises

Exercise 1

Explain what it means for loss to follow a "power law" as scale increases, and explain specifically why this predictability is practically useful to a lab deciding whether to fund an expensive large-scale training run.

📄 View solution

Exercise 2

Using this chapter's own Gopher/Chinchilla comparison, explain what the Chinchilla paper actually found, and explain why "bigger model, same compute" turned out to be the wrong lesson to draw from Kaplan's original scaling laws.

📄 View solution

Exercise 3

Explain how a discontinuous, all-or-nothing evaluation metric can make a genuinely gradual improvement look like sudden "emergence," and explain why this chapter treats the emergent-abilities question as a real, unresolved debate rather than settled in either direction.

📄 View solution

Chapter 8 Quick Reference

Scaling laws — loss decreases as a smooth power law with model size, data size, and compute; a straight line on a log-log plot
Predictability lets labs extrapolate small, cheap experiments to forecast large, expensive training runs before funding them
Kaplan (2020): favored scaling model size faster than data — GPT-3-era models, often undertrained
Chinchilla (2022): model size and data size should scale roughly equally for a fixed compute budget — 70B Chinchilla beat 280B Gopher
Emergent abilities: a genuine, honestly unresolved debate — some apparent "sudden" capability jumps are measurement artifacts of discontinuous metrics, not necessarily proof of true discontinuity
Scaling laws govern pretraining loss only — not usefulness, safety, or alignment; that's llm1-9's own job
Next chapter: Fine-Tuning & RLHF: From GPT-3 to ChatGPT