Pretraining at Scale: Self-Supervised Learning

LLMs

Chapter 7 · Pretraining at Scale: Self-Supervised Learning

llm1-6 named next-token prediction as GPT's own training objective without unpacking it. This chapter does — and delivers, with real numbers, nlp1-10's own "self-supervised pretraining taken further" claim.

The Self-Supervised Thread, Traced From nlp1-5

nlp1-5's word2vec and nlp1-9's GloVe both trained on a self-supervised signal — predict a word from its surrounding context, with no hand-labeled data required. LLM pretraining is the same underlying idea, scaled from "predict one missing word in a local window" to "predict every next token across an entire corpus," using the full Transformer stack llm1-5 built rather than a shallow lookup table.

Causal Language Modeling — The Actual Objective

Given a training sequence, the model's job at every single position is to predict the token that actually comes next, using only the tokens before it — exactly the constraint llm1-6's own causal masking enforces.

sequence = ["The", "capital", "of", "France", "is", "Paris"]

# one training sequence yields a prediction target at every position:
# "The"                                  -> predict "capital"
# "The capital"                          -> predict "of"
# "The capital of"                       -> predict "France"
# "The capital of France"                -> predict "is"
# "The capital of France is"             -> predict "Paris"

A single sequence of N tokens yields N separate next-token training signals, all computable in one forward pass — because llm1-6's own causal mask already prevents any position from seeing ahead, every position's prediction can be trained simultaneously, in parallel, rather than one at a time. This is a genuine efficiency advantage over nlp1-6's own strictly sequential LSTM training, where each step had to wait for the one before it.

The Loss Function

At each position, the model produces a probability distribution over the entire vocabulary (llm1-2's own tens of thousands of subword tokens) via softmax, and cross-entropy loss compares that distribution against the actual next token.

nlp1-7's own softmax, at vastly larger scale

This is directly the same softmax-over-multiple-classes mechanism nlp1-7 introduced for NER tagging — except instead of choosing among a handful of entity tags, the model chooses among an entire vocabulary of tens of thousands of possible next tokens, at every single position in every training sequence.

Contrast: Causal LM vs. BERT's Masked LM

	Causal LM (GPT)	Masked LM (BERT)
Training signal per sequence	Every position — dense	Only the ~15% of randomly masked positions
Context used	Only earlier tokens (causal)	Both directions (bidirectional)
Usable for left-to-right generation?	Yes — training and generation are consistent	No — assumes the whole sequence already exists

This is the concrete, mechanical reason llm1-6's own architecture split holds: BERT's bidirectional context is exactly what makes coherent left-to-right generation impossible for it — predicting a token using information from tokens that come after it has no equivalent at real generation time, when those later tokens don't exist yet. GPT's causal objective has no such mismatch.

What "Scale" Actually Means

Real pretraining corpora span hundreds of billions to trillions of tokens — web text, books, code, and more. Real models range from hundreds of millions to hundreds of billions of parameters. This is nlp1-10's own "scale" claim, given its first real numbers; llm1-8 covers why more of each helps, and by how much, in a predictable way.

Why self-supervision is what makes this scale possible at all

Hand-labeling even a fraction of a trillion tokens is not remotely feasible — compare this to nlp1-2/nlp1-3's own small, manually-inspectable datasets. Self-supervision sidesteps this entirely: the "label" for every position is simply the next token already present in the raw text. Any text scraped from the web, digitized from books, or pulled from code repositories becomes usable training data with zero manual annotation — the exact property that makes training at this scale achievable in the first place, not just theoretically desirable.

An Honest Note: This Produces a Base Model, Not an Assistant

A model trained purely on next-token prediction learns to continue text plausibly — it has no built-in notion of "follow this instruction" or "refuse this harmful request." Turning a raw pretrained base model into something that behaves like an assistant is llm1-9's own job: fine-tuning and RLHF.

Hands-On Exercises

Exercise 1

Using this chapter's own "The capital of France is Paris" example, explain why a single training sequence of N tokens produces N separate training signals, and explain why this can be computed in parallel rather than one position at a time.

📄 View solution

Exercise 2

Explain why BERT's own masked language modeling objective cannot be used for coherent left-to-right generation, while GPT's causal language modeling objective can, using this chapter's own compare-table.

📄 View solution

Exercise 3

Explain specifically why self-supervised training is what makes training on trillions of tokens feasible at all, contrasting this with nlp1-2/nlp1-3's own small, hand-inspectable datasets.

📄 View solution

Chapter 7 Quick Reference

Causal language modeling — predict the next token at every position, using only earlier tokens; N tokens yields N training signals, computed in parallel
Cross-entropy loss over the full vocabulary — nlp1-7's own softmax mechanism, at a far larger scale
Causal LM vs. masked LM (BERT): dense per-position signal vs. sparse masked-position signal; only causal LM stays consistent with real generation
Scale: hundreds of billions to trillions of training tokens, hundreds of millions to hundreds of billions of parameters — nlp1-10's own "scale" claim, given real numbers
Self-supervision (no hand-labeling required) is the specific property that makes this scale achievable at all
Honest note: pretraining alone produces a base model, not an assistant — llm1-9 covers the fine-tuning/RLHF step that changes that
Next chapter: Scaling Laws