Decoder-Only vs. Encoder-Decoder: GPT vs. BERT vs. T5

llm1-5 built one Transformer block. The 2017 "Attention Is All You Need" paper nlp1-8 already cited actually proposed stacking these blocks into an encoder-decoder pair, for machine translation specifically — nlp1-8's own real motivating task. This chapter surveys the three architecture families that emerged from arranging Transformer blocks differently, and explains why one of them became dominant.

Three Families, Three Attention Patterns

Encoder-Only: BERT's Bidirectionality

BERT (2018) stacks only encoder-style Transformer blocks, where every token can attend freely to every other token in the sequence, in both directions. This is the exact Transformer-era counterpart of nlp1-7's own bidirectional LSTM — both exist to let a token's own representation be informed by context on both sides, not just what came before. BERT trains via masked language modeling: randomly hide some tokens in the input and predict them using the full surrounding context, generalizing nlp1-5's own context-prediction training signal to a full Transformer stack.

Encoder-Decoder: T5 and the Original Transformer

This is the architecture nlp1-8 already introduced conceptually — an encoder reads the entire input sequence (bidirectionally), and a decoder generates the output sequence one token at a time, at each step cross-attending back to the encoder's own output. T5 generalizes this into a "text-to-text" framing, treating every task — translation, summarization, classification — as converting one string of text into another.

Decoder-Only: Causal Masking

GPT's own family uses no separate encoder at all — one single stack of Transformer blocks, with a critical modification to llm1-4's own attention formula: a causal mask that forces each token to attend only to itself and tokens before it, never tokens that come later.

scores = (Q @ K.T) / sqrt(d_k)
scores = scores.masked_fill(future_positions, -inf)   # block attention to later tokens
weights = softmax(scores, axis=-1)                      # -inf becomes exactly 0 after softmax
output = weights @ V

Setting future-position scores to negative infinity before softmax guarantees their resulting attention weight is exactly zero — a token generating its own prediction can never "peek" at tokens that haven't been generated yet. This single change is what makes autoregressive generation coherent: predicting each next token uses only what has genuinely already been produced.

Why Decoder-Only Became the Dominant General-Purpose Shape

This also explains the simplicity advantage: one stack, one training objective (predict the next token), rather than BERT's separate masked-prediction objective or T5's split encoder/decoder training. Because generating a coherent next token requires the model to have effectively "understood" everything so far, useful general-purpose representations emerge as a side effect of the generative objective alone — no separate understanding-specific objective needs to be designed in.

The GPT Lineage

This decoder-only, causally-masked architecture is the actual mechanism behind historyai3-7's own GPT-1 through GPT-3 lineage namecheck — the architecture story, not yet the training story. How pretraining at real scale actually works is llm1-7's own job; how GPT-3 became ChatGPT through fine-tuning and RLHF is llm1-9's.

Hands-On Exercises