Transformers
🔍 Transformers
🔄 The RNN/LSTM Bottleneck
Before this chapter, sequence tasks like machine translation or language modeling were dominated by recurrent neural networks (RNNs) and their more capable successor, the LSTM (Long Short-Term Memory) network, introduced by Hochreiter and Schmidhuber back in 1997. Both process a sequence one element at a time — reading a sentence word by word, carrying forward a running internal state — which creates two real problems. First, that step-by-step processing can't be parallelized across the sequence during training, making it inherently slow on the GPU hardware Chapter 4 showed off. Second, information from early in a long sequence tends to fade by the time the network reaches the end, a variant of Chapter 3's vanishing gradient problem showing up again in a new form.
👀 Attention, Before Transformers
In 2014, researchers including Dzmitry Bahdanau introduced an attention mechanism as an add-on to RNN-based translation models — instead of forcing the entire input sentence into one fixed-length summary vector, the model learned to "attend" to the most relevant input words while producing each output word. It was a genuine improvement to translation quality, but still bolted onto the same slow, sequential RNN backbone.
⚡ "Attention Is All You Need" (2017)
A team of researchers at Google — Ashish Vaswani, Noam Shazeer, and colleagues — took attention's role a step further and published a paper with a title that stated its own thesis outright: "Attention Is All You Need." Their proposed architecture, the Transformer, discarded recurrence entirely. No RNN, no LSTM, no step-by-step processing at all — just self-attention layers and feedforward layers, applied to the whole sequence at once.
Removing recurrence had an immediate practical payoff: because every position in the sequence could now be processed simultaneously rather than one after another, Transformers could be trained far more efficiently on the parallel GPU hardware that had already proven decisive in Chapter 4. Bigger models, trained on bigger datasets, in less time, became practical almost immediately.
🧩 How Self-Attention Works
At a high level, self-attention lets every element in a sequence directly compute how relevant every other element is to it, regardless of distance — a word at the start of a paragraph can influence a word at the end just as directly as its immediate neighbor, solving the long-range dependency problem RNNs struggled with. Multi-head attention runs several of these relevance computations in parallel, each free to specialize in a different kind of relationship (grammatical structure, topic relevance, and so on). And because dropping recurrence also means the model has no inherent sense of word order, Transformers add explicit positional encoding — extra information injected into each input telling the model where in the sequence it sits.
The Transformer's debut benchmark was, fittingly, machine translation — the same task whose 1966 rule-based failure (Course 2, Chapter 5's ALPAC Report) triggered AI's first funding winter, and whose 1990s statistical revival (this course's Chapter 1) gave that story its redemption arc. Transformers set new state-of-the-art results on standard English-German and English-French translation benchmarks, while training substantially faster than the RNN-based systems they replaced — a third chapter, decades apart, in the same task's ongoing story.
📜 Sequence Models, Before and After
| RNN / LSTM (Before) | Transformer (After) |
|---|---|
| Processes a sequence step by step, in order | Processes the entire sequence simultaneously |
| Training is inherently sequential — hard to parallelize | Highly parallelizable — trains efficiently on GPU hardware |
| Long-range relationships fade over distance | Self-attention connects any two positions directly, regardless of distance |
Almost every technique covered so far in this course was built for one kind of problem — SVMs and Bayesian networks for statistical classification, convolutional networks for images, Monte Carlo Tree Search for board games. The Transformer turned out to be startlingly general-purpose: the same core architecture went on to power vision models (Vision Transformers, applied to images rather than text), DeepMind's AlphaFold (protein structure prediction — the same research lab from Chapter 5's AlphaGo), audio processing, and, most consequentially for the next chapter, large language models. Very few ideas in this course's history generalized this far beyond the problem they were originally built to solve.
🤔 Questions to Sit With
"Attention Is All You Need" is a notably confident title for a research paper. Having tracked this course's recurring pattern of bold claims and their consequences (the Dartmouth proposal, expert systems' commercial promises), does a title this bold make you more skeptical going in, or did the Transformer's actual impact simply earn it?
The Transformer solved RNNs' long-range dependency problem by letting every position attend directly to every other position, rather than passing information step by step. Can you think of a human process or organization that struggles with the same "information fades over distance" problem RNNs had?
Unlike most breakthroughs in this course, the Transformer turned out to generalize far beyond its original task. Do you think that generality was a foreseeable consequence of the architecture's design, or a genuine surprise even to the researchers who built it?
🎯 What's Next
Next chapter: The GPT Lineage & the ChatGPT Moment — how this chapter's architecture became the foundation for large language models, and the specific product moment in 2022 that put them in front of hundreds of millions of ordinary people almost overnight.