From RNNs to Attention — NLP's Own Motivation for Transformers

nn1-9 previewed self-attention conceptually and deliberately deferred full depth to llm1. This chapter doesn't skip ahead of that plan — it grounds the preview in the real, historically accurate problem attention was invented to solve, which happens to be an NLP problem: machine translation.

Seq2Seq: nlp1-6's Own Architecture, Turned Toward Generation

Translating a sentence is naturally framed as sequence-to-sequence: an encoder LSTM reads the entire source sentence, and a decoder LSTM generates the target sentence word by word. Look closely at the encoder's own job, though — it is doing exactly what nlp1-6's classifier did: processing a whole sequence and compressing it down into a single final hidden state. The only difference here is what happens to that summary afterward — instead of feeding it to a sigmoid, it's handed to a second LSTM as its own starting point.

The Fixed-Vector Bottleneck

For a short sentence, squeezing everything into one vector works reasonably well. For a long sentence, it's a genuine problem: the encoder's final hidden state has to somehow carry everything the decoder will need — every noun, every clause, every dependency — through one fixed-size vector. Early words in a long sentence get progressively diluted by every step that comes after them, the same accumulating-history mechanism nlp1-6 relied on to preserve order now works against fidelity at length.

Attention: Let the Decoder Look Back

Bahdanau's 2014 fix (pre-dating the 2017 Transformer paper by three years) was simple in concept: instead of forcing the decoder to work from one final encoder hidden state, let it look back at every encoder hidden state at every step it generates, and compute a weighted combination — attending more heavily to whichever source words are most relevant to the word being generated right now.

# conceptual attention score at one decoder step
scores = [dot(decoder_state, encoder_state_i) for encoder_state_i in encoder_states]
weights = softmax(scores)                     # how much to "attend" to each source word
context = sum(w * s for w, s in zip(weights, encoder_states))   # weighted blend, not one fixed vector

Translating "the black cat" into French, the decoder generating "noir" (black) would learn to assign a high attention weight to the source word "black" and comparatively little to "the" or "cat" — a direct, interpretable link between output and source, something the single fixed vector could never expose.

Self-Attention: The 2017 Leap

The real leap in "Attention Is All You Need" (2017) — the paper historyai3-6 already namechecked — was applying this exact same mechanism within a single sequence, not just from decoder back to encoder. In self-attention, every token computes attention weights against every other token in the same sequence, building a representation of each word informed by every other word it's relevant to — no recurrence required at all.

What's Still Deferred

nn1-9 already flagged this honestly, and it's still true here: self-attention has no inherent sense of token order — swap two tokens and the set of pairwise relationships computed is identical. A real Transformer needs positional encoding to reintroduce order information, and the full multi-head, multi-layer Transformer architecture built from self-attention is genuinely substantial. Both are deliberately left for llm1's own full depth, exactly as planned.

Hands-On Exercises