A Transformer Preview

Neural Networks & Deep Learning

Chapter 9 · A Transformer Preview

nn1-8 closed on a real, unsolved bottleneck: LSTMs fixed vanishing gradients but never fixed RNNs' own inherently sequential computation. This chapter previews the architecture that solved that specific problem — and, along the way, delivers part of historyai3-6's own "Attention Is All You Need" namecheck. The rest is llm1's own job.

The Motivating Question

Can a network capture relationships between distant elements in a sequence — nn1-8's own hidden/cell state's whole purpose — without forcing every step to wait for the one before it?

Self-Attention — A Direct, All-at-Once Relationship

Instead of carrying a compressed hidden state step by step, self-attention lets every position in a sequence directly look at every other position simultaneously, computing a weighted combination based on how relevant each other position actually is. Nothing is mediated through a chain of intermediate steps — the relationship between position 1 and position 50 is computed exactly as directly as the relationship between two adjacent positions.

A concrete intuition

In "The cat sat on the mat because it was tired," resolving what "it" refers to requires directly relating it back to "cat." Self-attention lets the token "it" attend strongly and directly to "cat," regardless of how many words sit between them — no need to pass that information step-by-step through every intervening word's own hidden state, the way nn1-8's own RNN would have had to.

Solving Both of nn1-8's Own Open Problems

nn1-8's own open problem	How self-attention addresses it
Long-range dependencies	Direct attention between any two positions — no long chain of repeated multiplications (nn1-8's own vanishing-gradient mechanism) between distant positions at all
Sequential computation bottleneck	Attention for every position can be computed simultaneously, as one parallel operation — the exact structural fix nn1-8 closed on needing

The long-range-dependency fix is genuinely structural, not a patch layered on top the way nn1-8's own LSTM gating was — there's no long chain for a gradient to vanish across in the first place, because the relationship is computed directly rather than relayed through many intermediate steps. The parallelization fix directly resolves nn1-8's own closing cliffhanger, and aligns naturally with nn1-6's own GPU-parallel training.

A Deliberately Honest Gap — Order

Self-attention, by itself, has no inherent notion of sequence order at all — computing how much "it" attends to "cat" doesn't naturally encode whether "cat" came before or after "it" in the sentence. Unlike an RNN, which processes tokens strictly in order by construction, a transformer needs a separate, explicit mechanism to inject position information back in — positional encoding, one of several real technical pieces this preview deliberately leaves for llm1 to cover in full.

What's Deliberately Deferred to llm1

This is a preview, not the full mechanism

Multi-head attention, positional encoding in full, the encoder/decoder architecture, and the actual 2017 "Attention Is All You Need" paper this chapter's own title echoes — all of it is llm1's own dedicated job. This chapter delivers the concept and the motivating "why"; llm1 delivers the real mechanism and its role in modern language models.

Beyond Language

The same self-attention idea generalized well past its original text-focused motivation — vision transformers apply the identical mechanism to image patches instead of word tokens, a real, documented extension of the architecture beyond the problem it was originally built to solve. Not this course's own focus; worth knowing the architecture didn't stay confined to text.

Hands-On Exercises

Exercise 1

Using this chapter's own "it"/"cat" example, explain specifically why self-attention resolves that reference without repeating nn1-8's own step-by-step hidden-state relay, and why that difference matters for long sequences specifically.

📄 View solution

Exercise 2

Using this chapter's own compare-table, explain why the long-range-dependency fix is described as "genuinely structural" while nn1-8's own LSTM gating is described as more of a patch, even though both address the same underlying vanishing-gradient concern.

📄 View solution

Exercise 3

Explain why this chapter specifically flags positional encoding as a real gap self-attention has, rather than glossing over it, and explain why an RNN never needed an equivalent mechanism.

📄 View solution

Chapter 9 Quick Reference

Self-attention — every position directly attends to every other position at once, no relay through intermediate steps
Fixes nn1-8's own two open problems: long-range dependencies (structurally, not patched) and the sequential-computation bottleneck (parallel by construction)
A real, honest gap: self-attention alone has no inherent sense of order — positional encoding fixes this, deferred to llm1
Multi-head attention, the encoder/decoder architecture, and the real 2017 paper are all llm1's own dedicated job
The same mechanism generalized beyond text — vision transformers, briefly noted
Next chapter: A Framework Tour: PyTorch vs. TensorFlow