Context Windows, Quadratic Attention & Honest Limitations
LLMs
Chapter 10 · Context Windows, Quadratic Attention & Honest Limitations
llm1-3 flagged, and deferred, exactly why a context window has a hard limit. This chapter delivers that in full — and gives two of the most commonly hand-waved LLM behaviors, hallucination and knowledge cutoff, real mechanical explanations instead.
What a Context Window Actually Is
The maximum number of tokens (llm1-2's own BPE tokens) a model can process in one forward pass. Two genuinely separate mechanisms both cap this number:
- Positional encoding's own limit —
llm1-3's own deferred warning: a learned positional embedding table simply has no entry beyond whatever maximum position it was trained with. - The computational cost of self-attention itself — this chapter's own new material, and the deeper, more fundamental constraint.
Quadratic Attention Cost
Recall llm1-4's own scaled dot-product attention: Q @ K.T computes a similarity score between every pair of tokens. For a sequence of length N, that's an N × N matrix of scores — both the compute and the memory required scale with N², not N.
| Sequence length (N) | Score pairs (N²) | Relative to N=1,000 |
|---|---|---|
| 1,000 | ~1,000,000 | 1× |
| 2,000 (2× longer) | ~4,000,000 | 4× |
| 10,000 (10× longer) | ~100,000,000 | 100× |
llm1-5's own multi-head attention computes this same N×N score matrix independently per head, multiplying the cost further by the number of heads on top of the underlying N² scaling.
Hallucination, Explained Mechanically
Per llm1-7, a model is trained to predict the most probable next token given context — nothing more. It has no built-in mechanism for verifying factual truth, only for producing plausible-sounding continuations shaped by patterns learned during pretraining.
llm1-9's own fine-tuning/RLHF pipeline can and does train a model to say "I don't know" more often in situations where that response is reward-preferred — but that's a learned behavioral pattern layered on top via SFT and RLHF, not a fact-checking mechanism built into the base architecture from llm1-5 onward.
Knowledge Cutoff, Explained Mechanically
Per llm1-7, pretraining uses a fixed, finite corpus, collected up to some point in time. Everything the model "knows" is entirely encoded in its trained parameters, shaped exclusively by whatever text existed in that corpus. Nothing about the pretraining process gives the model any live, ongoing access to information published after its own training data was collected — there is no built-in internet connection, no update mechanism running in the background.
Anything that happened after the corpus's own collection date is simply absent from the patterns the model learned — not because the model is vaguely "unaware," but because that information never existed in any data used to shape its parameters at all. Retrieval-augmented generation, already covered in claude-adv1, works around this by injecting live external information directly into the prompt at inference time — a way of supplying missing context, not a way of updating the model's own trained parameters.
Closing the Loop Back to Chapter 1
llm1-1 opened by claiming this course would explain why prompt1's own techniques work, not replace them. Techniques like providing sources in a prompt, or explicitly instructing a model to say "I don't know" when uncertain, work precisely because of the mechanisms covered in this chapter — they compensate for a model with no built-in fact-checker and no live knowledge feed, exactly as described here, mechanically, rather than as an unexplained quirk.
Hands-On Exercises
Using this chapter's own table, explain why doubling a context window's length roughly quadruples attention's own computational cost, and explain why this is described as a real architectural constraint rather than an arbitrary product limit.
📄 View solutionExplain the mechanical reason hallucination happens, tracing it back to llm1-7's own training objective, and explain why RLHF-trained "I don't know" responses (llm1-9) don't contradict this explanation.
📄 View solutionExplain why knowledge cutoff is a direct, structural consequence of how pretraining works (llm1-7), and explain why retrieval-augmented generation is described as working around this limitation rather than fixing it.
📄 View solutionChapter 10 Quick Reference
- Context window limit — capped by both positional encoding's own max length (llm1-3) and attention's own quadratic cost (this chapter)
- Quadratic attention cost: N tokens → N² score pairs; doubling length roughly quadruples cost — a real, hard architectural ceiling
- Real mitigations (sparse/sliding-window attention, FlashAttention) reduce practical cost but don't eliminate N² scaling for full dense attention
- Hallucination: the model has no built-in fact-checker — only next-token plausibility (llm1-7); RLHF (llm1-9) layers "I don't know" as a learned behavior, not a core capability
- Knowledge cutoff: a direct consequence of training on a fixed, finite corpus (llm1-7) — RAG supplies live context at inference time, without updating the model's own parameters
- Closes the loop to llm1-1: this is exactly why prompt1's own techniques (sourcing, "say if unsure") work
- Next chapter: Capstone — Tracing a Prompt Through a Real LLM, End to End