Embeddings & Positional Encoding at Scale

nlp1-8 closed with an honest, deliberately unresolved gap: self-attention has no inherent sense of token order. This chapter delivers the fix in full — and along the way, revisits what an "embedding" even means once it's trained as part of one enormous model rather than as a separate step.

Token Embeddings — Trained End-to-End, Not Frozen

Every token ID produced by llm1-2's own BPE vocabulary maps to a learned vector through an embedding lookup table — mechanically similar to nlp1-5's own trainable nn.Embedding. The real difference is what trains it: nlp1-9's own GloVe vectors were pretrained separately and then either frozen or fine-tuned as a distinct step. An LLM's own token embeddings are trained jointly, from the very start, alongside every other parameter in the entire network — there is no separate embedding-training phase at all.

The Problem This Chapter Actually Solves

Per nlp1-8: swap two tokens in a sequence, and self-attention computes the exact same set of pairwise relationships either way. Nothing about the raw token embeddings, on their own, encodes where in the sequence a token appears. Positional encoding exists to inject that missing information before attention ever runs.

Sinusoidal Positional Encoding — The Original Fix

The original Transformer paper's own solution: a fixed, non-learned pattern of sine and cosine waves, one pair of frequencies per pair of embedding dimensions, added directly to each token's embedding based on its position.

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

# pos = the token's position in the sequence (0, 1, 2, ...)
# i   = which pair of embedding dimensions this is
# d   = the total embedding dimension

Each position gets a unique, deterministic "fingerprint" — no two positions ever produce the same pattern. The specific choice of sine/cosine pairs has a genuinely elegant mathematical property: the encoding for any fixed relative offset (position p+k relative to position p) can be expressed as a simple linear transformation of the encoding at position p, which makes it easier for the model to learn to attend by relative position, not just absolute position.

Learned Positional Embeddings — GPT's Own Approach

GPT and many later models take a simpler route: instead of a fixed formula, learn a separate embedding vector for each position index (0 through some maximum), trained jointly with everything else, exactly like the token embeddings themselves.

Combining Token and Position

token_embedding = embedding_table[token_id]           # llm1-2's token, looked up
position_embedding = positional_encoding[position]     # sinusoidal or learned
input_vector = token_embedding + position_embedding     # what actually enters layer 1

Revisit nlp1-4's own "dog bites man" vs. "man bites dog" example. Before any attention runs at all, "dog" at position 0 and "dog" at position 2 (in the reversed sentence) now receive genuinely different input vectors, purely because their positional components differ — the exact information self-attention alone could never supply on its own, now supplied directly at the input.

Hands-On Exercises