Tracing a Prompt Through a Real LLM, End to End

llm1-1 opened with a deliberately incomplete preview of "The capital of France is", promising the full trace would wait until every piece was built. Every piece is now built. Here is the full trace.

Step 1 — Tokenization (llm1-2)

prompt = "The capital of France is"
tokens = ["The", " capital", " of", " France", " is"]   # BPE subword tokens
token_ids = [464, 3139, 286, 4881, 318]                    # looked up in the vocabulary

llm1-2's own Byte-Pair Encoding breaks the raw prompt string into subword tokens, each mapped to an integer ID in the model's fixed vocabulary — no out-of-vocabulary risk, per that chapter's own byte-level fallback guarantee.

Step 2 — Embedding & Positional Encoding (llm1-3)

x = embedding_table[token_ids]              # each token ID -> its learned vector
x = x + positional_encoding[0:5]              # position 0..4 added, per token

Each token ID is looked up in the jointly-trained embedding table, then combined with its own positional encoding — the exact mechanism that gives the model any sense of order at all, resolving nlp1-8's own deferred gap. This produces the actual vectors that enter the first Transformer block.

Step 3 — Multi-Head Self-Attention Across Stacked Layers (llm1-4 / llm1-5 / llm1-6)

for layer in range(num_layers):                     # dozens of stacked blocks
    x = x + multi_head_attention(layer_norm(x), causal_mask=True)
    x = x + feed_forward(layer_norm(x))

At every layer, each token's own vector is reshaped by causally-masked (llm1-6) multi-head attention (llm1-5), built from the Query/Key/Value mechanism (llm1-4). The token at position 4 (" is") can attend to itself and every earlier token — never anything later, since nothing later exists yet in this generation step. By the final layer, its own vector has been shaped by learned relevance to "The", "capital", "of", and "France", repeatedly, across every stacked block.

Step 4 — Next-Token Prediction (llm1-7 / llm1-8)

logits = final_layer_output[-1] @ output_projection    # project to vocabulary size
probabilities = softmax(logits)
# probabilities["Paris"] ≈ highest, if pretraining (llm1-7) at sufficient scale (llm1-8) went well

The final vector at the last position is projected into a probability distribution over the entire vocabulary — llm1-7's own causal language modeling objective, at generation time. If the model was pretrained on enough data, at enough scale, following llm1-8's own measured power-law relationship, "Paris" receives the highest probability of any token in the vocabulary.

Step 5 — The Autoregressive Loop (llm1-6)

The predicted token is appended to the sequence, and the entire process — embed, attend across every layer, predict — repeats to generate the token after that, one token at a time, each one causally valid given only what has genuinely been generated so far.

An Honest Aside: Real Deployments Aren't Raw Completions

Chapter Attribution Table

Closing the Loop on nlp1-10's Three Claims

This is also the LLM analogue of what imgai1 did for image models, exactly as llm1-1 set out — a full mechanism, traced end to end, sitting underneath prompt1's and claude-adv1's own practical territory rather than replacing it.

Hands-On Exercises

Scope Note — What This Course Deliberately Doesn't Cover