Word Embeddings & Word2Vec

NLP

Chapter 5 · Word Embeddings & Word2Vec

nlp1-4 diagnosed Problem 2: every word is its own isolated, unrelated dimension. This chapter fixes it — and reveals that the tool doing the fixing is something you already know how to build.

Word Embeddings — The Core Idea

Instead of a huge, sparse vector with one arbitrary dimension per vocabulary word, represent each word as a small, dense vector — commonly 100–300 numbers — learned so that semantically similar words end up genuinely close together in that vector space. nlp1-4's own example, resolved directly: in embedding space, "good" and "great" sit near each other; "good" and "car" sit far apart. A real, learned, geometrically meaningful relationship — not an arbitrary one.

Word2Vec — Learning This From Raw Text Alone

Word2vec learns embeddings self-supervised — no manual labeling at all, purely from patterns in how words co-occur across a huge corpus of ordinary, unlabeled text. Two real training setups:

Approach	Predicts
CBOW	The target word, from its surrounding context words
Skip-gram	The surrounding context words, from the target word

A deliberate forward-pointer to llm1

The "labels" here are just other words already present in the raw text — the task supervises itself. This is conceptually the same underlying idea (though vastly different in scale and architecture) as the self-supervised "predict the next token" pretraining llm1 covers for large language models. Word2vec is a genuine, small-scale ancestor of that approach, not an unrelated technique.

Word2Vec Is Literally nn1-1's Own Neuron, Generalized

Word2vec's own architecture is a small feedforward network, built entirely from nn1-1's own vocabulary: an input layer (a one-hot encoded word), a single hidden layer, and an output layer predicting the context or target word.

The detail most explanations skip

The embedding isn't some separate output the network produces — it is the hidden layer's own learned weight matrix, once training finishes. Look up a word's embedding, and you're literally reading off one row of weights this small network learned via nn1-5's own backpropagation. Nothing new was needed to build word2vec — it's nn1-1's own generalized layer stack, applied to exactly this self-supervised prediction task.

The Geometric Payoff — Honestly Framed

Trained on a large enough corpus, the resulting embedding space captures genuine semantic and syntactic relationships well enough that vector arithmetic becomes meaningful: the famous king − man + woman ≈ queen.

A real property, presented honestly

This is a genuine, documented, widely-cited illustration — not a fabricated demo. It's also a deliberately clean, popular example chosen because it works well for illustration. Real embeddings capture many subtler relationships beyond this one case, but the arithmetic doesn't hold with the same clean precision for every possible word relationship in practice. Treat it as a genuine, illustrative property of the learned space, not a guarantee that this exact kind of arithmetic works perfectly and universally.

In Practice — Pretrained, Not Trained From Scratch

Training word2vec from scratch needs a genuinely large corpus. In practice, pretrained embeddings — trained once, on massive text collections, by someone else — are typically used directly rather than retrained per project. nlp1-9 covers this properly; for now, know that it's the normal way embeddings actually get used.

What's Still Broken

nlp1-4's meaning problem is solved. Its other diagnosed problem — word order — is completely untouched by anything in this chapter. Averaging or otherwise combining word embeddings without regard to their sequence still loses order entirely. nlp1-6 is where that gets fixed.

Hands-On Exercises

Exercise 1

Explain the difference between CBOW and skip-gram, and explain why both count as genuinely self-supervised despite requiring no human-provided labels at all.

📄 View solution

Exercise 2

Using this chapter's own finding-box, explain precisely what a word's embedding actually is inside word2vec's own network, and explain why this chapter calls it "nn1-1's own generalized layer stack" rather than a new kind of model.

📄 View solution

Exercise 3

Explain why this chapter presents "king − man + woman ≈ queen" with an explicit caveat rather than as unqualified proof that embeddings perfectly capture all semantic relationships.

📄 View solution

Chapter 5 Quick Reference

Word embeddings — dense, small vectors where semantic closeness becomes geometric closeness, fixing nlp1-4's meaning problem
Word2vec — self-supervised: CBOW predicts a word from context, skip-gram predicts context from a word; no manual labeling — a genuine small-scale ancestor of llm1's own next-token pretraining
The embedding is the hidden layer's own weight matrix — word2vec is literally nn1-1's own neuron/layer stack, generalized
"king − man + woman ≈ queen" is real but illustrative — not a guarantee of perfectly clean arithmetic everywhere
Pretrained embeddings are the normal path in practice — full coverage in nlp1-9
Meaning: solved. Order: still completely unaddressed — Next chapter: Sequence Models for Text