Pretrained Embeddings & Transfer Learning

NLP

Chapter 9 · Pretrained Embeddings & Transfer Learning

nlp1-5 trained word2vec embeddings from scratch, on whatever corpus happened to be on hand. This chapter asks the obvious next question: what if, instead, someone else already trained embeddings on a corpus vastly larger than any single project could gather — and all that's needed is to reuse them?

The Cold-Start Problem nlp1-5 Quietly Assumed Away

Word2vec's own skip-gram/CBOW training (nlp1-5) needs a large corpus and real training time to learn good relationships — "excellent" only ends up near "great" in vector space after seeing both words in enough similar contexts, many times over. A small project's own dataset — the spam examples from nlp1-2, or the toy sentences carried through nlp1-6/nlp1-7 — is nowhere near large enough to learn quality embeddings from scratch. Training on too little text produces noisy, unreliable vectors no matter how correct the algorithm is.

GloVe — A Genuinely Different Training Approach

GloVe (Global Vectors, Stanford) is the other major pretrained-embedding family alongside word2vec, and it gets there differently. Word2vec learns from local context windows, one sliding prediction task at a time. GloVe instead builds a full word-by-word co-occurrence matrix across an entire corpus (how often does word A appear near word B, counted globally) and factorizes that matrix directly to produce vectors — a genuinely different mathematical approach reaching a similar kind of representation. Trained on massive corpora (Wikipedia, Common Crawl — billions of words), the resulting vectors are freely downloadable and ready to use without any training of your own.

Transfer Learning — A Pattern, Not Just an NLP Trick

Reusing embeddings trained on one (huge, general) task for a different (smaller, specific) task is an instance of transfer learning — a general pattern, not unique to text. nn1-7's own AlexNet, trained on ImageNet's millions of labeled images, produces internal features so generally useful that they get reused as a starting point for entirely different image tasks the network was never trained on. Pretrained word embeddings are the same idea in a different modality: a representation learned once, on a huge general corpus, reused as a head start rather than relearned from nothing.

Frozen vs. Fine-Tuned

There are two honest ways to use a pretrained embedding layer:

Approach	What happens during training	When it's the right call
Frozen	Pretrained vectors loaded, never updated	Small task-specific dataset — not enough data to safely specialize further
Fine-tuned	Pretrained vectors loaded as a starting point, then updated during training	Enough task-specific data to adapt vectors to this domain without losing what was already learned

# frozen — the classic transfer-learning move for a small dataset
embedding = nn.Embedding.from_pretrained(glove_vectors, freeze=True)

# fine-tuned — start from GloVe, keep adapting during training
embedding = nn.Embedding.from_pretrained(glove_vectors, freeze=False)

Compare this to nlp1-5's own nn.Embedding(vocab_size, embed_dim) — randomly initialized, learned entirely from this project's own limited data. Loading GloVe's own pretrained values instead means training starts from vectors that already encode real semantic relationships, rather than from random noise.

The Honest Limitation: Out-of-Vocabulary Words

A fixed, pretrained vocabulary

GloVe's own vocabulary is fixed at whatever it was trained on. Domain-specific jargon, brand names, typos, or genuinely novel words simply aren't in it — these out-of-vocabulary (OOV) words get no meaningful vector at all, typically falling back to a generic "unknown" placeholder that carries none of the real word's own meaning. A pretrained embedding is a head start, not a complete solution for every dataset.

The Bridge Into llm1

The same shape, at a much larger scale

This chapter's own frozen-vs-fine-tuned distinction is the exact same shape as the pretrain-then-finetune paradigm llm1 covers in full — a model trained once on a massive, general task, then either used as-is or further adapted on a smaller, specific one. The only real difference is scale: here it's a lookup table of word vectors; there it's an entire deep network with billions of parameters. The underlying logic — don't relearn from nothing what's already been learned well elsewhere — is identical.

Hands-On Exercises

Exercise 1

Explain why nlp1-5's own from-scratch word2vec training would likely produce noisy, unreliable vectors if trained only on the small spam or toy datasets used earlier in this course, and explain specifically what pretrained embeddings like GloVe fix about this.

📄 View solution

Exercise 2

Explain the genuine parallel between this chapter's own use of pretrained GloVe embeddings and nn1-7's own use of AlexNet features trained on ImageNet, and explain why both count as transfer learning despite operating on entirely different kinds of data.

📄 View solution

Exercise 3

Explain when freezing pretrained embeddings is the right choice versus fine-tuning them, and explain specifically why this chapter's own frozen-vs-fine-tuned distinction is described as the same underlying shape as llm1's own pretrain-then-finetune paradigm.

📄 View solution

Chapter 9 Quick Reference

The cold-start problem — nlp1-5's own from-scratch training needs a large corpus; small task-specific datasets can't reliably supply one
GloVe — global co-occurrence-matrix factorization, a genuinely different training approach from word2vec, trained on massive corpora and freely downloadable
Transfer learning — reusing a representation learned on a huge general task for a smaller specific one; nn1-7's ImageNet-trained AlexNet features are the same pattern in a different modality
Frozen — pretrained vectors never updated, best for small datasets · Fine-tuned — pretrained vectors continue updating, best with enough task-specific data
OOV words — domain jargon/novel words missing from GloVe's fixed vocabulary get no meaningful vector
This chapter's frozen-vs-fine-tuned split is the same shape, at far smaller scale, as llm1's own pretrain-then-finetune paradigm
Next chapter: Capstone — Building a Real NLP Pipeline & Why LLMs Are Different