TF-IDF — Weighting Words by Distinctiveness

NLP

Chapter 3 · TF-IDF — Weighting Words by Distinctiveness

nlp1-2's own raw counts treat every word's occurrence as equally meaningful. It isn't — a word appearing constantly across every document tells you almost nothing about what makes any one of them distinctive.

Why Raw Counts Alone Are Misleading

A word like "now" might appear often in a spam message — and just as often in an ordinary one. Its raw count is high, but it does little to actually distinguish spam from not-spam. A genuinely rare word like "viagra", by contrast, might appear only once — but that single occurrence is far more informative about what kind of message this is.

TF — Term Frequency

How often a word appears within this document, typically normalized by the document's own total word count — ds1-6's own proportion vocabulary, applied directly: not a raw count, but a rate, so a short and a long document can be compared fairly.

TF(word, doc) = (count of word in doc) / (total words in doc)

IDF — Inverse Document Frequency

How rare a word is across the whole corpus. A word appearing in nearly every document gets a low IDF (not distinctive); a word appearing in only a handful gets a high IDF (distinctive).

IDF(word) = log( total documents / documents containing word )

The logarithm is a deliberate dampener — without it, an extremely rare word's own score could grow disproportionately large and dominate everything else; the log keeps the scale reasonable while still rewarding rarity.

TF-IDF = TF × IDF

A word scores high when it's frequent in this document and rare across the corpus overall — genuinely distinctive for this specific document. It scores low when it's either rare here or common everywhere.

Word	Frequent in this doc?	Common across corpus?	TF-IDF score
"now" (appears in most messages)	Yes	Yes	Low
"viagra" (appears rarely, distinctively)	Yes, here	No	High

Built entirely from ds1-6's own vocabulary

TF is a rate — the exact "proportion" concept ds1-6 already defined. IDF's own document-count ratio is the same "how common is this across the whole population" reasoning ds1-6 used for its own statistics, just applied to documents containing a word rather than data points sharing a value.

In Practice — A Drop-In Replacement

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)   # same shape, same downstream ml1-5 workflow

TfidfVectorizer slots directly into nlp1-2's own pipeline in place of CountVectorizer — everything downstream, including ml1-5's own LogisticRegression.fit(), is unchanged.

What TF-IDF Does Not Fix

Still bag-of-words underneath

TF-IDF only changes how strongly each word's own count is weighted — it's still fundamentally a bag-of-words representation, and nlp1-2's own word-order loss applies completely unchanged. "dog bites man" and "man bites dog" still produce identical TF-IDF vectors — the exact same words, the exact same counts, just each one now weighted by distinctiveness rather than left as a raw count. TF-IDF solves one narrow problem (unequal word importance); it does nothing at all for the order problem nlp1-4 covers next.

Hands-On Exercises

Exercise 1

Using this chapter's own IDF formula, explain why a word appearing in every single document in a corpus receives an IDF score of exactly zero, and explain what that means for its overall TF-IDF score regardless of how frequently it appears in any one document.

📄 View solution

Exercise 2

Explain, using this chapter's own comparison table, why "viagra" and "now" can both be frequent within a specific document yet receive dramatically different TF-IDF scores.

📄 View solution

Exercise 3

Explain, using this chapter's own warn-box, why "dog bites man" and "man bites dog" still produce identical TF-IDF vectors, and explain precisely what problem TF-IDF solves versus what problem it leaves completely untouched.

📄 View solution

Chapter 3 Quick Reference

TF — a word's own rate within one document (ds1-6's own proportion vocabulary)
IDF — log(total docs / docs containing the word) — rarer across the corpus means a higher score
TF-IDF = TF × IDF — high only when frequent here and rare elsewhere
TfidfVectorizer — a drop-in replacement for nlp1-2's own CountVectorizer, same downstream ml1-5 workflow
Still bag-of-words underneath — word order is completely unaffected; "dog bites man" = "man bites dog," unchanged from nlp1-2
Next chapter: The Limits of Bag-of-Words