Bag-of-Words & Text Vectorization

NLP

Chapter 2 · Bag-of-Words & Text Vectorization

nlp1-1 left off with clean tokens — still just words. This chapter turns them into the one thing every model since ml1-3 has actually required: numbers.

The Bag-of-Words Idea

Build a vocabulary — every unique word across the whole corpus. Represent each document as a vector with one position per vocabulary word, where the value is how many times that word appears in that document. "Bag" because order is discarded entirely — as if every word were tipped out of the document into an unordered bag and only the counts were kept. 

Doc 1: "free money now"
Doc 2: "call me now"

Vocabulary: [call, free, me, money, now]

Doc 1 vector: [0, 1, 0, 1, 1]
Doc 2 vector: [1, 0, 1, 0, 1]

Feeding This Directly Into ml1-5's Own Logistic Regression

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)   # exactly the "X" ml1-3/ml1-5 already used
y = labels                                # spam / not spam

model = LogisticRegression()
model.fit(X, y)                            # ml1-5's own workflow, unchanged
Text is now just tabular data
Each vocabulary word is now literally a column, exactly like ml1-3's own mileage and year columns, or ml1-5's own salary and age. Nothing about LogisticRegression.fit() changes at all — it has no idea the columns started life as words rather than numeric measurements.

Two Real, Practical Details

Sparse Vectors

A real corpus can easily have a vocabulary of tens of thousands of unique words — but any single document only ever uses a tiny fraction of them. The resulting vectors are overwhelmingly zeros, a property called sparsity. This is a genuinely different practical concern from ds1's own typically dense numeric tables, though most libraries (including CountVectorizer) store and compute with sparse vectors efficiently rather than wastefully storing every zero.

Words Never Seen During Training

The vocabulary is built from the training data. A brand-new document at prediction time containing a word the vocabulary has never seen simply has nowhere to put it — that word is typically dropped entirely, silently. A real, honest limitation worth naming plainly.

What This Chapter Deliberately Doesn't Fix

Order is gone — on purpose, for now
Nothing about a bag-of-words vector distinguishes "dog bites man" from "man bites dog" — both produce the identical set of word counts. This isn't an oversight; nlp1-4 demonstrates exactly why this matters, concretely, before nlp1-5 and nlp1-6 separately fix it.

Hands-On Exercises

Exercise 1

Using this chapter's own two-document example, build the bag-of-words vector for a third document, "free call now," and explain how you determined each position's value.

📄 View solution
Exercise 2

Explain, using this chapter's own finding-box, precisely why LogisticRegression.fit() requires no code changes at all to work with bag-of-words vectors instead of ml1-3's own numeric features.

📄 View solution
Exercise 3

Explain what sparsity means for a bag-of-words vector, and explain why a real corpus's own vocabulary size makes sparsity essentially unavoidable rather than a rare edge case.

📄 View solution

Chapter 2 Quick Reference

  • Bag-of-words — one vector position per vocabulary word, value = count in that document, order discarded entirely
  • Feeds directly into ml1-5's own LogisticRegression.fit() — text becomes ordinary tabular data, one column per word
  • Sparsity — real vocabularies are huge; any one document's own vector is overwhelmingly zeros
  • Words unseen during training simply can't be represented at prediction time — dropped silently
  • Word order is completely lost — deliberately left unresolved, demonstrated concretely in nlp1-4
  • Next chapter: TF-IDF — Weighting Words by Distinctiveness