The Limits of Bag-of-Words

NLP

Chapter 4 · The Limits of Bag-of-Words

nlp1-2 and nlp1-3 both flagged the same loss and moved on. This chapter stops and actually proves it, then reveals a second, entirely separate blind spot neither chapter mentioned at all.

Problem 1: Word Order — Proven, Not Just Asserted

Vocabulary: [bites, dog, man]. 

"dog bites man" → [1, 1, 1]
"man bites dog" → [1, 1, 1]

Identical vectors. Not approximately similar — identical, down to the last digit, whether built with nlp1-2's own raw counts or nlp1-3's own TF-IDF weights. A model trained on either representation has no way whatsoever to distinguish these two sentences, because as far as its own input is concerned, they aren't two sentences at all — they're the exact same input, twice.

Why journalism has a name for this exact inversion
"Dog bites man" is famously described in journalism as not news — ordinary, unremarkable. "Man bites dog" is the same three words, reordered, describing something genuinely surprising and newsworthy. A representation that treats these as identical isn't failing at some minor technicality — it's failing at exactly the distinction a human reader would consider most important.

Problem 2: No Notion of Meaning — A Genuinely Separate Issue

Every vocabulary word is its own independent, arbitrary dimension. "good" and "great" are just two unrelated vector positions — as far as the representation itself is concerned, "good" is exactly as similar to "great" as it is to "car" or "purple". Nothing about bag-of-words or TF-IDF encodes synonymy or semantic closeness at all.

A concrete, practical consequence
A sentiment classifier trained mostly on reviews using the word "great" has no built-in way to recognize "excellent" as equally positive if "excellent" rarely appeared during training — the model can only generalize from the literal, specific words it happened to see, never from what those words actually mean.

This also reframes nlp1-2's own out-of-vocabulary limitation as a direct symptom of this deeper problem, not just an annoying edge case: a genuinely new word like "excellent" gets silently dropped rather than recognized as close in meaning to "great," precisely because nothing in the representation has any concept of "close in meaning" to begin with.

Two Separate Problems, Two Separate Fixes

Problem 1

Word Order

Fixed by nlp1-6 — sequence models, revisiting nn1-8's own RNN/LSTM material, applied to text for real.

Problem 2

Word Meaning

Fixed by nlp1-5 — word embeddings, representing words as dense vectors positioned by genuine semantic closeness.

Neither fix alone solves both problems
Fixing meaning doesn't fix order, and fixing order doesn't fix meaning — they're genuinely independent failures with genuinely independent solutions. Even after nlp1-5's own embeddings arrive, order remains completely unaddressed until nlp1-6 adds a sequence model on top.

Hands-On Exercises

Exercise 1

Using this chapter's own vocabulary and vectors, explain precisely why "dog bites man" and "man bites dog" produce mathematically identical vectors, not merely similar ones, under both nlp1-2's and nlp1-3's own representations.

📄 View solution
Exercise 2

Explain why this chapter reframes nlp1-2's own out-of-vocabulary limitation as "a direct symptom" of the meaning problem rather than a separate, unrelated issue.

📄 View solution
Exercise 3

Explain why this chapter insists the order problem and the meaning problem require two genuinely separate fixes, using this chapter's own warn-box, and explain what would still be broken if only nlp1-5's own embeddings were applied without nlp1-6's own sequence modeling.

📄 View solution

Chapter 4 Quick Reference

  • Problem 1 — Word order: "dog bites man" and "man bites dog" produce mathematically identical vectors under both nlp1-2 and nlp1-3
  • Problem 2 — Word meaning: every word is its own unrelated dimension — "good" and "great" are as unrelated as "good" and "car"
  • Out-of-vocabulary words (nlp1-2) are a direct symptom of the meaning problem, not a separate edge case
  • Two genuinely independent problems, two genuinely independent fixes — neither alone solves both
  • Next chapter: Word Embeddings & Word2Vec (fixes meaning) · nlp1-6 fixes order