What NLP Actually Is & The Text Preprocessing Pipeline

NLP

Chapter 1 · What NLP Actually Is & The Text Preprocessing Pipeline

ds1 taught data handling, ml1 taught classical models, nn1 taught neural architectures — all three, so far, applied to numbers and categories. This course applies all of it to text, a genuinely messier, more ambiguous kind of data than anything the first three courses ever handed you.

Why Text Is ds1-4's Own Data-Cleaning Material, Taken to Its Extreme

ds1-4 covered missing values, duplicates, inconsistent formatting, and string cleaning — real problems, but confined to a handful of structured columns with a known, fixed shape. A single paragraph of raw text has every one of those same problems and none of the fixed structure: no defined length, no fixed vocabulary, genuine ambiguity (the same word meaning different things in different contexts), and grammar and word order that carry real information a plain column of numbers never has to represent at all.

The Text Preprocessing Pipeline

  • Tokenization — splitting raw text into individual units.
  • Lowercasing — normalizing case.
  • Stopword removal — dropping very common, low-information words.
  • Stemming or lemmatization — reducing words to a base form.

    • Tokenization

    "I don't think it's working." → ["I", "do", "n't", "think", "it", "'s", "working", "."]

    Splitting text into tokens — usually words, sometimes subwords or characters — is the first, foundational step everything else builds on.

    Not a solved, trivial problem everywhere
    Contractions like don't already force a real decision (one token or two?). Languages without whitespace between words at all — Chinese, Japanese — need a genuinely different tokenization approach entirely. Tokenization looks trivial in English and stops looking trivial the moment the assumptions behind it are questioned even slightly.

    Lowercasing

    Treating "The" and "the" as the same token reduces vocabulary size and noise — genuinely useful for most tasks, and a real, honest trade-off worth naming: "US" (the country) and "us" (the pronoun) collapse into the same token once lowercased, quietly discarding a real distinction some tasks would actually need.

    Stopword Removal

    Words like "the", "a", "is" appear constantly and usually carry little distinguishing information — removing them reduces noise for many tasks.

    A real, important gotcha — "not" is a stopword too
    A standard stopword list typically includes negation words like "not". Removing it from "I do not like this movie" leaves something dangerously close to the opposite sentiment. For any task where negation genuinely matters — sentiment analysis very much included — indiscriminate stopword removal can silently flip a sentence's own meaning. This isn't a hypothetical edge case; it's a real, well-documented trap.

    Stemming vs. Lemmatization

    How it worksExample
    StemmingCrude, rule-based chopping of common suffixes — fast, no dictionary needed"running" → "run" — but an aggressive stemmer can produce non-words, e.g. "running" → "runn"
    LemmatizationDictionary/grammar-aware reduction to a genuine root word — slower, more accurate"better" → "good" — something pure rule-based stemming could never do at all

    What This Pipeline Doesn't Do Yet

    After all four steps, you have a cleaner set of tokens — still just words, not numbers. Every model this course eventually uses, from ml1's own classifiers to nn1's own networks, needs numeric input. nlp1-2 covers the first real way to get there.

    This Course's Own Roadmap

    ChapterDelivers
    nlp1-2 / nlp1-3Turning cleaned text into numbers a model can use
    nlp1-4What that numeric representation still loses
    nlp1-5 / nlp1-6Fixing meaning (embeddings) and order (sequence models) separately
    nlp1-7A real, practical NLP task — entity/tag recognition
    nlp1-8 / nlp1-9Why attention mattered for NLP specifically, and pretrained models in practice
    nlp1-10A real pipeline, and why LLMs are a genuinely different approach

    Hands-On Exercises

    Exercise 1

    Explain, using this chapter's own comparison to ds1-4, why text is described as data cleaning's "most extreme case yet" rather than simply a different kind of data.

    📄 View solution
    Exercise 2

    Using this chapter's own warn-box, explain concretely why removing "not" as a stopword is a real, serious risk for a sentiment-analysis task specifically, with a worked example.

    📄 View solution
    Exercise 3

    Using this chapter's own "better" → "good" example, explain why lemmatization can do something stemming structurally cannot, and identify what stemming would most plausibly produce for the same word instead.

    📄 View solution

    Chapter 1 Quick Reference

    • Text is ds1-4's own data-cleaning problem, with no fixed structure and genuine ambiguity added on top
    • Tokenization — splitting text into units; genuinely hard in some languages, not just an English afterthought
    • Lowercasing — reduces noise, but can collapse real distinctions ("US" vs. "us")
    • Stopword removal — reduces noise, but can remove negation words like "not," silently flipping meaning
    • Stemming (crude, rule-based) vs. lemmatization (dictionary-aware, can map "better" → "good")
    • Cleaned tokens still aren't numbers — Next chapter: Bag-of-Words & Text Vectorization