Sentiment Analyzer: Classifying Real Text at Scale

What We're Building

The identical architecture nlp1-10 already built and proved works: preprocessing (nlp1-1) → frozen pretrained GloVe embeddings (nlp1-9) → an LSTM sequence classifier (nlp1-6). Nothing about the model changes here — what changes is scale: a real, freely available, canonical sentiment dataset instead of a toy example, and everything that comes with training on tens of thousands of real examples instead of a handful.

Step 1: Loading the Real Dataset

import pandas as pd

df = pd.read_csv("imdb_reviews.csv")   # columns: review, sentiment ("positive"/"negative")
print(df.shape)
print(df["sentiment"].value_counts())

The IMDb movie review dataset is the canonical benchmark for exactly this task — 25,000 real reviews, roughly balanced between positive and negative, freely available and genuinely unremarkable to work with (no scraping, no missing values, no cleaning pipeline needed here — that work belongs to Chapter 2's own dataset, not this one).

Step 2: Preprocessing — nlp1-1's Own Pipeline, Reused Directly

import re

def preprocess(text):
    text = text.lower()
    text = re.sub(r"[^a-z\s]", "", text)
    return text.split()

df["tokens"] = df["review"].apply(preprocess)

nlp1-1's own tokenization/cleaning approach, applied unchanged — nothing here needed to be redesigned for real data, since preprocessing was always meant to generalize beyond whatever toy examples originally demonstrated it.

Step 3: Building a Real Vocabulary

from collections import Counter

all_tokens = [tok for tokens in df["tokens"] for tok in tokens]
vocab = {word: i + 2 for i, (word, count) in enumerate(Counter(all_tokens).most_common(20000))}
vocab["<pad>"] = 0
vocab["<unk>"] = 1
print(f"Vocabulary size: {len(vocab)}")

Step 4: Loading Only the GloVe Vectors This Vocabulary Actually Needs

import numpy as np

embedding_dim = 100
embedding_matrix = np.zeros((len(vocab), embedding_dim))

with open("glove.6B.100d.txt", encoding="utf-8") as f:
    for line in f:
        parts = line.split()
        word = parts[0]
        if word in vocab:
            embedding_matrix[vocab[word]] = np.array(parts[1:], dtype="float32")

The full GloVe file (nlp1-9's own pretrained embeddings) contains vectors for roughly 400,000 words — loading every single one into memory just to use 20,000 of them would be genuine, avoidable waste. This loop reads the file once and keeps only the rows this specific vocabulary needs.

Step 5: The Exact Model From nlp1-6/nlp1-10 — Unchanged

import torch
import torch.nn as nn

class SentimentClassifier(nn.Module):
    def __init__(self, embedding_matrix, hidden_dim=128):
        super().__init__()
        self.embedding = nn.Embedding.from_pretrained(
            torch.tensor(embedding_matrix, dtype=torch.float32), freeze=True
        )
        self.lstm = nn.LSTM(embedding_matrix.shape[1], hidden_dim, batch_first=True)
        self.output = nn.Linear(hidden_dim, 1)

    def forward(self, x):
        embedded = self.embedding(x)
        _, (hidden, _) = self.lstm(embedded)
        return torch.sigmoid(self.output(hidden[-1]))

Step 6: Training — Now With Real Batching

from torch.utils.data import Dataset, DataLoader

def encode(tokens, max_len=200):
    ids = [vocab.get(t, 1) for t in tokens[:max_len]]
    return ids + [0] * (max_len - len(ids))

class ReviewDataset(Dataset):
    def __init__(self, df):
        self.X = [encode(t) for t in df["tokens"]]
        self.y = (df["sentiment"] == "positive").astype("float32").values

    def __len__(self):
        return len(self.X)

    def __getitem__(self, i):
        return torch.tensor(self.X[i]), torch.tensor(self.y[i])

loader = DataLoader(ReviewDataset(df), batch_size=64, shuffle=True)

model = SentimentClassifier(embedding_matrix)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.BCELoss()

for epoch in range(3):
    for X_batch, y_batch in loader:
        optimizer.zero_grad()
        preds = model(X_batch).squeeze()
        loss = loss_fn(preds, y_batch)
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch+1} done.")

Step 7: Evaluating on Real Reviews

sample_reviews = [
    "This movie was absolutely fantastic, best film I've seen all year.",
    "Waste of two hours. Terrible acting, worse plot.",
]
for review in sample_reviews:
    encoded = torch.tensor([encode(preprocess(review))])
    score = model(encoded).item()
    print(f"{score:.2f} — {review[:50]}...")

A genuinely satisfying moment this project earns that a toy example can't: real, freshly-written sentences, never seen during training, scored by a model trained on real human-written reviews rather than a handful of illustrative examples.

What's Next

Chapter 2: Customer Churn Predictor — the full ml1 depth Chapter 7 of the Beginner course deliberately declined to use: real feature engineering, a genuine model comparison, cross-validation, and the full precision/recall/F1 picture.