Sequence Models for Text

nlp1-5 fixed meaning. This chapter fixes the other half of nlp1-4's own diagnosis — word order — by finally putting nn1-8's own RNN/LSTM material to real, concrete use on real text.

The Real Pipeline, End to End

Resolving nlp1-4's Own Cliffhanger, Concretely

nn1-8's own hidden-state update at every step depends on both the current input and the previous hidden state — order-dependent by construction. Feed the embeddings for "dog", "bites", "man" into the LSTM in that exact order, and the resulting final hidden state is genuinely, mathematically different from feeding "man", "bites", "dog" — because at every intermediate step, the hidden state carries forward a different history depending on what came before it.

A Real Worked Task — Sentiment Classification

import torch.nn as nn

class TextClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)   # nlp1-5's own embeddings
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)   # nn1-8
        self.output = nn.Linear(hidden_dim, 1)                  # nn1-1's own closing callback, again

    def forward(self, x):
        embedded = self.embedding(x)              # one embedding per token, in order
        _, (hidden, _) = self.lstm(embedded)        # the final hidden state — nn1-8's own "memory" of the whole sequence
        return torch.sigmoid(self.output(hidden[-1]))   # ml1-5's own logistic regression, one more time

Both Problems, Genuinely Solved Together

nlp1-4 diagnosed two independent problems — meaning and order — and insisted neither fix alone solves both. This chapter's own pipeline is the actual combination: nlp1-5's embeddings supply meaning, this chapter's own sequence model supplies order. Only together do they produce a representation that genuinely distinguishes "dog bites man" from "man bites dog" and recognizes "excellent" as similar to "great."

What's Next

This chapter's own model produces exactly one output for an entire sequence — a single sentiment label per document. nlp1-7 extends the identical underlying idea to produce one output per token instead — real, practical tasks like recognizing names and grammatical roles word by word.

Hands-On Exercises