Recurrent Neural Networks (RNNs) & LSTMs

Neural Networks & Deep Learning

Chapter 8 · Recurrent Neural Networks (RNNs) & LSTMs

nn1-7 covered data where 2D position matters. This chapter covers data where order matters instead — text, time series, audio — and delivers the real, worked-out version of nn1-4's own vanishing-gradient cliffhanger, in exactly the setting that makes it most severe.

Why Neither Feedforward Networks Nor CNNs Naturally Handle Sequences

Every network so far processes one fixed-size input and produces one output, with no notion of "what came before." Predicting the next word in a sentence genuinely needs context from every preceding word, not just a fixed local window — a fundamentally different requirement from anything nn1-1–nn1-7 were built to handle.

The RNN Idea — A Hidden State Carried Through Time

A recurrent neural network maintains a hidden state — a vector updated at every step of the sequence. At each time step, the network combines the current input with the previous hidden state to compute a new one, which then carries forward to the next step. This hidden state functions as a compressed memory of everything the network has seen in the sequence so far.

Parameter sharing again — across time, not space
The exact same weights are reused at every time step — a direct structural cousin of nn1-7's own convolutional kernel, which reused the same weights at every spatial position. Here, the sharing happens across time instead of across space, but it's the identical underlying idea: one small, learnable transformation, applied repeatedly.

Unrolling Through Time — Backpropagation Through Time

Conceptually "unrolling" an RNN means drawing the same cell repeatedly, once per time step, each copy feeding its own output forward as the next copy's own input. Training happens via backpropagation through time (BPTT) — literally nn1-5's own backpropagation and chain rule, applied across the unrolled time steps instead of across depth-wise layers.

Vanishing Gradients — nn1-4's Own Cliffhanger, Now Worse

nn1-4 previewed saturation shrinking gradients toward zero. In an RNN, this becomes genuinely more severe, for a specific, mechanical reason: because the same weight is reused at every time step, backpropagation through time multiplies that same weight's own gradient contribution by itself, repeatedly, once per time step. If each individual factor is even slightly less than 1 — easily the case with saturating activations — the cumulative product shrinks toward zero exponentially fast across a long sequence.

Why this is worse than nn1-5's own general case
nn1-5's own chain rule multiplies together different weights at each layer — bad luck at any one layer doesn't guarantee bad luck at every other. An RNN multiplies the same weight by itself, over and over, across every time step — a single unfavorable value compounds relentlessly rather than merely contributing once. A plain RNN effectively "forgets" anything from more than a handful of time steps ago, because gradients from far in the past barely survive the trip back to influence early weights at all.

LSTMs — The Real, Documented Fix

Long Short-Term Memory networks introduce a separate cell state that flows through time with only carefully controlled modifications, plus three gates — small sigmoid-activated sub-networks that learn how much old information to forget, how much new information to add, and how much of the current state to actually output at each step.

Why gates actually fix the vanishing-gradient problem
Instead of forcing the same fixed multiplicative shrinkage at every single time step regardless of content, the gates let the network learn when to preserve information essentially unchanged and when to actually update it. This provides a much more direct path for gradients to flow backward through the cell state across many time steps, largely avoiding the repeated same-weight-multiplied-by-itself collapse plain RNNs suffer from.

What Comes Next — A Different Kind of Limit

LSTMs fixed the vanishing-gradient problem. They didn't fix a separate, genuinely different limitation: an RNN (LSTM or not) is inherently sequential — each time step's own computation depends on the previous step's own output, which means, unlike nn1-7's own CNN feature maps or nn1-6's own parallel mini-batches, the steps within one single sequence can't be computed simultaneously. nn1-9 covers the architecture that solved this specific bottleneck instead.

Hands-On Exercises

Exercise 1

Using this chapter's own tip-box, explain the precise structural parallel between an RNN's own parameter sharing across time and a CNN's own parameter sharing across space, from nn1-7.

📄 View solution
Exercise 2

Explain, using this chapter's own reasoning, why the vanishing-gradient problem is genuinely worse for a plain RNN processing a long sequence than for nn1-5's own general multi-layer case.

📄 View solution
Exercise 3

Explain why this chapter says LSTMs fixed the vanishing-gradient problem but did NOT fix RNNs' own inherently sequential nature, and explain why these are genuinely two separate limitations rather than the same one.

📄 View solution

Chapter 8 Quick Reference

  • RNN — a hidden state updated at every time step, the same weights reused at every step (parameter sharing across time)
  • BPTT — nn1-5's own backpropagation, applied across unrolled time steps instead of layers
  • Vanishing gradients — nn1-4's own cliffhanger, made worse: the same weight multiplied by itself repeatedly across long sequences
  • LSTM — a separate cell state plus learned gates (forget/input/output), providing a direct path for gradients to survive long sequences
  • LSTMs fixed vanishing gradients — they didn't fix RNNs' own inherently sequential, non-parallelizable computation
  • Next chapter: A Transformer Preview