Forward Propagation, Loss Functions & Backpropagation

Neural Networks & Deep Learning

Chapter 5 · Forward Propagation, Loss Functions & Backpropagation

nn1-3 proved a multi-layer network can solve XOR, using weights a human worked out by hand. historyai3-3 named "1986 backpropagation" as part of AI's own narrated history without ever explaining the mechanism. This chapter delivers both — the real algorithm that finds those weights automatically, for any network, from data alone.

Forward Propagation — Formally Naming What Every Example Already Did

Forward propagation is simply running the network forward: input values flow into the first layer, each neuron computes its weighted sum plus activation, those outputs become the next layer's inputs, and so on until the final output layer produces a prediction. Every worked example since nn1-1 has already done this — this chapter just gives it a name before building on it.

Loss Functions — Cross-Entropy, Classification's Own Counterpart to MSE

ml1-4's own MSE measured regression error. Classification networks typically use a different loss: cross-entropy, which specifically measures how far a predicted probability distribution sits from the true, correct answer.

Why not just reuse MSE?
Cross-entropy could technically be replaced with MSE for a classification task, but the real, honest reason it isn't: cross-entropy grows sharply as the predicted probability for the correct class approaches zero — a confident, wrong prediction is penalized far more heavily than MSE would penalize it. This produces a much stronger, more useful gradient signal exactly when the network is most wrong and most confident about it — precisely the situation where learning needs to happen fastest.

The Chain Rule — How Blame Travels Backward

A multi-layer network is a chain of functions — one layer's output feeding the next. If a small change to an early-layer weight changes that layer's output, which changes the next layer's output, which changes the final loss, the total effect of that one weight on the final loss is the product of each individual step's own local effect, chained together — precisely the calculus chain rule. Conceptually: if A affects B, and B affects C, then A's own effect on C is A's effect on B, multiplied by B's effect on C.

Backpropagation — The Real Algorithm

  • Forward pass. Run the network forward (as above), compute the prediction, compute the loss against the true answer.
  • Compute the output layer's own gradient. Using the loss function's own derivative, work out how much a small change to each output-layer weight would change the loss.
  • Propagate backward, layer by layer. Using the chain rule, compute each earlier layer's own contribution to that same final loss — each layer's own gradient calculation reuses the gradient already computed one layer downstream.
  • Update every weight. Once every layer's own gradient is known, adjust each weight via gradient descent.

    • This is the real, 1986 breakthrough (Rumelhart, Hinton, and Williams) — a general, automatic method for training a network of any depth, not a hand-derivation limited to one small, already-understood problem like nn1-3's own XOR example. This is the piece Minsky and Papert's own 1969 book (nn1-2) left as an open question, resolved for real seventeen years later.

    Gradient Descent — The Actual Update Rule

    new_weight = old_weight - learning_rate * gradient

    The learning rate controls how large each step is. Too high, and weight updates overshoot, bouncing past good solutions or diverging entirely; too low, and training crawls, taking an impractically long time to converge. nn1-6 covers this, and several real variants of the basic gradient-descent rule, in practical depth.

    Closing nn1-3's Own Deferred Promise

    What actually happens now, for real
    Trained via backpropagation on labeled XOR examples, a two-layer network with randomly initialized weights will converge to a set of weights that correctly solves XOR — not necessarily nn1-3's own exact OR/NAND decomposition, but a mathematically valid solution reached automatically, with no human ever needing to work out the underlying logical structure by hand. nn1-3's own chapter proved a solution exists; this chapter is what actually finds one.

    Hands-On Exercises

    Exercise 1

    Explain, using this chapter's own reasoning, why cross-entropy produces a stronger gradient signal than MSE specifically when a network is confidently wrong, and why that specific situation is exactly when a strong signal matters most.

    📄 View solution
    Exercise 2

    Using this chapter's own A-affects-B-affects-C explanation of the chain rule, explain why computing an early layer's own contribution to the final loss requires information from every layer between it and the output, not just the early layer alone.

    📄 View solution
    Exercise 3

    Explain precisely what backpropagation actually resolved that nn1-3's own worked example couldn't, and why "a mathematically valid solution, not necessarily nn1-3's own exact one" is the accurate way to describe what training would find.

    📄 View solution

    Chapter 5 Quick Reference

    • Forward propagation — running the network forward to produce a prediction, already used implicitly since nn1-1
    • Cross-entropy — classification's own counterpart to ml1-4's MSE, penalizing confident wrong answers much more heavily
    • Chain rule — A's effect on C is A's effect on B times B's effect on C, chained across every layer
    • Backpropagation — forward pass, compute output-layer gradient, propagate backward via the chain rule, update every weight — the real 1986 breakthrough, delivering historyai3-3's own namecheck
    • Gradient descent — new_weight = old_weight − learning_rate × gradient; nn1-6 covers this in practical depth
    • Closes nn1-3's own promise: a network can now find valid XOR weights automatically, with no hand-derivation required
    • Next chapter: Training in Practice — Regularization for Neural Networks