Multi-Layer Perceptrons & Why Depth Solves XOR

Neural Networks & Deep Learning

Chapter 3 · Multi-Layer Perceptrons & Why Depth Solves XOR

nn1-2 proved, geometrically, that no single straight line separates XOR. This chapter builds an actual, working two-layer network that solves it — and shows the general principle that makes the solution possible, not just this one specific case.

The Core Idea: A Hidden Layer Transforms the Space

A hidden layer doesn't just add more tunable numbers — it re-represents the input as a new set of coordinates, computed by the hidden neurons themselves. A problem that's unsolvable by a straight line in the original input space can become solvable by a straight line in this new, transformed space. That's the entire trick depth relies on.

A Real, Worked XOR Solution

XOR can be built from two simpler, individually linearly-separable pieces: XOR(x1, x2) = OR(x1, x2) AND NAND(x1, x2) — true exactly when at least one input is 1, and not both are 1. Both OR and NAND are themselves linearly separable (a single line can solve each one alone). One concrete set of weights that implements this — one valid solution among several that would work:

Hidden neuron 1 (approximates OR):   step(x1 + x2 - 0.5)
Hidden neuron 2 (approximates NAND): step(-x1 - x2 + 1.5)
Output neuron (AND of both hidden outputs): step(h1 + h2 - 1.5)

x1	x2	h1 (OR)	h2 (NAND)	output (AND)	Correct XOR?
0	0	0	1	0	✓
0	1	1	1	1	✓
1	0	1	1	1	✓
1	1	1	0	0	✓

Where the transformation actually happened

In the original (x1, x2) space, XOR was unsolvable by any line — nn1-2's own proof. In the new (h1, h2) space the hidden layer produces, the four points become (0,1), (1,1), (1,1), (1,0) — and a single line does separate the "output 1" points from the "output 0" point in this new space. The hidden layer didn't add raw capacity so much as it re-drew the map the final line gets to work with.

Why This Requires Genuine Nonlinearity

This trick only works because the hidden layer's own activation function (step above; nn1-4 covers the real options) is nonlinear. Stack two purely linear layers with no nonlinearity between them, and the math collapses: a linear function of a linear function is still just one linear function — mathematically indistinguishable from a single layer, with none of this chapter's own transformation trick available at all. nn1-4 picks this up directly.

Closing nn1-2's Own Historical Thread

This is, in principle, exactly the resolution Minsky and Papert's own book left open in nn1-2 — a multi-layer network genuinely can solve XOR. What this chapter's own worked example doesn't do is learn these weights automatically from data the way ml1-3's own regression or nn1-2's own perceptron rule did — these specific numbers were hand-derived for this one small, known problem. A real, general, automatic training method for networks like this one didn't arrive until 1986 — nn1-5's own backpropagation, the actual historical breakthrough that made deep networks practical rather than hand-built curiosities.

Beyond XOR — Why This Is "Depth," Not Just "Size"

The same principle scales: each additional layer can compose the previous layer's own transformed representation into something even more re-arranged, letting a network build up genuinely hierarchical features — this is the real, technical meaning behind "deep" in deep learning, and it's a fundamentally different lever from simply adding more neurons to one single layer (more width, same transformation depth).

Hands-On Exercises

Exercise 1

Using this chapter's own worked table, explain why the four points become linearly separable in (h1, h2) space even though nn1-2 proved they aren't separable in the original (x1, x2) space.

📄 View solution

Exercise 2

Explain why this chapter says stacking two purely linear layers with no nonlinearity would collapse into a single linear layer, and why this matters for whether the hidden-layer trick in this chapter would even work at all.

📄 View solution

Exercise 3

Explain what this chapter's own worked XOR solution does and doesn't prove, specifically regarding whether these weights were learned automatically, and identify what nn1-5 still needs to deliver.

📄 View solution

Chapter 3 Quick Reference

A hidden layer transforms the input into a new coordinate space — a problem unsolvable by a line in the original space can be solvable in the new one
XOR = OR AND NAND — two linearly-separable sub-problems, composed by a second linear layer
This requires genuine nonlinearity in the hidden layer — purely linear layers collapse into one, previewing nn1-4
This worked example is hand-derived, not learned — nn1-5's own backpropagation (1986) is the real historical breakthrough that made this automatic
"Deep" means composing transformations across layers — a genuinely different lever from adding more neurons to one layer
Next chapter: Activation Functions