AlphaGo & Reinforcement Learning
⚫⚪ AlphaGo & Reinforcement Learning
🌌 A Game Brute Force Couldn't Touch
Go is played on a 19×19 grid — compared to chess's 8×8 — and its number of legal board positions has been estimated at roughly 2×10170, a number that dwarfs chess's game-tree complexity (roughly 10120) so completely that the comparison barely conveys the gap. Deep Blue's winning strategy — evaluate roughly 200 million positions per second and search deep enough to see the win coming — simply doesn't scale to a space this size, no matter how much hardware you throw at it. For decades, many AI researchers considered Go a benchmark that brute-force methods would never crack, and one that genuine machine intelligence — not just faster search — would be required to solve.
🏢 DeepMind
DeepMind, a UK AI research company founded in 2010 by Demis Hassabis and colleagues (acquired by Google in 2014), took on Go as a flagship research challenge. Their system, AlphaGo, combined three distinct ingredients that, together, sidestepped the brute-force wall entirely rather than trying to power through it.
🧩 Three Ingredients
AlphaGo paired two deep neural networks — descendants of Chapter 4's convolutional architectures — with a smarter search strategy. A policy network learned to predict which moves a strong player would consider, narrowing the search to plausible options instead of every legal move. A value network learned to estimate how good a given board position was, without needing to play the game out to the end. Both networks then guided Monte Carlo Tree Search — a search algorithm that explores a relatively small number of promising simulated game lines rather than exhaustively evaluating everything, the way Deep Blue's minimax search did.
🎮 What Is Reinforcement Learning?
AlphaGo's networks were trained using reinforcement learning — a third major learning paradigm, distinct from both approaches covered earlier in this course. Chapter 1's statistical methods and Chapter 4's ImageNet-trained networks learn from a fixed set of labeled examples (this image is a cat; this is the correct translation). Reinforcement learning instead has an agent take actions in an environment and receive rewards or penalties based on the outcome, gradually adjusting its behavior to maximize reward over time — no human needs to label the "correct" move for every possible board position, because the system discovers what works by playing.
AlphaGo was first trained on a database of human expert games (a supervised warm start), then improved dramatically through self-play — playing millions of games against itself, with each game's win or loss used as the reward signal to refine both networks further.
🏆 March 2016 — Lee Sedol
In March 2016, AlphaGo faced Lee Sedol, one of the world's top-ranked professional Go players, in a five-game match in Seoul, South Korea, broadcast live to a global audience. AlphaGo won 4 games to 1. Many AI researchers had publicly predicted, only a year or two earlier, that a computer beating a top human at Go was still a decade or more away — making the timing itself, not just the result, a genuine shock to the field's own expectations.
In Game 2, AlphaGo played a move so unconventional that professional commentators initially assumed it was an error — a move no human player of Lee Sedol's caliber would seriously consider. It turned out to be a brilliant, winning move. "Move 37" is now widely cited as a moment where an AI system appeared to demonstrate something closer to creative insight than raw calculation — a striking contrast to Chapter 2's dismissal of Deep Blue as "just brute force, not real intelligence."
🔁 AlphaGo Zero and AlphaZero
DeepMind pushed the underlying idea even further in 2017 with AlphaGo Zero, which learned to play Go entirely through self-play, starting from nothing but the rules of the game — no human game data at all — and surpassed the original Lee-Sedol-beating AlphaGo within days of training. A generalized version, AlphaZero, applied the same self-play approach to chess and shogi, and decisively defeated the world's strongest existing chess engines. That's a genuine full-circle moment for this course: the same game Deep Blue conquered through raw brute-force search in 1997 fell again, twenty years later, to a system that had never been taught a single human chess strategy — it simply played against itself until it discovered better ones.
📜 Two Ways to Beat a World Champion
| Deep Blue (Chapter 2, 1997) | AlphaGo (This Chapter, 2016) |
|---|---|
| Exhaustive brute-force search over the full game tree | Narrow, guided search over only the most promising lines |
| Hand-tuned evaluation function (grandmaster-assisted) | Learned evaluation function (value network, trained by self-play) |
| Works because chess's state space is small enough to search | Works despite Go's state space being far too large to search exhaustively |
It's worth being precise about what actually changed here. AlexNet (Chapter 4) proved deep networks could learn extraordinarily well from a large fixed dataset. AlphaGo proved something different: a system can learn to master a task with no complete "correct answer" dataset available at all, by generating its own experience and improving from it. That distinction — learning from a fixed dataset versus learning by interacting with an environment — turns out to matter enormously for the next architecture this course covers, which didn't come from reinforcement learning at all, but from a different rethink of how neural networks should process sequences of information.
🤔 Questions to Sit With
"Move 37" is often described as looking creative rather than merely calculated. Having read Chapter 2's skepticism about calling Deep Blue's win "real" intelligence, do you find AlphaGo's case genuinely different, or is this the same convincing-performance question in a new costume?
AlphaGo Zero surpassed the human-trained AlphaGo by learning purely from self-play, with no human game data at all. What does it suggest about human expertise — in Go or elsewhere — that a system improved faster once it stopped learning from us?
Go fell to AI roughly a decade earlier than most experts predicted. Thinking back to Course 2's recurring theme of overconfident timelines (the Dartmouth Workshop's two-month promise, in particular) — does an expert prediction being wrong in the optimistic direction feel like a different kind of mistake than being wrong in the overconfident direction, or is it the same misjudgment either way?
🎯 What's Next
Next chapter: Transformers — the 2017 "Attention Is All You Need" paper, and the architecture shift that would go on to power every major language model that followed.