ImageNet & AlexNet
🖼️ ImageNet & AlexNet
📚 Building ImageNet
Starting around 2007, Fei-Fei Li and collaborators at Princeton (later Stanford) set out to build something the field hadn't really had before: a genuinely massive, carefully labeled image dataset, organized according to the WordNet lexical hierarchy. The finished ImageNet dataset eventually contained more than 14 million labeled images across roughly 20,000 categories — orders of magnitude larger than anything computer vision researchers had previously trained on.
Labeling 14 million images by hand was only feasible because Li's team used Amazon Mechanical Turk — the crowdsourcing platform Amazon deliberately named after Course 1 Chapter 3's famous 18th-century chess-playing hoax, precisely because both involve genuine human intelligence doing the real work behind an interface that makes it look automated. The historical Turk fooled audiences for eighty years by hiding a human operator inside a "mechanical" illusion. Its modern namesake openly hires human operators to do exactly the kind of labeling work computers of the era still couldn't do reliably themselves — and in doing so, it quietly built the dataset that would trigger deep learning's breakthrough.
🏆 The ILSVRC Challenge
Starting in 2010, an annual competition — the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) — used a 1.2-million-image, 1,000-category subset of ImageNet to benchmark image classification algorithms, continuing the benchmark-driven culture Chapter 1 traced back to the 1987 UCI Machine Learning Repository. Through 2010 and 2011, the best-performing systems relied on the era's standard computer vision toolkit: hand-crafted features like SIFT, fed into statistical classifiers such as Chapter 1's support vector machines. Progress was real but incremental — top entries hovered around a 25–26% top-5 error rate, improving by a percentage point or two each year.
⚡ AlexNet's 2012 Landslide
In 2012, a deep convolutional neural network built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton — the same Hinton from Chapter 3's backpropagation paper and 2006 deep belief network revival — entered ILSVRC under the name AlexNet. It achieved a top-5 error rate of roughly 15.3%, against a second-place entry's 26.2%. That's not an incremental improvement; it's a margin so large that most of the field's typical year-over-year gains for the entire history of the competition combined wouldn't have closed it.
AlexNet's architecture built directly on Yann LeCun's CNN work from Chapter 3, but three things made it practical at this new scale: ReLU activation functions (much faster to train than the traditional sigmoid/tanh functions), dropout (a technique that randomly disables parts of the network during training to prevent overfitting on so much data), and — critically — training on two NVIDIA GTX 580 GPUs, repurposing graphics hardware originally built for rendering video games into a massively parallel engine for the matrix math neural network training actually requires.
📜 Before AlexNet, After AlexNet
| Before 2012 | After 2012 |
|---|---|
| Hand-crafted features (SIFT) + statistical classifiers (SVMs) | Features learned automatically inside a deep convolutional network |
| Training on standard CPUs | Training on GPUs, repurposed from graphics rendering |
| Incremental, percentage-point-a-year progress | A single ~11-point leap, unmatched by any prior year's combined gains |
Chapter 3 closed by naming exactly what neural networks were still waiting for: enough data, and enough compute. This chapter shows both arriving at once — ImageNet supplied the data, GPUs supplied the compute, and the algorithm (backpropagation, 1986) and architecture (convolutional networks, LeCun, late 1980s–90s) had already been sitting ready for decades. AlexNet isn't really a new idea. It's Chapter 3's old idea, finally given the resources it needed to prove itself.
💥 A Field Transformed Almost Overnight
Within roughly two to three years of AlexNet's win, hand-crafted feature engineering — the dominant computer vision approach for over a decade — was largely abandoned across the research community in favor of deep convolutional networks. Hinton himself was hired by Google shortly afterward. Companies including Google, Facebook, and Microsoft moved quickly and aggressively into deep learning research and hiring. Few technical results in this course's history — including Deep Blue's 1997 chess win — triggered as fast and as complete a shift in what the entire field considered the obviously correct approach.
🤔 Questions to Sit With
AlexNet's win came from combining old ideas (backpropagation, CNNs) with new resources (ImageNet, GPUs), not from a new theoretical breakthrough. Does that change how "revolutionary" the moment feels to you, compared to if the core algorithm itself had been newly invented in 2012?
Course 2 Chapter 8 named a recurring hype-cycle pattern: bold promise, real but narrower delivery, a widening gap, then collapse. The deep learning boom that started here in 2012 is now over a decade old and still going. Does that make it fundamentally different from the expert systems boom, or is it simply a boom that hasn't reached its collapse phase yet?
Modern Mechanical Turk workers did the invisible human labor that made ImageNet — and therefore AlexNet — possible, echoing the original Turk's hidden human operator. How much of today's AI progress do you think still depends on similarly invisible human labor, and how often does that labor get properly credited?
🎯 What's Next
Next chapter: AlphaGo & Reinforcement Learning (2016) — how DeepMind combined deep neural networks with a learning method that generates its own training data, to conquer a game whose state space made Chapter 2's brute-force chess approach hopeless.