Training in Practice: Regularization for Neural Networks
Neural Networks & Deep Learning
Chapter 6 · Training in Practice: Regularization for Neural Networks
nn1-5 gave the mechanism. This chapter covers how it's actually run at real scale, and how to stop a network with enormous capacity from doing exactly what ml1-7's own unconstrained decision tree did — memorizing the training set instead of learning from it.
Batch, Stochastic & Mini-Batch Gradient Descent
| Variant | Gradient computed from | Trade-off |
|---|---|---|
| Batch GD | The entire training set, every update | Accurate, stable gradient — slow, memory-heavy, one update per full pass |
| Stochastic GD (SGD) | One random example at a time | Noisy, but the noise itself genuinely helps escape shallow local minima |
| Mini-batch GD | A small batch (32–128 examples) | The practical standard — balances stability against update frequency, and maps efficiently onto GPU parallelism |
Mini-batch is the near-universal real-world default specifically because a batch of, say, 64 examples can be processed simultaneously on GPU hardware — the same parallel-computation advantage nn1-10's own framework tour will cover directly.
Epochs & Learning Curves
One full pass through the entire training set is one epoch. Training runs for many epochs, and plotting loss on the training and validation sets against epoch number is exactly ml1-8's own learning curve, applied here directly — the same diagnostic tool, the same reading (a persistent gap between the two curves signals overfitting; both curves converging to a poor value signals underfitting).
Learning Rate Schedules
nn1-5 already named the learning rate's own basic trade-off. In practice, many training runs use a schedule — starting with a larger learning rate for fast early progress, then gradually shrinking it as training proceeds, allowing the network to settle into a good solution more precisely once it's already in the right general area.
Why Neural Networks Overfit — ml1-8's Own Framing, New Model Family
A network with enough layers and neurons has an enormous number of learnable parameters — genuinely enough capacity to memorize a training set outright, exactly the same failure mode ml1-7's own unconstrained decision tree demonstrated most visibly. ml1-8's own bias-variance tradeoff applies unchanged: more capacity can mean lower bias and higher variance, and every technique below is a deliberate, controlled trade of one against the other, for this specific model family.
Dropout — A Genuine Cousin of ml1-7's Own Random Forest
Dropout randomly "turns off" (zeros out) a fraction of a layer's own neurons on each individual training pass — a different random subset every time. No single neuron can be relied upon to always be present, so the network is forced to spread useful information across many redundant pathways rather than concentrating it in a few neurons that happen to fit training-set quirks especially well.
ml1-7's own random forest trains many trees, each on a random subset of data and features, then averages them — idiosyncratic per-tree errors cancel, genuine signal survives. Dropout achieves something genuinely similar within a single network: each training pass effectively trains a slightly different, randomly-thinned sub-network, and the fully-connected network used at inference time behaves like an implicit average across all of them. Different mechanism, same underlying spirit — enforced diversity and redundancy fighting overfitting.
Early Stopping — Catching Overfitting Live
ml1-8's own overfitting signature — low training error, rising validation error — doesn't have to be diagnosed only after the fact. Early stopping monitors ml1-2's own validation set during training itself and halts once validation loss starts climbing even as training loss keeps falling, keeping the weights from the point right before overfitting began rather than the final, most-overfit epoch.
Combining the Toolkit
ml1-8's own L1/L2 regularization (often called "weight decay" in a neural-network context) are all routinely combined in the same training run — each addresses a genuinely different angle of the same underlying bias-variance tradeoff, and using more than one is standard practice, not redundant.
Hands-On Exercises
Explain, using this chapter's own compare-table, why mini-batch gradient descent became the practical standard rather than pure batch or pure stochastic gradient descent.
📄 View solutionExplain the specific similarity this chapter draws between dropout and ml1-7's own random forest, being precise about what's actually similar (the underlying fight against overfitting) and what's genuinely different (the mechanism).
📄 View solutionExplain why early stopping is described as catching ml1-8's own overfitting signature "live" rather than diagnosing it after the fact, and explain what specifically early stopping does once it detects that signature.
📄 View solutionChapter 6 Quick Reference
- Batch / Stochastic / Mini-batch GD — a stability-vs-speed tradeoff; mini-batch is the practical standard, enabling GPU parallelism
- Epoch — one full pass through training data; loss-per-epoch curves are ml1-8's own learning curves, applied here
- A network's own huge capacity can memorize training data — ml1-7's own unconstrained-tree overfitting, in a new model family
- Dropout — randomly zeroing neurons per pass, a genuine cousin of ml1-7's own random-forest idea (different mechanism, same enforced-diversity spirit)
- Early stopping — halting on ml1-2's own validation loss the moment it starts rising, catching overfitting live
- Dropout, early stopping, and weight decay (ml1-8's own L1/L2) are routinely combined, not mutually exclusive
- Next chapter: Convolutional Neural Networks (CNNs)