Overfitting, Underfitting & Regularization

ml1-7 showed the clearest possible overfitting example in this course — a tree memorizing training rows one at a time. ml1-2 previewed cross-validation and promised this chapter would deliver it in full. Both threads close here.

Two Failure Modes, Formally

The diagnostic signature is the gap: overfitting shows low training error but high test error; underfitting shows high error on both — the model was never good, even on data it directly saw.

The Bias-Variance Tradeoff — Built on ds1-6's Own Vocabulary

Bias is systematic, consistent wrongness — error from a model too simple to capture the real pattern, present regardless of which training sample it happened to see (underfitting). Variance reuses ds1-6's own definition directly: how much would this model's own predictions change if trained on a different random sample from the same population? A single unconstrained tree (ml1-7) has high variance — a different training sample would grow a noticeably different tree. A random forest has lower variance — averaging many trees stabilizes the result regardless of which specific sample any one tree happened to see.

Reading a Learning Curve

Plotting training error and validation error against training-set size (or model complexity) gives a real diagnostic picture. High bias shows both curves converging to a similarly poor error — more data doesn't help, because the model itself is too simple to use it. High variance shows a persistent gap — low training error, meaningfully higher validation error — that more data typically narrows, since a larger, more representative sample gives a flexible model less room to fit pure noise.

Regularization — Deliberately Trading Bias for Variance

Regularization adds a penalty term to ml1-3's own least-squares fitting objective, discouraging large coefficient values. This is the bias-variance tradeoff, applied on purpose: a regularized model fits the training data very slightly worse (a bit more bias) in exchange for being noticeably less sensitive to the specific training sample it saw (less variance).

from sklearn.linear_model import Ridge, Lasso

ridge = Ridge(alpha=1.0)   # L2
lasso = Lasso(alpha=1.0)   # L1

Cross-Validation — ml1-2's Own Promise, Delivered

ml1-2 flagged the real risk: a single validation split can be unlucky on a small dataset. k-fold cross-validation fixes this directly: split the data into k equal folds, train on k−1 of them and validate on the one held out, rotate which fold is held out across k full rounds, then average the k resulting scores.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5)
scores.mean(), scores.std()

Every row gets used for validation exactly once and for training k−1 times — a far more reliable estimate of real performance than any single split, and the standard deviation across the k scores is itself informative: a small spread means the model performs consistently regardless of which slice of data it sees; a large spread is itself a sign of high variance.

A Practical Checklist

Hands-On Exercises