The Train/Test/Validation Split & Why It Exists
Machine Learning Fundamentals
Chapter 2 · The Train/Test/Validation Split & Why It Exists
ml1-1's own roadmap named this as the very first practical step, before any model gets trained at all. It has to come first — every technique in this course depends on it being done honestly.
Why You Can't Evaluate a Model on Its Own Training Data
Imagine grading a student using the exact same practice questions they studied from, answer key included. A perfect score wouldn't tell you whether they actually understand the material — only whether they memorized those specific answers. A machine learning model has exactly the same failure mode: shown enough examples, it can simply memorize the relationship between each specific input and its specific answer, rather than learning a pattern that generalizes to a new example it's never seen. Testing it on that same training data would make even a model that did nothing but memorize look flawless — and tell you nothing about whether it would be any good on a genuinely new used car it's never encountered.
ml1-8's own overfitting chapter names this exact failure mode formally and shows what it looks like in practice.
The Train/Test Split
The practical fix: split the dataset into two pieces before training anything. The training set is what the model actually learns from. The test set is held back, completely untouched during training, and used only afterward, to check how the model performs on data it has genuinely never seen.
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
An 80/20 or 70/30 split is common — a convention, not a law. random_state fixes the random shuffle so the same split can be reproduced later, useful for genuinely comparing two different models on identical data.
A Subtler Trap: Leaking Information Through Repeated Peeking
A train/test split alone doesn't fully solve the problem. If a model is trained, checked against the test set, adjusted based on what went wrong, checked against the test set again, adjusted again — the test set is no longer serving its original purpose. Even without formally training on it, repeatedly using test-set performance to guide decisions lets information about the test set quietly leak into the model through every one of those adjustment cycles. The final "test accuracy" ends up flattered by the same underlying problem this chapter opened with, just one step more indirect.
The Third Piece: A Validation Set
The fix is a third split: a validation set, used for exactly this kind of iterative tuning, while the test set stays completely untouched until one single, final check.
| Set | Used for | Touched how often |
|---|---|---|
| Training | Fitting the model itself | Every training run |
| Validation | Comparing/tuning different models or settings | As many times as needed during development |
| Test | One final, honest performance check | Exactly once, at the very end |
A First Preview of Cross-Validation
With a small dataset, a single validation split can just be unlucky — by chance, an unusually easy or unusually hard subset. Cross-validation gets more reliable mileage out of limited data by rotating which portion serves as the validation set across several rounds, then averaging the results. This chapter only needs the concept to exist; ml1-8 delivers the full technique, once ml1-3–ml1-7 have given it real models to actually validate.
Why the Split Itself Has to Be Random
Splitting a dataset without shuffling it first is a real, easy-to-miss trap. If ds1-9's own used-car listings were sorted by year and split straight down the middle with no shuffling, the test set could end up containing only the newest cars while training saw only older ones — the model would never see a genuinely representative mix of ages during training, and the test set wouldn't represent the real population of cars it's meant to evaluate against either.
ds1-6 defined sampling bias as "a flawed sampling method producing a misleading conclusion even when every individual calculation performed on it is done correctly," using a weekday-only sales sample as its own example. An unshuffled train/test split is exactly that same failure, one layer up: the split itself is the sampling method, and getting it wrong produces a misleading performance number no matter how correctly every later calculation is done. train_test_split()'s own default behavior shuffles automatically for exactly this reason.
Hands-On Exercises
Using this chapter's own student-exam analogy, explain precisely why a perfect score on training data doesn't tell you what you actually want to know about a model, and identify the one thing it does tell you.
📄 View solutionExplain why a plain train/test split alone isn't enough once a model is being iteratively tuned, and explain specifically what a validation set fixes that a train/test split by itself doesn't.
📄 View solutionUsing this chapter's own tip-box and ds1-6's own definition of sampling bias, explain why an unshuffled train/test split on a year-sorted dataset counts as the same category of mistake as ds1-6's own weekday-sampling example.
📄 View solutionChapter 2 Quick Reference
- Evaluating a model on its own training data can't distinguish genuine learning from memorization — ml1-8 names this failure formally
- Train — fit the model · Validation — tune/compare, as often as needed · Test — one final, honest check, touched exactly once
- Repeatedly checking test-set performance during tuning leaks information into the model indirectly — the reason a validation set exists at all
- Cross-validation previewed here, delivered in full in ml1-8
- An unshuffled split is ds1-6's own sampling bias, applied to model evaluation —
train_test_split()shuffles by default for exactly this reason - Next chapter: Linear Regression