Evaluation Metrics for Regression

Machine Learning Fundamentals

Chapter 4 · Evaluation Metrics for Regression

ml1-3 closed one question (which feature matters more) and left another open on purpose: is the model's own price prediction actually any good? Four metrics, applied directly to ml1-3's own fitted model, answer that.

MAE

Mean Absolute Error — average |predicted − actual|, in the original units (dollars). The simplest, most directly readable metric.

MSE

Mean Squared Error — average squared error. This is literally what ml1-3's own least-squares fitting minimizes during training.

RMSE

√MSE — MSE's own sensitivity to large errors, brought back into interpretable original units.

R²

A genuinely different kind of metric — not an error size, but a proportion of variance explained.

MAE — The Plain-English Metric

from sklearn.metrics import mean_absolute_error
mae = mean_absolute_error(y_test, model.predict(X_test_scaled))

Directly readable: "on average, this model's price predictions are off by about $X." Every error contributes to the average in exact proportion to its own size — a $10,000 miss counts exactly ten times as much as a $1,000 miss, no more, no less.

MSE — The Training Objective, Reused as a Metric

ml1-3 already explained least squares as minimizing total squared error during fitting. MSE is that exact same squared-error idea, averaged, and computed on the test set instead of the training set — the fitting objective, repurposed as an honest, out-of-sample evaluation number. The squaring inherits ds1-6's own variance-style property: large errors are penalized disproportionately more than small ones. The real cost: the result is in dollars squared, a unit with no direct real-world meaning.

RMSE — MSE's Own Sensitivity, Back in Real Units

from sklearn.metrics import mean_squared_error
import numpy as np
rmse = np.sqrt(mean_squared_error(y_test, model.predict(X_test_scaled)))

RMSE is simply MSE's own square root — dollars-squared brought back to dollars, while keeping MSE's own large-error sensitivity intact. RMSE is mathematically guaranteed to be greater than or equal to MAE on the same data, and the gap between them is informative in its own right: a large gap means a few big misses are driving the error total; a small gap means errors are fairly uniform in size.

ds1-9's own Jaguar, revisited

If a genuine outlier like ds1-9's own vintage Jaguar ended up in the test set, RMSE would spike sharply while MAE moved only modestly — a direct, concrete consequence of squaring: one enormous miss contributes its squared value to MSE (and therefore RMSE), but only its own plain, unsquared size to MAE. A large MAE-vs-RMSE gap is often the first real clue that a small number of unusual cases are dominating a model's own error.

R² — A Different Kind of Question Entirely

R² doesn't measure error size at all — it measures what proportion of price's own total variance (ds1-6's own vocabulary, directly) the model actually accounts for. R² = 1.0 means the model explains all of it (a perfect fit); R² = 0 means the model does no better than simply always predicting the average price; a negative R² means the model is doing worse than that trivial baseline.

from sklearn.metrics import r2_score
r2 = r2_score(y_test, model.predict(X_test_scaled))

Applying All Four to ml1-3's Own Model

Illustrative results, on ml1-3's own fitted model

MAE ≈ $2,100 — typical predictions land within about $2,100 of the real price. RMSE ≈ $2,800 — noticeably higher than MAE, consistent with a handful of larger misses pulling it up. R² ≈ 0.82 — the model accounts for roughly 82% of the real variance in price. Together: a genuinely useful model, not a perfect one, with a modest number of harder-to-predict cases worth a closer look.

Choosing a Metric

Metric	Best when
MAE	A simple, robust, directly interpretable "typical error" is what matters
RMSE	Large errors are genuinely more costly than small ones and should be weighted that way
R²	A scale-free "how good is the overall fit" number, comparable across different problems

Two honest limits, before moving on

A single train/test split's own metrics can be noisy on a small dataset — a different random split could shift these numbers meaningfully, exactly the reason ml1-2 previewed cross-validation and ml1-8 delivers it in full. And a strong R² still doesn't upgrade ml1-3's own honest confounding-variable caveat into proof of causation — it only says the fit is close, not that the underlying relationship is fully understood.

Hands-On Exercises

Exercise 1

Explain, using this chapter's own Jaguar warn-box, why a genuine outlier in the test set would spike RMSE much more sharply than MAE, tracing the difference back to what squaring an error actually does.

📄 View solution

Exercise 2

Explain why R² is described as "a genuinely different kind of question entirely" compared to MAE/MSE/RMSE, using this chapter's own definitions to explain what each type of metric actually measures.

📄 View solution

Exercise 3

Using this chapter's own illustrative results (MAE ≈ $2,100, RMSE ≈ $2,800, R² ≈ 0.82), explain what each individual number tells you about the model, and explain what the gap between MAE and RMSE specifically suggests.

📄 View solution

Chapter 4 Quick Reference

MAE — average absolute error, original units, treats every error proportionally
MSE — the training objective itself, reused as an out-of-sample metric; squared units
RMSE — √MSE, MSE's own large-error sensitivity in real units; MAE-RMSE gap flags outlier-driven error
R² — proportion of ds1-6's own variance explained; 1.0 perfect, 0 no better than predicting the mean, negative worse than that
A single split's metrics can be noisy — ml1-2's own cross-validation preview, delivered fully in ml1-8, exists for this reason
Next chapter: Logistic Regression & Classification Basics