Evaluation Metrics for Classification

Machine Learning Fundamentals

Chapter 6 · Evaluation Metrics for Classification

ml1-5 answered which factors matter for attrition, not how good the model actually is at predicting who leaves. ml1-5 also flagged something ml1-4 never had to deal with: attrition is genuinely imbalanced — far more No than Yes in most real companies. That fact changes which metric is trustworthy here.

The Confusion Matrix — Four Real Outcomes, Not One

Every prediction on a binary target falls into exactly one of four buckets, comparing the model's own prediction against the real outcome:

Predicted: Left
Predicted: Stayed
Actually Left
True Positive
Correctly caught
False Negative
Missed — the costly one
Actually Stayed
False Positive
False alarm
True Negative
Correctly cleared

Accuracy — And Why It's Dangerous Here Specifically

Accuracy is simply (correct predictions) ÷ (total predictions) — the single most intuitive metric, and, on an imbalanced target, a genuinely misleading one.

The accuracy paradox, concretely
If 90% of employees in the data actually stayed, a model that predicts "No, they won't leave" for absolutely everyone — never once actually examining salary, tenure, or department — scores 90% accuracy without learning anything at all. On ml1-5's own genuinely imbalanced attrition data, a high accuracy number alone proves nothing about whether the model is actually any good at its real job: catching the employees who are, in fact, at risk of leaving.

Precision & Recall — Answering Two Different Questions

Precision = True Positives ÷ (True Positives + False Positives) — of everyone the model flagged as a flight risk, what fraction actually left? A precision-focused model minimizes false alarms. Recall = True Positives ÷ (True Positives + False Negatives) — of everyone who actually left, what fraction did the model catch? A recall-focused model minimizes missed cases.

For attrition specifically, a missed at-risk employee (a False Negative) is usually the more expensive mistake — a resignation nobody saw coming, with no chance to intervene — while a false alarm (a False Positive) costs, at worst, an unnecessary retention conversation. This is a genuine, business-specific judgment call about which error type matters more, not a fact the data itself decides.

F1 — One Number, When You Need One

F1 is the harmonic mean of precision and recall — a single combined number that only stays high when both precision and recall are reasonably good, punishing a model that's strong on one and collapses on the other far more than a plain average would. Useful specifically when precision and recall need to be compared or ranked with one number rather than two.

Applying This to ml1-5's Own Model

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score

preds = model.predict(X_test_scaled)
confusion_matrix(y_test, preds)
accuracy_score(y_test, preds)
precision_score(y_test, preds)
recall_score(y_test, preds)
f1_score(y_test, preds)
Illustrative results, on ml1-5's own fitted model
Accuracy ≈ 86% — looks strong on its own, but per this chapter's own accuracy-paradox warning, that alone proves little on imbalanced data. Precision ≈ 0.71 — most flagged employees genuinely were at risk. Recall ≈ 0.58 — the model still misses a meaningful share of the employees who actually left. F1 ≈ 0.64 — the honest, combined picture: a genuinely useful model, with real room to catch more at-risk employees.

Adjusting the Threshold — ml1-5's Own Preview, Delivered

ml1-5 flagged the 0.5 threshold as adjustable, not fixed. Lowering it flags more employees as at-risk — recall rises (fewer real leavers slip through), precision falls (more false alarms). Raising it does the reverse. Given this chapter's own reasoning that a missed at-risk employee is usually the costlier mistake for this specific problem, a real deployment might deliberately lower the threshold below 0.5, accepting more false alarms in exchange for catching more genuine flight risks — a business decision the metrics inform but don't make automatically.

Hands-On Exercises

Exercise 1

Using this chapter's own accuracy-paradox warn-box, explain exactly how a model that never examines any feature at all could still score 90% accuracy, and explain why this makes accuracy alone untrustworthy specifically for ml1-5's own attrition data.

📄 View solution
Exercise 2

Explain the difference between precision and recall using this chapter's own confusion-matrix definitions, and explain why this chapter argues a False Negative is usually more costly than a False Positive for the attrition problem specifically.

📄 View solution
Exercise 3

Explain, using this chapter's own reasoning about threshold adjustment, why lowering the classification threshold below 0.5 would raise recall and lower precision, and explain why the chapter frames the actual threshold choice as a business decision rather than something the metrics decide automatically.

📄 View solution

Chapter 6 Quick Reference

  • Confusion matrix — True/False Positive/Negative, the foundation every other metric here is built from
  • Accuracy — intuitive, but genuinely misleading on imbalanced data (the accuracy paradox)
  • Precision — of flagged cases, how many were real? · Recall — of real cases, how many were caught?
  • F1 — one combined number, punishing a model strong on only one of the two
  • The 0.5 threshold (ml1-5) is adjustable — lowering it trades precision for recall, a business decision the metrics inform, not decide
  • Next chapter: Decision Trees & Random Forests