Decision Trees & Random Forests

ml1-3 and ml1-5 both fit a smooth mathematical formula — a weighted sum of features, squashed through a sigmoid for classification. This chapter introduces a structurally different family entirely: a model that looks, on paper, almost exactly like a flowchart of yes/no questions.

What a Decision Tree Actually Is

A decision tree is a series of nested yes/no questions about features — "is salary below $50,000?" — each one branching to a child node, until a leaf gives a final prediction. Unusually, the same structure works for both jobs this course has covered separately: predicting a category (a leaf holding a class label) or a continuous number (a leaf holding an average value) — one model family doing what took ml1-3 and ml1-5 two separate techniques to cover.

The Uncanny Resemblance to ml1-1's Own Expert Systems

How the Tree Actually Picks Its Own Questions

Entropy measures how mixed a group of labels is: a node where every single employee left has entropy 0 (perfectly "pure" — no uncertainty left at all); a node split evenly 50/50 between leaving and staying has maximum entropy (as uncertain as a coin flip). Information gain measures how much a candidate split would reduce that entropy. At every node, the tree-building algorithm greedily tries every possible feature and threshold, and picks whichever single split produces the largest information gain — the biggest reduction in mixed-up-ness.

A Worked Split — And an Independent Confirmation

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=4, random_state=42)
tree.fit(X_train, y_train)

The Clearest Overfitting Example in This Course

Left unconstrained, a decision tree can keep splitting until individual leaves contain a single training example each — a tree that has, in effect, memorized the training set row by row. ml1-2's own warning about training-data evaluation becomes vividly concrete here: an unconstrained tree can reach close to 100% training accuracy while performing noticeably worse on the test set — the most visually obvious overfitting example this course covers, more intuitive than either regression model's own more abstract version of the same failure.

Random Forests — Averaging Away the Overfitting

A random forest builds many individual trees — each trained on a random, bootstrapped subset of the training rows, and restricted to a random subset of features at each split — then combines their predictions (majority vote for classification, average for regression).

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)

Why this actually works: each individual tree, trained on its own random slice of data, tends to overfit in its own idiosyncratic way — memorizing quirks specific to whichever rows and features it happened to see. Averaged across many differently-trained trees, those individual idiosyncrasies tend to cancel each other out, while the genuine, real pattern every tree independently rediscovers — ds1-6's own variance vocabulary applies directly here — survives the averaging. The forest's own predictions end up with meaningfully lower variance than any single tree's own predictions, without needing to constrain any individual tree very much at all.

Hands-On Exercises