[Classification/Regression]
Decision Tree.
Creates a tree-like model of decisions based on feature values.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Axis-aligned splits carve the space into rectangles - each additional split sharpens a region.
C — How it actually works
Play twenty-questions with your data. At every node the tree asks the single yes/no question that best un-mixes the classes (or reduces variance for regression), splits the data, and recurses. Predictions follow the questions down to a leaf. The result is a flowchart a domain expert can audit line by line.
D — The math
Greedy recursive partitioning: at each node choose feature j and threshold t maximising impurity decrease, with Gini G = 1 − Σpᵢ² or entropy H = −Σpᵢ log pᵢ. Regression trees minimise within-leaf variance.
Training
O(n·d·log n)
Inference
O(depth) ≈ O(log n)
E — When NOT to use it
- You need the best accuracy - a single tree is almost always beaten by its ensembled versions
- Smooth linear relationships dominate (a tree approximates a line with clumsy stair-steps)
- Small data with noisy labels - deep trees will memorise the noise
F — Tuning that matters
- Control complexity with min_samples_leaf (start 1-5% of data) rather than max_depth alone
- ccp_alpha (cost-complexity pruning) is the principled way to prune - cross-validate it
- For imbalance, set class_weight instead of resampling first
- Cap max_depth at 3-4 when the deliverable is a human-readable flowchart
G — Production pitfalls
- Deploying an unpruned tree - the canonical overfitting machine
- Instability: tiny data changes produce a totally different tree, so do not over-narrate one tree’s structure
- Reading feature_importances_ as causal truth - they are biased toward high-cardinality features
- Believing the tree extrapolates: outside the training range every prediction is a constant leaf value
H — Minimal starting point
PYTHONfrom sklearn.tree import DecisionTreeClassifier, plot_tree
clf = DecisionTreeClassifier(min_samples_leaf=20, ccp_alpha=0.001)
clf.fit(X_train, y_train)
plot_tree(clf, feature_names=feats, filled=True) # audit it visuallyI — The interview question
Why are decision trees high-variance, and what fixes it?
The greedy splitting means one different early split cascades into an entirely different tree - small data perturbations change the model a lot. Averaging many decorrelated trees (bagging → random forest) collapses that variance while keeping the low bias.
J — In the wild
Hospital triage protocols are literally shallow decision trees on purpose: a four-level tree predicting deterioration risk can be printed on a laminated card, executed by a nurse with no computer, and defended in a clinical audit.
K — Consider instead
- Random Forest— same building block, dramatically better accuracy and stability
- Gradient Boosting (XGBoost/LightGBM)— when you can trade a little interpretability for state-of-the-art tabular accuracy
Other classification/regression options