[Classification/Regression]
Random Forest.
Ensemble of decision trees that votes on predictions, reducing overfitting.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Many noisy trees vote; averaging their rectangles smooths the boundary and cancels individual mistakes.
C — How it actually works
Train hundreds of deliberately different trees - each on a bootstrap sample of the rows and a random subset of features per split - then let them vote. Individual trees overfit in different directions; averaging cancels their errors. It is the "ask a diverse crowd, not one expert" principle, formalised.
D — The math
Bagging plus feature subsampling: each tree sees a bootstrap sample; each split considers only m ≈ √d features. Prediction = majority vote / mean. Variance of the average falls like ρσ² + (1−ρ)σ²/B, so decorrelating trees (small ρ) is the whole trick.
Training
O(B·n·d·log n), embarrassingly parallel
Inference
O(B·depth)
E — When NOT to use it
- Hard latency budgets - hundreds of trees per prediction is slow without distillation
- You must explain individual decisions precisely (use a shallow tree or linear model, or add SHAP)
- Very high-dimensional sparse text - linear models and boosting usually win there
- You need predictions outside the training range - forests cannot extrapolate at all
F — Tuning that matters
- n_estimators: more never hurts accuracy, only time - 300-500 is a solid default
- max_features is the main knob: √d for classification, d/3 for regression, lower = more decorrelation
- Use oob_score_=True for a free validation estimate without a holdout split
- min_samples_leaf 1-5 for classification; larger for smoother regression
G — Production pitfalls
- Trusting impurity-based feature importances - they inflate high-cardinality features; prefer permutation importance
- Using forests on time series without time-aware splits (leakage makes them look brilliant, then they fail live)
- Assuming OOB error replaces proper cross-validation under group/time structure
- Shipping a 2 GB pickled forest when 50 trees would have matched accuracy
H — Minimal starting point
PYTHONfrom sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(
n_estimators=400, max_features="sqrt",
min_samples_leaf=2, oob_score=True, n_jobs=-1)
clf.fit(X_train, y_train)
print(clf.oob_score_) # free validation estimateI — The interview question
Why subsample features per split when bagging already subsamples rows?
With row-bagging alone, one dominant feature tops every tree, so the trees stay highly correlated and averaging barely reduces variance. Forcing each split to choose among a random feature subset makes trees genuinely different - and variance reduction scales with decorrelation.
J — In the wild
Kaggle-era fraud teams at payment processors shipped random forests for years: near-boosting accuracy, almost no tuning, robust to junk features, and parallel training that fits an overnight batch window.
K — Consider instead
- Gradient Boosting (XGBoost/LightGBM)— usually a few points more accurate on tabular data, at the cost of tuning care
- Decision Tree— when a single auditable flowchart matters more than accuracy
Other classification/regression options