[Classification/Regression]
Gradient Boosting (XGBoost/LightGBM).
Sequentially builds trees that correct previous errors, achieving high accuracy.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Trees added one after another, each correcting what the ensemble still gets wrong - the region sharpens round by round.
C — How it actually works
Build the model one small tree at a time, where each new tree is trained on the errors the ensemble is still making. Every round nudges predictions in the direction that most reduces the loss - literally gradient descent, but the "step" is a tree. Modern implementations (XGBoost, LightGBM, CatBoost) are the default winner on tabular data.
D — The math
Additive model F_m(x) = F_{m−1}(x) + ν·h_m(x), where h_m fits the negative gradient of the loss at current predictions (residuals, for squared error). ν is the learning rate; regularisation comes from shrinkage, tree depth, subsampling, and L1/L2 on leaf weights.
Training
O(M·n·d) with histogram tricks; sequential across rounds
Inference
O(M·depth)
E — When NOT to use it
- Tiny noisy datasets - boosting will happily fit the noise
- You need heavy uncertainty quantification out of the box
- Unstructured data (images, audio, raw text) - deep learning owns those
- The team cannot maintain a tuned model - a forest degrades more gracefully
F — Tuning that matters
- Couple learning_rate with n_estimators via early stopping: set lr=0.05, estimators=2000, stop on validation
- Depth 4-8 (or LightGBM num_leaves 31-127) covers most problems
- subsample≈0.8 and colsample_bytree≈0.8 are cheap regularisers
- CatBoost first when you have many categorical features - its target encoding avoids leakage
G — Production pitfalls
- Tuning on the same fold you early-stop on - quietly optimistic results
- Target leakage via naive mean-encoding of categories computed on the full dataset
- Chasing +0.001 AUC with 500-trial sweeps that will not survive data drift
- Ignoring monotonic constraints when the business requires them (price ↑ → risk should not ↓)
H — Minimal starting point
PYTHONimport lightgbm as lgb
model = lgb.LGBMClassifier(
n_estimators=2000, learning_rate=0.05, num_leaves=63,
subsample=0.8, colsample_bytree=0.8)
model.fit(X_train, y_train,
eval_set=[(X_val, y_val)],
callbacks=[lgb.early_stopping(100)])I — The interview question
Boosting vs bagging - what does each reduce?
Bagging trains independent deep trees in parallel and averages them: it reduces variance. Boosting trains shallow trees sequentially, each correcting the last: it reduces bias. That is why forests use deep trees and boosting uses stumps-to-medium trees.
J — In the wild
Every major ride-hailing app prices and matches with gradient-boosted trees: ETA prediction, surge modelling, and fraud all run on LightGBM-class models because tabular features + tight latency + constant retraining is exactly boosting’s home turf.
K — Consider instead
- Random Forest— when you want 95% of the accuracy with 10% of the tuning
- Neural Network (MLP)— when features are unstructured or you need multi-task/embedding learning
Other classification/regression options