[Classification]
Logistic Regression.
Uses logistic function to model probability of binary or multi-class outcomes.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Logistic regression draws one straight boundary; regions show which side wins and the gradient shows confidence.
C — How it actually works
Linear regression squeezed through a sigmoid: compute a weighted score, then map it to a probability between 0 and 1. The decision boundary is still a straight line - what changes is that the output is a calibrated "how sure am I", and the weights are trained to make observed labels as likely as possible.
D — The math
P(y=1|x) = σ(βᵀx) with σ(z) = 1/(1+e⁻ᶻ). Trained by maximising log-likelihood, equivalently minimising log-loss: −Σ[yᵢ log pᵢ + (1−yᵢ) log(1−pᵢ)]. Coefficients are log-odds: e^β is the odds multiplier per unit of the feature.
Training
O(n·d) per iteration
Inference
O(d)
E — When NOT to use it
- Decision boundary is strongly non-linear and feature crosses cannot fix it
- Classes are perfectly separable - weights diverge without regularisation
- You have millions of sparse one-hot features but need interactions - trees handle those natively
F — Tuning that matters
- C in sklearn is INVERSE regularisation - smaller C = stronger penalty
- class_weight="balanced" is the first lever for imbalanced data, before touching the threshold
- The 0.5 threshold is not sacred: pick it from the precision/recall trade-off your product needs
- liblinear for small data, saga for large sparse data + L1
G — Production pitfalls
- Quoting accuracy on a 99/1 imbalanced problem (predict-all-negative scores 99%)
- Interpreting coefficients as probabilities instead of log-odds
- Skipping calibration checks - regularisation can distort probability calibration
- One-hot encoding high-cardinality ids and wondering why the model memorises
H — Minimal starting point
PYTHONfrom sklearn.linear_model import LogisticRegression
clf = LogisticRegression(C=1.0, class_weight="balanced", max_iter=1000)
clf.fit(X_train, y_train)
proba = clf.predict_proba(X_test)[:, 1] # calibrated-ish scoresI — The interview question
Why log-loss instead of squared error for classification?
Squared error on probabilities is non-convex through the sigmoid and its gradients vanish when the model is confidently wrong. Log-loss is convex, punishes confident mistakes hard, and its gradient (p − y) keeps learning fast exactly when the model is most wrong.
J — In the wild
Most bank credit-scoring models in production are still regularised logistic regressions: regulators demand reason codes for every decline, and log-odds coefficients translate directly into "your utilisation raised your risk score by X".
K — Consider instead
- Gradient Boosting (XGBoost/LightGBM)— nearly always more accurate on tabular data when interpretability can relax
- Naive Bayes— for text with tiny training sets, it is an even faster baseline