[Regression]
Linear Regression.
Fits a linear relationship between input features and a continuous target variable.
SPEC SHEET
FIG — THE MECHANISM, LIVE
The loss surface for a line fit - gradient descent steps downhill until the error stops shrinking.
C — How it actually works
Draw the single straight line (or hyperplane) through your data that makes the smallest total squared mistake. Each coefficient says "hold everything else fixed - when this feature goes up by one unit, the prediction moves by this much". That readability is the entire appeal: the model IS its explanation.
D — The math
Minimise the residual sum of squares: RSS = Σ(yᵢ − Xᵢβ)². Closed-form solution β = (XᵀX)⁻¹Xᵀy; with L2 penalty (ridge) the objective becomes RSS + λ‖β‖², which shrinks coefficients and tames multicollinearity.
Training
O(n·d²) for the normal equation, O(n·d) per epoch with SGD
Inference
O(d)
E — When NOT to use it
- The relationship is clearly non-linear and you cannot engineer features to linearise it
- Heavy outliers dominate the target (squared loss amplifies them - use Huber or quantile loss)
- Features outnumber samples badly without regularisation
- You need calibrated class probabilities - that is logistic regression, not linear
F — Tuning that matters
- Always start with ridge (L2) at a small alpha; move to lasso (L1) only when you want automatic feature elimination
- Standardise features before regularising - penalties are scale-sensitive
- Check residual plots, not just R²: curvature in residuals means missing non-linear terms
- Use statsmodels when you need p-values and confidence intervals; sklearn when you need pipelines
G — Production pitfalls
- Reading causal claims off observational coefficients
- Ignoring multicollinearity - correlated features make individual coefficients meaningless while predictions stay fine
- Extrapolating far outside the training range, where the line has no evidence
- Forgetting that R² always rises when you add features - use adjusted R² or holdout error
H — Minimal starting point
PYTHONfrom sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
model = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
model.fit(X_train, y_train)
print(model.score(X_test, y_test)) # R² on holdoutI — The interview question
Why can adding a feature never decrease training R², and why is that a problem?
OLS can always set the new coefficient to zero, so the fit can only improve or stay equal on training data. That makes training R² useless for feature selection - it rewards complexity unconditionally. Judge on held-out error or penalised criteria (AIC/BIC, adjusted R²).
J — In the wild
Zillow-style home-price baselines still start with regularised linear models: they train in seconds on millions of rows, the coefficients survive legal and compliance review, and they set the bar any fancier model has to beat.
K — Consider instead
- Gradient Boosting (XGBoost/LightGBM)— when relationships are non-linear and you care about accuracy over readability
- Decision Tree— when you want interpretability but the data has thresholds and interactions