[Classification/Regression]
Support Vector Machine (SVM).
Finds the optimal hyperplane that maximally separates classes.
SPEC SHEET
FIG — THE MECHANISM, LIVE
A margin-based boundary between the classes - SVM would place it to maximise the corridor between the two groups.
C — How it actually works
Find the widest possible "street" separating the classes and take its centre line as the boundary - only the points on the kerb (the support vectors) matter. When no straight street exists, the kernel trick implicitly lifts the data into a higher-dimensional space where one does, without ever computing that space.
D — The math
Minimise ½‖w‖² + C·Σξᵢ subject to yᵢ(wᵀxᵢ + b) ≥ 1 − ξᵢ. The dual depends on data only through inner products, so replace them with a kernel K(xᵢ,xⱼ); RBF kernel K = exp(−γ‖xᵢ−xⱼ‖²).
Training
O(n²)–O(n³) for kernel SVM - the scaling wall
Inference
O(sv·d), sv = support vectors
E — When NOT to use it
- More than ~50-100k samples with a kernel - training time explodes; use LinearSVC or boosting
- You need probability estimates (Platt scaling is a bolted-on afterthought)
- Data is mostly noise with heavy overlap - the margin concept stops meaning much
F — Tuning that matters
- Grid C and gamma on log scales (C: 0.1-1000, gamma: 1e-4-1); they interact strongly
- ALWAYS standardise features first - RBF distances are meaningless across scales
- Try LinearSVC on high-dimensional sparse data before any kernel: text is usually linearly separable
- Cache size and shrinking flags matter for wall-clock time on medium data
G — Production pitfalls
- Skipping feature scaling (the single most common SVM bug)
- Using default C=1, gamma="scale" and concluding "SVM does not work"
- Kernel SVM on a million rows - it will not finish
- Forgetting that class_weight matters just as much here for imbalance
H — Minimal starting point
PYTHONfrom sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
clf = make_pipeline(StandardScaler(),
SVC(C=10, gamma=0.01, class_weight="balanced"))
clf.fit(X_train, y_train)I — The interview question
What is the kernel trick, actually?
The dual optimisation touches data only through inner products xᵢᵀxⱼ. A kernel function computes the inner product of a high-dimensional (even infinite, for RBF) mapping without materialising it - so you get a non-linear boundary for the price of a function evaluation.
J — In the wild
Bioinformatics labs still default to RBF-SVMs for gene-expression classification: thousands of features, a few hundred samples - exactly the wide-short regime where margins beat both trees and deep nets.
K — Consider instead
- Logistic Regression— linear kernel + probabilities + faster on big sparse data
- Gradient Boosting (XGBoost/LightGBM)— no scaling requirements and better accuracy on most tabular sets
Other classification/regression options