[Classification]
Naive Bayes.
Probabilistic classifier based on Bayes theorem with feature independence assumption.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Class regions from per-feature likelihoods - simple assumptions, surprisingly usable boundaries.
C — How it actually works
Flip prediction around with Bayes’ rule: how likely would this input be under each class? Assume every feature contributes independently (the "naive" part - obviously false, surprisingly harmless), multiply the per-feature likelihoods with the class prior, and pick the winner. For word counts, this is a few additions per document.
D — The math
P(y|x) ∝ P(y)·Πⱼ P(xⱼ|y). Multinomial NB for counts, Bernoulli for presence/absence, Gaussian for continuous features. Laplace smoothing (add-α) keeps unseen words from zeroing the product.
Training
O(n·d) - one counting pass
Inference
O(d), effectively instant
E — When NOT to use it
- You need well-calibrated probabilities (NB is famously over-confident - its winner is right, its 0.9999 is not)
- Feature interactions carry the signal ("not good" vs "good")
- Dense correlated numeric features - Gaussian NB gets outclassed fast
F — Tuning that matters
- Tune the smoothing alpha on a log grid (0.01-10) - it is the main knob
- ComplementNB is a drop-in that handles class imbalance on text better
- TF-IDF sometimes helps, raw counts sometimes win - try both, it is cheap
- Cap vocabulary (min_df/max_df) to strip noise words before counting
G — Production pitfalls
- Trusting its probability outputs downstream without recalibration
- Forgetting smoothing and getting hard zeros from unseen features
- Using Gaussian NB on obviously non-Gaussian features without transforms
- Benchmarking a whole NLP project on NB when a linear SVM costs one more line
H — Minimal starting point
PYTHONfrom sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import make_pipeline
clf = make_pipeline(CountVectorizer(min_df=2), MultinomialNB(alpha=0.5))
clf.fit(docs_train, y_train)I — The interview question
Why does naive Bayes work despite the independence assumption being false?
Classification only needs the argmax to be right, not the probabilities. Correlated features inflate confidence in a roughly class-symmetric way, so the ranking of classes survives even when the estimates are badly mis-calibrated.
J — In the wild
The spam filters of the 2000s (and plenty of today’s email routing and language-ID systems) are multinomial naive Bayes: train in one pass on billions of tokens, update online as spam evolves, classify in microseconds.
K — Consider instead
- Logistic Regression— same speed class, better calibrated, handles correlated features
- Transformer— when meaning and word order matter more than word presence