[Deep Learning]
Neural Network (MLP).
Multi-layer perceptron that learns complex non-linear patterns.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Training is gradient descent on a bumpier surface - same idea, millions of dimensions.
C — How it actually works
Stack layers of weighted sums and simple non-linearities, and let gradient descent shape them into whatever function the data demands. Each layer re-represents the input a little more abstractly; depth composes simple bends into arbitrarily complex boundaries. The price: lots of data, lots of knobs, and explanations get hard.
D — The math
Layer: a⁽ˡ⁾ = φ(W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾), φ usually ReLU. Universal approximation says one wide hidden layer suffices in principle; backpropagation applies the chain rule to get all gradients in one backward pass; Adam/SGD do the descent.
Training
O(epochs · n · parameters)
Inference
O(parameters)
E — When NOT to use it
- Small tabular datasets - gradient boosting wins there embarrassingly often
- Strict interpretability or audit requirements
- No GPU budget and tight latency on CPU
- When a linear baseline has not even been tried yet
F — Tuning that matters
- Get one batch to overfit first - if the net cannot memorise 32 samples, the wiring is broken
- Tune learning rate before anything else, on a log scale; use warmup + cosine decay
- BatchNorm/LayerNorm + early stopping + dropout cover most regularisation needs
- Two hidden layers of 64-256 units is plenty for most tabular problems - resist depth for its own sake
G — Production pitfalls
- Judging by training loss while validation quietly diverges
- Forgetting to scale inputs - first-layer gradients get wrecked
- Random-seed roulette: report mean ± std over seeds, not the best run
- Reaching for a neural net because it sounds serious, on 3,000 tabular rows
H — Minimal starting point
PYTHONimport torch.nn as nn
model = nn.Sequential(
nn.Linear(d_in, 128), nn.ReLU(), nn.BatchNorm1d(128),
nn.Dropout(0.2),
nn.Linear(128, 64), nn.ReLU(),
nn.Linear(64, n_classes))
# train with AdamW(lr=3e-4), early stop on validation lossI — The interview question
Why do deep nets need non-linear activations?
Without them, stacked linear layers collapse algebraically into one linear map - W₃W₂W₁ is just another matrix - so depth adds nothing. The non-linearity between layers is what lets composition build genuinely new, more expressive functions.
J — In the wild
Streaming platforms rank home-screen rows with mid-sized feed-forward nets over user/item embeddings: the embedding layers do the heavy lifting, and the MLP head turns them into a click-probability - retrained daily on billions of events.
K — Consider instead
- Gradient Boosting (XGBoost/LightGBM)— the honest default for tabular data
- Transformer— when the input is sequences and pretrained models exist