[Dimensionality Reduction]
Principal Component Analysis (PCA).
Transforms data to lower dimensions while preserving maximum variance.
SPEC SHEET
C — How it actually works
Rotate the axes to point along the directions where the data actually varies, ordered by how much variance each direction carries. Keep the first few, drop the rest, and you have compressed the data with the least possible information loss (in the linear, squared-error sense). One rotation - no learning loop, no local minima.
D — The math
Eigendecomposition of the covariance matrix (or SVD of centred X): components = top eigenvectors, explained variance = eigenvalues. Projection Z = XW keeps the top-k columns.
Training
O(n·d²) or O(n·d·k) with truncated SVD
Inference
O(d·k)
E — When NOT to use it
- The signal lives on a curved manifold (use UMAP/t-SNE/autoencoders for that)
- You need the reduced features to stay individually meaningful to stakeholders
- Variance ≠ importance for your task: low-variance directions can carry the label signal
F — Tuning that matters
- ALWAYS standardise first, or the highest-variance raw feature owns PC1
- Keep components covering 90-95% cumulative variance, or read the scree-plot elbow
- Use randomised/truncated SVD for wide or sparse matrices
- Fit PCA inside the CV pipeline - fitting on all data before splitting is leakage
G — Production pitfalls
- Interpreting components as causal factors - they are variance directions, nothing more
- Dropping "small" components that happen to hold the discriminative signal
- Applying PCA to one-hot categoricals and calling the result meaningful
- Reporting a 2-component scatter that explains 30% variance as "the data"
H — Minimal starting point
PYTHONfrom sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95))
Z = pipe.fit_transform(X) # keeps 95% of variance
print(pipe[-1].explained_variance_ratio_)I — The interview question
Why must you standardise before PCA?
PCA maximises variance, and variance has units. A feature measured in a big-number scale (salary in rupees) dwarfs one in a small scale (age in years), so PC1 just becomes that feature. Standardising puts all features on unit variance so directions reflect structure, not units.
J — In the wild
Quant finance compresses hundreds of correlated yield-curve and factor series with PCA daily - the first three components ("level, slope, curvature") are so stable they are named, monitored, and traded on.
K — Consider instead
- t-SNE— for 2-D visualisation of local neighbourhood structure
- Neural Network (MLP)— autoencoders when the manifold is non-linear and data is plentiful