[Deep Learning]
Transformer.
Attention-based architecture that powers modern LLMs and NLP.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Pretraining is this walk at planetary scale - billions of parameters descending one loss surface.
C — How it actually works
Drop recurrence entirely: every token looks at every other token at once and decides, via attention, which ones matter for its meaning. "Bank" attends to "river" and resolves itself. Because nothing is sequential, training parallelises across the whole sequence - which is what made pretraining on internet-scale text possible, and that pretraining is the real superpower you inherit.
D — The math
Attention(Q,K,V) = softmax(QKᵀ/√d_k)V, with multiple heads learning different relation types; positional encodings inject order; stacked blocks of attention + feed-forward with residuals and layer-norm. Cost is quadratic in sequence length.
Training
O(T²·d) per layer - the quadratic attention wall
Inference
same; KV-caching makes generation O(T·d) per new token
E — When NOT to use it
- Tabular business data - boosting still wins there
- Tiny latency/memory budgets on edge hardware
- A few hundred labelled examples AND no pretrained model for your domain/language
- When a regex or a linear model already solves it - transformer maintenance is not free
F — Tuning that matters
- Never pretrain yourself - fine-tune (or LoRA/adapter-tune) an existing checkpoint
- Fine-tuning learning rates live around 1e-5 to 5e-5 with warmup; higher rates catastrophically forget
- LoRA/QLoRA fine-tunes billon-parameter models on a single GPU - default to it
- For classification, try zero/few-shot prompting of an instruction model before training anything
G — Production pitfalls
- Fine-tuning on contaminated data (test set leaked into pretraining or prompts)
- Ignoring tokenisation - domain jargon shredded into subwords quietly hurts
- Serving cost surprise: attention memory grows with context length, concurrency multiplies it
- Treating generative output as deterministic - temperature and sampling change everything
H — Minimal starting point
PYTHONfrom transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer
tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased", num_labels=2)
# tokenize dataset, then Trainer(...).train() - fine-tuning, never from scratchI — The interview question
Why divide by √d_k inside attention?
Dot products of d_k-dimensional random vectors have variance proportional to d_k, so raw scores grow with dimension and push softmax into saturation where gradients die. Scaling by √d_k keeps score variance near 1, softmax stays soft, and training stays stable.
J — In the wild
Every frontier assistant - ChatGPT, Claude, Gemini - is a decoder-only transformer; so are the code-completion models in your editor and the embedding models behind enterprise search. One architecture, pretrained at scale, fine-tuned everywhere.
K — Consider instead
- RNN/LSTM— streaming/edge sequence tasks with tight compute
- Naive Bayes— as the honest 5-minute text baseline before anything heavy
Other deep learning options