[Deep Learning]
RNN/LSTM.
Neural networks designed for sequential data with memory.
SPEC SHEET
FIG — THE MECHANISM, LIVE
Backpropagation through time is still gradient descent - the gates exist to keep this signal alive across steps.
C — How it actually works
Read a sequence one step at a time, carrying a memory vector forward. An LSTM adds trainable gates that decide what to write into memory, what to erase, and what to reveal - so signals can survive across hundreds of steps instead of dissolving. For order-matters data with modest scale, it is still a strong, cheap tool.
D — The math
LSTM cell: forget gate f, input gate i, output gate o (all sigmoids of [h_{t−1}, x_t]); cell state c_t = f⊙c_{t−1} + i⊙tanh(W[h_{t−1},x_t]); h_t = o⊙tanh(c_t). The additive cell-state path is what defeats vanishing gradients.
Training
O(T · h²) per sequence - inherently sequential, hard to parallelise
Inference
O(T · h²), streamable step by step
E — When NOT to use it
- Language tasks where pretrained transformers exist (they win, usually by a lot)
- Very long sequences (thousands of steps) - attention handles distance better
- When training throughput matters and you have GPUs sitting idle (RNNs cannot use them well)
F — Tuning that matters
- GRU first - two gates instead of three, similar accuracy, faster
- 1-2 layers of hidden size 64-512 covers most non-language sequence tasks
- Clip gradient norm at ~1.0; exploding gradients remain real
- Bidirectional wrappers help whenever the future of the sequence is available at prediction time
G — Production pitfalls
- Shuffling away temporal order in batching, or leaking future windows into training
- Padding chaos: mask properly or short sequences train on garbage tails
- Normalising with statistics computed over the whole series (future leakage)
- Reaching for LSTM when the true dependency is 3 steps long - a small CNN or features would do
H — Minimal starting point
PYTHONimport torch.nn as nn
class SeqModel(nn.Module):
def __init__(self, d_in, h=128, n_out=1):
super().__init__()
self.rnn = nn.GRU(d_in, h, num_layers=2, batch_first=True)
self.head = nn.Linear(h, n_out)
def forward(self, x): # x: (batch, time, features)
out, _ = self.rnn(x)
return self.head(out[:, -1])I — The interview question
How exactly does the LSTM cell state fight vanishing gradients?
The cell state updates additively - c_t = f⊙c_{t−1} + i⊙ĉ_t - so the backward gradient flows through an elementwise multiply by the forget gate rather than repeated full matrix multiplications. With f near 1, gradients pass across many steps nearly unchanged: a learnable "gradient highway".
J — In the wild
Wearables decode heart-rhythm anomalies with small GRUs running on-device: the sequential nature fits streaming sensor data, the model fits in kilobytes, and inference sips battery - places a transformer still struggles to live.
K — Consider instead
- Transformer— any language task, and long sequences generally
- Gradient Boosting (XGBoost/LightGBM)— engineered lag/window features often beat RNNs on business time series