Recurrent Neural Networks Explained for Sequential Data

Recurrent neural networks (RNNs) model a sequence by carrying a learned hidden state from one time step to the next. That state gives the network memory of prior inputs, but repeated nonlinear updates also make gradients vanish or explode, which is why LSTMs and GRUs often replace vanilla RNNs for longer dependencies.

Key Takeaways

Recurrent neural networks are neural models for ordered data: they consume one element at a time and update a hidden state that summarizes the preceding context. The central design choice is parameter sharing across time, which lets one model process sequences of different lengths without creating separate parameters for each position.

• An RNN models order through recurrence. At time step $t$, the hidden state $h_t$ depends on both the current input $x_t$ and the previous state $h_{t-1}$, so the same input can produce different representations depending on its history.

• The hidden state is a learned, compressed memory. It is not a literal transcript of the past; it is a fixed-width vector optimized to preserve information useful for the training objective.

• Weight sharing makes sequence processing economical. A single transition matrix is reused at every time step, allowing the model to process variable-length sequences with parameter counts independent of sequence length.

• Training requires backpropagation through time. The model is unrolled across time, and gradients flow through every recurrent transition. Long chains multiply Jacobian matrices, causing vanishing or exploding gradients.

• Vanilla RNNs are weak at long-term dependencies. A distant input must survive repeated state updates and nonlinear transformations before influencing a later prediction.

• LSTMs add a separately controlled cell state. Input, forget, and output gates regulate what information is written, retained, and exposed, creating a path through which gradients can travel more easily.

• GRUs simplify the LSTM mechanism. A GRU combines memory and hidden representation into one state and uses update and reset gates, often reducing computation and parameter count while retaining gated memory.

What Problem Do Recurrent Neural Networks Solve?

Recurrent neural networks solve the problem of learning from ordered, variable-length observations when the meaning of the current input depends on preceding inputs. Instead of treating each observation independently, an RNN applies the same transition repeatedly and carries a hidden state that summarizes relevant history.

A feedforward network maps a fixed input vector to an output:

$$y = f(x; \theta)$$

That formulation does not inherently distinguish the sequences A, B, C and C, B, A if both are represented by the same unordered collection. One workaround is to concatenate a fixed number of time steps into a large vector, but this creates two problems:

1. The maximum sequence length must be chosen in advance.
2. The number of input features grows with the window size.

A feedforward model can process a sliding window, such as the last 24 hourly temperatures, but it has no natural mechanism for retaining information from hour 25 or for sharing the same temporal operation across every position. Padding and masking can make lengths compatible, but they do not themselves create memory.

An RNN introduces a state transition:

$$h_t = \phi(W_{xh}x_t + W_{hh}h_{t-1} + b_h)$$

Here, $x_t$ is the input at time $t$, $h_{t-1}$ is the previous hidden state, $W_{xh}$ maps the input into the hidden space, $W_{hh}$ maps the prior state back into that space, $b_h$ is a bias, and $\phi$ is usually $\tanh$ in a vanilla RNN. The state $h_t$ becomes the model’s representation of the prefix $x_1,\ldots,x_t$.

The same matrices $W_{xh}$ and $W_{hh}$ are reused at every time step. This is the sequential analogue of applying the same convolutional filter at every spatial location. The reuse imposes an inductive bias: the rule for updating context should not depend on whether an event occurs at position 3 or position 300.

Sequence tasks and output layouts

The output structure depends on the task.

Task	Inputs	Typical output	Example
Many-to-one	Entire sequence	One vector or label	Sentiment classification
One-to-many	Initial vector or token	Sequence	Caption generation
Many-to-many, aligned	Sequence	One output per step	Part-of-speech tagging
Many-to-many, delayed	Input sequence	Output sequence with different timing	Machine translation

For sequence classification, the final hidden state may feed a classifier:

$$\hat{y} = \operatorname{softmax}(W_{hy}h_T + b_y)$$

For per-time-step prediction, the model produces an output at every step:

$$o_t = W_{hy}h_t + b_y$$

The output layer need not share the hidden state’s dimensionality. A language model might use a hidden vector of width 1,024 and project it to 50,000 vocabulary logits.

RNNs are useful when the data-generating process has temporal or sequential structure: sensor measurements, transactions, words, audio frames, user events, or physiological signals. The key question is not whether data has timestamps, but whether earlier observations change the interpretation of later ones.

For example, in the sentence “The bank approved the loan,” the token “bank” is ambiguous in isolation. A recurrent model can let preceding words influence the state used to represent it. In forecasting, a sudden temperature change may matter differently depending on whether the preceding hours were stable or already trending downward.

How Does an RNN Hidden State Carry Context?

An RNN hidden state carries context by repeatedly transforming the current input together with the previous state. The state is a fixed-size summary, the output layer reads that summary to make predictions, and parameter sharing makes the same update rule operate at every position in an unrolled sequence.

The canonical vanilla RNN equations are:

$$a_t = W_{xh}x_t + W_{hh}h_{t-1} + b_h$$ $$h_t = \tanh(a_t)$$ $$o_t = W_{hy}h_t + b_y$$ $$\hat{y}_t = g(o_t)$$

The hidden preactivation $a_t$ combines new evidence with prior context. The nonlinear function $\tanh$ bounds each hidden coordinate between $-1$ and $1$. The output activation $g$ depends on the task: softmax for mutually exclusive classes, sigmoid for binary labels, or the identity function for regression.

Suppose the input sequence is the words “dogs chase cats.” At the first step, $h_1$ encodes information from “dogs.” At the second, $h_2$ is computed from both “chase” and the information carried in $h_1$. At the third, $h_3$ receives “cats” and the state representing the preceding phrase. No separate “position-1” or “position-2” weight matrices exist.

A useful mathematical interpretation is that the state is a deterministic function of the prefix:

$$h_t = F_\theta(x_1,\ldots,x_t)$$

The function $F_\theta$ is not guaranteed to preserve every fact in the prefix. If the hidden dimension is small or later updates overwrite earlier information, two different prefixes can map to nearly identical states. “Memory” therefore means learned representational capacity, not unlimited storage.

Unrolling the recurrence

During inference, the recurrence can be executed as a loop. During training, it is often visualized as an unrolled computation graph:

x1 ---> [RNN cell] ---> h1 ---> y1
             |
x2 ---> [RNN cell] ---> h2 ---> y2
             |
x3 ---> [RNN cell] ---> h3 ---> y3

The cells look repeated because they are repeated applications of the same parameters. They are not independent copies with independent learned weights. Unrolling duplicates graph nodes, not parameter tensors.

For a batch of sequences, the input commonly has shape (batch, time, features) in a batch-first API. A recurrent layer with hidden width $H$ receives $x_t \in \mathbb{R}^{D}$ and produces $h_t \in \mathbb{R}^{H}$. The recurrent parameters include:

• $W_{xh} \in \mathbb{R}^{H \times D}$
• $W_{hh} \in \mathbb{R}^{H \times H}$
• $b_h \in \mathbb{R}^{H}$

The recurrent parameter count is therefore approximately:

$$HD + H^2 + H$$

For an output vocabulary of size $V$, the output projection adds approximately $VH + V$ parameters. In language modeling, that projection can dominate the recurrent core when $V$ is large.

Context is compressed, not retrieved

An RNN does not normally retrieve an arbitrary earlier hidden state during inference. It must preserve useful information in the current state. This distinction matters when interpreting its behavior:

• A state can retain a running total efficiently because each update can add to the total.
• A state may struggle to retain an exact token from 500 steps earlier because every intermediate transition can alter the representation.
• A larger hidden width supplies capacity but does not guarantee that optimization will discover a stable memory strategy.

This compression explains both the appeal and the limitation of recurrence. The model processes a stream with bounded working memory, but the bounded state creates an information bottleneck.

How Are Recurrent Neural Networks Trained?

Recurrent neural networks are trained by minimizing a loss over outputs generated at one or more time steps, then propagating gradients backward through the unrolled recurrence. Backpropagation through time accounts for the fact that one shared parameter set influenced every position, while truncated BPTT limits the backward horizon for long streams.

For a sequence of targets $y_1,\ldots,y_T$, a per-step cross-entropy objective is commonly written as:

$$\mathcal{L}(\theta) = \sum_{t=1}^{T} \ell(\hat{y}_t,y_t)$$

For language modeling, the target at time $t$ is often the next token $x_{t+1}$:

$$\mathcal{L} = -\sum_{t=1}^{T-1}\log p_\theta(x_{t+1}\mid x_{\leq t})$$

The total loss may be averaged over valid tokens rather than summed, especially when batches contain padding. A mask excludes padded positions:

$$\mathcal{L} = -\frac{1}{\sum_t m_t} \sum_{t=1}^{T}m_t\log p_\theta(y_t\mid x_{\leq t})$$

where $m_t \in \{0,1\}$ indicates whether position $t$ is real data.

Backpropagation through time

Backpropagation through time, or BPTT, applies ordinary reverse-mode automatic differentiation to the unrolled graph. Although the same matrix $W_{hh}$ appears at every step, its gradient accumulates contributions from every use:

$$\frac{\partial \mathcal{L}}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial \mathcal{L}}{\partial h_t} \frac{\partial h_t}{\partial W_{hh}}$$

The hidden state also receives gradient from future states. With a simplified recurrence $h_t = f(h_{t-1},x_t)$, the influence of $h_k$ on $h_T$ contains a product:

$$\frac{\partial h_T}{\partial h_k} = \prod_{t=k+1}^{T} \frac{\partial h_t}{\partial h_{t-1}}$$

This product is the mathematical source of the long-term dependency problem discussed later. It is also why unrolling a sequence consumes memory: the training system must retain intermediate activations or recompute them.

A minimal PyTorch-style training loop looks like this:

import torch
import torch.nn as nn

class VanillaRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.x_to_h = nn.Linear(input_size, hidden_size)
        self.h_to_h = nn.Linear(hidden_size, hidden_size)
        self.h_to_y = nn.Linear(hidden_size, output_size)

    def forward(self, x, h=None):
        # x: [batch, time, input_size]
        batch_size, time, _ = x.shape
        if h is None:
            h = x.new_zeros(batch_size, self.hidden_size)

        outputs = []
        for t in range(time):
            h = torch.tanh(self.x_to_h(x[:, t]) + self.h_to_h(h))
            outputs.append(self.h_to_y(h))

        return torch.stack(outputs, dim=1), h

model = VanillaRNN(32, 128, 10)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

logits, final_state = model(inputs)
loss = criterion(logits.reshape(-1, 10), targets.reshape(-1))
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

Gradient clipping is a practical defense against exploding gradients. It rescales the gradient vector when its norm exceeds a threshold:

$$g \leftarrow g \min\left(1,\frac{\tau}{\lVert g\rVert}\right)$$

Clipping does not solve vanishing gradients or make the underlying recurrence stable; it prevents an unusually large update from destabilizing the optimizer.

Truncated BPTT

For an indefinitely running stream, retaining the entire history is infeasible. Truncated BPTT divides the stream into chunks of length $K$, computes forward states across chunks, and stops gradients at chunk boundaries. The state can continue numerically from one chunk to the next while its computational graph is detached:

state = None

for inputs, targets in stream:
    logits, state = model(inputs, state)
    loss = criterion(logits.reshape(-1, vocab_size),
                     targets.reshape(-1))

    optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()

    # Preserve the value of state, but not its old graph.
    state = state.detach()

Truncation creates a deliberate distinction between forward memory and backward credit assignment. The model may receive information carried from earlier chunks, but the current update cannot directly adjust parameters based on losses beyond the truncation horizon. If $K=128$, a dependency of 1,000 steps can affect the current state numerically but receives no direct gradient path through all 1,000 steps.

Why Do Vanilla RNNs Struggle With Long-Term Dependencies?

Vanilla RNNs struggle with long-term dependencies because gradients through many recurrent steps are products of state-transition Jacobians. If their typical singular values are below one, gradients shrink exponentially; if above one, they grow exponentially, producing vanishing or exploding gradients and making distant information difficult to learn.

For the recurrence

$$h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t+b_h)$$

the Jacobian with respect to the previous state is:

$$\frac{\partial h_t}{\partial h_{t-1}} = \operatorname{diag}\left(1-\tanh^2(a_t)\right)W_{hh}$$

The gradient from time $k$ to time $T$ multiplies these Jacobians:

$$\frac{\partial \mathcal{L}_T}{\partial h_k} = \frac{\partial \mathcal{L}_T}{\partial h_T} \prod_{t=k+1}^{T} \frac{\partial h_t}{\partial h_{t-1}}$$

If the norm of each factor is roughly $0.9$, after 100 steps the corresponding scale is about $0.9^{100}\approx 2.7\times 10^{-5}$. If it is $1.1$, the scale is approximately $1.1^{100}\approx 13,780$. Actual networks have nonuniform matrices and nonlinearities, but the exponential sensitivity remains.

The $\tanh$ derivative is largest near zero and approaches zero when the preactivation saturates near $-1$ or $1$. A saturated hidden unit therefore transmits little gradient. ReLU avoids saturation on its positive side, but recurrent ReLU networks can create unstable state growth unless initialization and regularization are handled carefully.

Why optimization becomes unstable

Suppose a useful event at step 10 should affect a prediction at step 500. The model must learn all of the following:

1. Detect the event.
2. Encode it in the current state.
3. Avoid overwriting it over 490 subsequent updates.
4. Preserve a gradient signal connecting the final loss to the earlier event.
5. Use that signal to adjust shared parameters that also affect every other step.

That is a difficult optimization problem because the same recurrent weights must serve as both an update mechanism and a memory-retention mechanism. A transition that reacts quickly to new inputs may overwrite old information; a transition that preserves information may fail to incorporate new evidence.

Exploding gradients are easier to observe: training loss can become NaN, hidden activations can grow, or an optimizer step can cause a sudden collapse. Gradient clipping, smaller learning rates, orthogonal initialization, and normalization can help.

Vanishing gradients are more subtle. Training may remain numerically stable while the model learns only short-range patterns. For character modeling, it might predict local spelling regularities but fail to maintain agreement across a long clause. For forecasting, it might respond to the last few measurements while ignoring a slowly changing seasonal regime.

The problem is not that an RNN mathematically cannot represent long dependencies. With sufficient state dimension and a suitable transition, it can. The practical issue is that gradient-based learning has difficulty discovering and maintaining the required computation.

How Do LSTMs and GRUs Improve Recurrent Modeling?

LSTMs and GRUs improve recurrent modeling by adding gates that learn when to write, retain, reset, and expose state. LSTMs use a separate cell state with three gates; GRUs use a simpler single state with update and reset gates, usually reducing parameters and computation at the cost of less explicit memory control.

Long short-term memory

A long short-term memory network, or LSTM, maintains a cell state $c_t$ and a hidden state $h_t$. Its common equations are:

$$i_t = \sigma(W_{xi}x_t + W_{hi}h_{t-1}+b_i)$$ $$f_t = \sigma(W_{xf}x_t + W_{hf}h_{t-1}+b_f)$$ $$o_t = \sigma(W_{xo}x_t + W_{ho}h_{t-1}+b_o)$$ $$\tilde{c}_t = \tanh(W_{xc}x_t + W_{hc}h_{t-1}+b_c)$$ $$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$$ $$h_t = o_t \odot \tanh(c_t)$$

The input gate $i_t$ controls candidate writes, the forget gate $f_t$ controls retention of the old cell state, and the output gate $o_t$ controls exposure of memory to the hidden state. The elementwise addition in the cell update creates a relatively direct path from $c_{t-1}$ to $c_t$. If $f_t$ remains near one, information and gradients can persist across many steps.

This architecture was introduced by Hochreiter and Schmidhuber in 1997. The later forget-gate formulation became standard after work including Gers, Schmidhuber, and Cummins (2000). The important mechanism is not simply “more layers”; it is learned additive state updates.

Gated recurrent units

A gated recurrent unit, or GRU, removes the separate cell state and combines memory with the hidden state. A common formulation is:

$$z_t = \sigma(W_{xz}x_t + W_{hz}h_{t-1}+b_z)$$ $$r_t = \sigma(W_{xr}x_t + W_{hr}h_{t-1}+b_r)$$ $$\tilde{h}_t = \tanh(W_{xh}x_t + W_{hh}(r_t\odot h_{t-1})+b_h)$$ $$h_t = (1-z_t)\odot h_{t-1}+z_t\odot\tilde{h}_t$$

The update gate $z_t$ interpolates between the previous state and a candidate state. The reset gate $r_t$ determines how much prior state contributes when constructing that candidate. Some libraries use the complementary convention, so the semantic label “update” must be checked against the implementation.

Direct comparison

Property	Vanilla RNN	LSTM	GRU
Persistent state	Hidden state $h_t$	Cell state $c_t$ plus hidden state $h_t$	Hidden state $h_t$
Gates	None	Input, forget, output	Update, reset
Recurrent matrix cost	Approximately $H^2$	Approximately $4H^2$	Approximately $3H^2$
Long-memory control	Implicit	Explicit and fine-grained	Gated but more compact
Typical implementation	Simplest	Most complex	Intermediate
Parallelism across time	Sequential	Sequential	Sequential

The gate multipliers refer to a standard dense implementation and ignore details such as projections, biases, fused kernels, and bidirectionality. LSTM and GRU gates are often computed in one fused matrix multiplication by concatenating gate projections, so the practical cost is lower than four separate kernel launches would suggest.

Choice	Favor it when	Main cost
Vanilla RNN	Sequences are short and latency or parameter count dominates	Weak long-range optimization
LSTM	Long dependencies and explicit memory behavior matter	More parameters and state
GRU	You want gated memory with a smaller recurrent core	Less separable memory control
Transformer	Sequence-wide interactions and training throughput matter	Attention memory and deployment cost
Temporal convolution	Fixed receptive fields and parallel training matter	Dependency range tied to architecture

There is no theorem that an LSTM must outperform a GRU. Results depend on sequence length, data volume, hidden width, regularization, and hardware. A useful engineering procedure is to establish a vanilla RNN baseline, compare GRU and LSTM at matched parameter budgets, and measure not just validation loss but throughput, peak memory, and tail latency.

When Should You Use an RNN Today?

Use an RNN today when data arrives as a stream, the model must maintain compact state, sequence lengths are moderate, and low-latency or low-memory inference matters more than unrestricted sequence-wide interaction. Transformers usually win for large-scale text and long-context training, while temporal convolutions can win when fixed receptive fields permit parallel computation.

Transformers replace recurrent state with attention: each token can directly compare itself with other tokens in the context window. This makes training parallel across positions, whereas an RNN must compute $h_t$ before $h_{t+1}$. The trade-off is that standard self-attention has quadratic interaction cost in sequence length, although efficient attention variants alter that scaling.

A temporal convolutional network applies filters over local windows, often with dilation to expand the receptive field. Training is parallel across time, but the model’s effective dependency range is determined by kernel sizes, dilation schedule, and depth. An RNN has a theoretically unbounded stream-oriented state, though its practical memory is limited by optimization.

Application-specific choices

Forecasting. RNNs remain reasonable for multivariate sensor streams, equipment telemetry, and demand signals when inference runs continuously and the forecast state can be updated one observation at a time. LSTMs and GRUs support hidden-state carryover between inference calls. A Transformer may be preferable when cross-series relationships and long seasonal context dominate; a temporal convolution may be preferable when the forecast horizon and receptive field are fixed.

Streaming speech. Causal recurrent models can emit outputs as audio frames arrive without waiting for a complete utterance. This is useful in embedded wake-word detection, online transcription, and voice activity detection. Transformer transducers and streaming attention models are common at larger scales, but their cache and memory management can be more demanding.

Text generation. RNN language models are educationally important and can be effective for small vocabularies, compact devices, or specialized low-latency streams. They are generally disadvantaged for large-scale language modeling because recurrent computation prevents full training parallelism and makes distant token interactions indirect.

Event streams. User actions, transactions, and operational logs naturally form asynchronous sequences. A GRU can update one state per event, avoiding the need to reprocess a large context window. Time gaps can be supplied as features or modeled with continuous-time extensions.

Resource-constrained deployment. A recurrent model can store only its current state rather than a complete token history. For a one-layer GRU with hidden width $H$, the runtime state is roughly $H$ values per direction, aside from buffers. A Transformer decoder must retain key-value caches for prior positions, and those caches grow with context length and layer count.

A recurrent model is not automatically cheaper. Sequential dependence can underutilize a GPU, and a large batch may favor Transformer kernels. On a microcontroller or CPU handling one stream at a time, however, a compact GRU can have a favorable latency and memory profile.

A practical selection test

Measure four quantities on representative hardware:

1. Per-step latency: Can the model finish before the next observation arrives?
2. State memory: How much persistent state must each stream retain?
3. Warm-start behavior: Can inference resume from a saved state?
4. Long-context quality: Does accuracy improve when the historical window grows?

Choose a recurrent model when the stateful streaming advantages are real and measurable. Choose a Transformer when training throughput, flexible long-range interaction, or ecosystem support outweighs cache costs. Choose a temporal convolution when a known finite receptive field enables fast parallel inference.

Frequently Asked Questions

What are the inputs and outputs of an RNN?

An RNN input is usually a sequence tensor with one feature vector per time step. In batch-first notation, its shape is commonly (batch, time, features), although some libraries use (time, batch, features). For text, each time step may begin as a token ID, which is converted into an embedding vector $x_t$. For sensor data, $x_t$ may contain measurements such as temperature, pressure, and acceleration.

The RNN produces a hidden state at every step, often represented as (batch, time, hidden_size), plus a final state. A many-to-one classifier can use the final state; a tagging system can use every state; an autoregressive generator can use the current output to select the next input.

Variable-length sequences are usually handled with padding and a mask, packed-sequence utilities, or a batch of individually processed streams. Padding must not contribute to the loss, and the final state should correspond to the last real item rather than the padded suffix.

The output dimension is independent of the hidden dimension. A 128-unit recurrent state can feed a 10-class classifier, a 3-value regression head, or a 50,000-token language-model projection. The correct shape depends on whether the task predicts once, at every time step, or with an input-output delay.

What is a bidirectional RNN?

A bidirectional RNN runs one recurrent network from left to right and another from right to left, then combines their states. At position $t$, the representation contains information from both the prefix and suffix:

$$h_t = [h_t^{\rightarrow};h_t^{\leftarrow}]$$

This helps when the entire sequence is available. In named-entity recognition, the word “bank” can be interpreted using both preceding and following words. In speech recognition, future audio frames may disambiguate the current frame.

A bidirectional model is not causal. It cannot be used unchanged for live forecasting or streaming transcription because the backward direction requires future observations. It also typically doubles recurrent state and computation, although the exact cost depends on the implementation.

For offline classification, transcription with complete utterances, and sequence labeling, bidirectional LSTMs or GRUs can improve representations. For real-time systems, use a forward-only model or a carefully bounded look-ahead window. A limited look-ahead design is neither fully bidirectional nor fully causal: it trades latency for future context.

When reporting results, specify directionality. A bidirectional model often has an unfair advantage if compared with a causal model on an offline benchmark. The deployment constraint should determine whether future context is legal.

How long can an RNN sequence be?

There is no fixed mathematical maximum sequence length for an RNN, but its useful memory length is limited by state capacity, optimization, numerical stability, and the training horizon. An RNN can process thousands of steps sequentially, yet that does not mean it will accurately preserve information from the first step.

Three lengths should be distinguished:

• Processing length: how many steps the inference loop accepts.
• Backpropagation length: how many steps receive direct gradient through BPTT.
• Effective memory length: how far back information measurably improves predictions.

Truncated BPTT makes the second length explicit. A model trained with chunks of 128 steps may still carry state across chunks, but parameter updates cannot directly assign credit through the complete stream. LSTMs and GRUs increase effective memory by creating gated paths, not by removing the information bottleneck.

The practical answer comes from an ablation: evaluate the model after resetting state, with state carried forward, and with histories of increasing length. If quality stops improving after 80 steps, processing 10,000 steps does not imply 10,000-step memory. For extremely long contexts requiring arbitrary retrieval, attention or external memory is usually a better fit.

How does RNN inference work on a stream?

RNN inference maintains a state vector between observations. For each new input $x_t$, the model computes $h_t = F_\theta(x_t,h_{t-1})$, produces an output, and stores $h_t$ for the next call. This gives constant-size persistent state for a fixed-width recurrent layer, rather than requiring the complete input history.

A production streaming interface often looks like:

def step(model, x_t, state):
    # x_t: [batch, features]
    output, next_state = model(x_t[:, None, :], state)
    return output[:, 0, :], next_state

The state must be reset at sequence boundaries. Failing to reset it can leak information from one user, device session, or document into another. Conversely, resetting too often destroys temporal context.

State serialization matters for fault tolerance and handoff between workers. Quantization can reduce arithmetic cost, but recurrent quantization should be tested for state drift because small errors are fed back repeatedly. Hidden-state values may require a different scale or calibration strategy from ordinary feedforward activations.

Training and inference also differ in input handling. Teacher forcing may feed the true previous token during training, while generation feeds the model’s sampled or selected token. This mismatch can create exposure bias in autoregressive applications.

Are RNNs cheaper than Transformers?

RNNs can be cheaper for single-stream, step-by-step inference with compact state, but they are not universally cheaper. A recurrent layer performs sequential matrix operations across time, which limits parallelism. A Transformer performs more work per layer but can process all positions in parallel during training and often benefits from highly optimized accelerator kernels.

For a dense recurrent layer with hidden width $H$ and input width $D$, each step costs on the order of $HD + H^2$ operations, multiplied by the number of gates for LSTMs or GRUs. During inference, this cost is constant per new step, and the persistent state is fixed-size.

A Transformer decoder must compute attention against cached prior keys and values. Standard causal attention requires work that grows with context length per new token, while its key-value cache grows with the number of layers, heads, head dimension, and cached positions. Efficient attention, recurrent memory, and state-space models change this trade-off.

During training, RNNs cannot freely parallelize the recurrence across time, although fused kernels reduce overhead. Therefore, compare end-to-end throughput and latency on the target device rather than comparing only parameter counts. A small GRU can beat a Transformer on an embedded stream; a Transformer can be faster for a large batch of fixed-length sequences on a GPU.

Should I choose a vanilla RNN, GRU, or LSTM?

Choose a vanilla RNN for short sequences, educational experiments, or a baseline where minimal parameter count is important. Choose a GRU when you need gated memory with a simpler and often faster recurrent core. Choose an LSTM when explicit separation between persistent cell memory and exposed hidden state is useful, or when validation experiments show a benefit on long dependencies.

The vanilla RNN has the fewest parameters and the simplest implementation, but its recurrent transition provides no explicit mechanism for preserving a value unchanged. It is consequently the most exposed to vanishing and exploding gradients.

A GRU has two gates and one state. It often offers a good accuracy-to-latency trade-off for forecasting, event streams, and compact sequence models. Its update gate can preserve the old state or replace it with a candidate, but memory and output are not separately represented.

An LSTM has three gates and two state vectors. It supplies finer control over writing, forgetting, and exposing memory, at the cost of more matrix multiplications and state. This can matter when a task contains multiple time scales or requires deliberate retention.

Do not decide from architecture reputation alone. Match parameter counts, train with the same data and stopping criteria, then evaluate quality, throughput, peak memory, and stability. Gate bias initialization, normalization, sequence chunking, and state-reset policy can affect results as much as the cell type.

Can an RNN be trained in parallel?

An RNN cannot generally compute $h_t$ and $h_{t+1}$ in parallel because $h_{t+1}$ depends on $h_t$. It can parallelize across batch items, feature dimensions, layers in limited schedules, and independent sequences. Fused recurrent kernels reduce overhead but do not remove the sequential dependency across time.

This distinction explains why RNN training often scales differently from Transformer training. A Transformer can calculate representations for all positions in a training sequence simultaneously because attention operates on the whole input matrix, even though the attention operation itself has substantial cost.

RNNs can still achieve high throughput when sequences are short, batches are large, or specialized hardware provides optimized recurrent kernels. Sequence packing can avoid wasted work on padding, and truncated BPTT bounds activation memory.

Architectures such as temporal convolutions and state-space models address the same sequential modeling problem with more parallel-friendly computation. The right choice depends on sequence length, batch size, hardware, and whether online stateful inference is required. “Sequential data” does not imply that a sequential implementation is optimal.

Do LSTMs eliminate vanishing gradients?

LSTMs reduce, but do not eliminate, vanishing and exploding gradients. Their additive cell-state update creates a path whose derivative includes the forget gate, allowing gradients to remain large when the gate stays near one. Saturated gates, poor initialization, long nonlinear computations, and exploding recurrent activations can still cause optimization problems.

For the cell update

$$c_t = f_t\odot c_{t-1}+i_t\odot\tilde{c}_t$$

the direct derivative with respect to the previous cell state is approximately $f_t$. If $f_t$ is consistently near one, the cell can carry information and gradient over many steps. If it is near zero, the model intentionally forgets.

This is a learned compromise, not a guarantee. The forget gate may close too often, the candidate update may saturate, or the output projection may still create unstable gradients. LSTMs commonly use gradient clipping, suitable learning rates, and careful sequence batching.

GRUs have a comparable interpolation path through their update gate. Neither gated architecture provides arbitrary retrieval of the full past. For contexts requiring exact access to distant events, attention, external memory, or a structured state-space model may be more appropriate.

Why do RNNs use teacher forcing?

Teacher forcing feeds the correct previous target to an autoregressive RNN during training instead of feeding the model’s own previous prediction. It makes the next-step prediction condition closer to the training data and often accelerates optimization, but it creates a train-inference mismatch.

For a language model, training may condition on the true prefix:

$$p_\theta(y_t\mid y_{<t})$$

During generation, the previous token is a model-selected output, so one error can alter all later inputs. The model is then operating on states it did not frequently see during teacher-forced training. This is called exposure bias.

Alternatives include scheduled sampling, where the training process gradually replaces true previous tokens with model outputs, and sequence-level objectives. Scheduled sampling can introduce its own optimization issues and is not a universal fix.

Teacher forcing is still common because token-level maximum likelihood is stable, efficient, and easy to batch. When deploying an RNN generator, evaluate it in the actual autoregressive mode, not only with teacher-forced validation loss. For classification and many forecasting setups, the distinction may be absent or less severe because the model does not feed discrete predictions back as inputs.

Conclusion

An RNN’s essential idea is compact stateful computation: apply one learned transition repeatedly, let $h_t$ summarize the prefix, and generate outputs from that state. Parameter sharing handles variable-length sequences, while the state provides a constant-size interface between past and present.

The same mechanism creates the central limitation. Learning a dependency across hundreds of recurrent transitions requires gradients to survive a product of Jacobians, and vanilla $\tanh$ updates offer no explicit way to preserve information. LSTMs and GRUs improve the situation by learning when to retain and replace state, but they remain sequential in time and bounded in memory.

The most actionable next step is to build a matched baseline on your own sequence task: compare a vanilla RNN, GRU, and LSTM under the same hidden width, training horizon, state-reset policy, and hardware measurements. Then test whether the task benefits from causal compact state or from direct long-range interaction, which is the decision boundary between recurrent models and alternatives such as attention mechanisms and temporal convolutional networks.

Two adjacent topics deserve attention next: backpropagation through time for understanding recurrent optimization, and modern state-space sequence models for architectures that seek recurrent-style inference with more parallel-friendly training.