Recurrent Neural Networks (RNN), LSTMs, and GRUs for Sequence Modeling

Standard feedforward neural networks and CNNs assume that all inputs are independent of one another. However, many real-world modalities — natural language sentences, time-series stock prices, DNA sequences, and audio waveforms — are inherently sequential. In language, the meaning of a word depends entirely on the preceding context: "The bank was flooded with water" vs "The bank approved the loan".

Recurrent Neural Networks (RNNs) process sequences step-by-step, maintaining an internal Hidden State ( mathbf{h}_t ) that acts as a working memory of all past tokens processed so far.

Vanilla RNN Architecture: The Recurrent Loop

At each time step ( t ), a Vanilla RNN takes the current input token vector ( mathbf{x}_t ) and the previous hidden state ( mathbf{h}_{t-1} ), computing the new hidden state ( mathbf{h}_t ):

Backpropagation Through Time (BPTT): To train an RNN, the recurrent loop is unrolled across time steps ( t=1, dots, T ). Gradients flow backward across time steps via the chain rule.

The Fatal Flaw: The Vanishing & Exploding Gradient Problem

When computing gradients across ( T ) time steps, the chain rule requires repeatedly multiplying by the recurrent weight matrix ( mathbf{W}_{hh} ):

• If the largest eigenvalue of ( mathbf{W}_{hh} < 1 ), the gradient shrinks exponentially to zero (Vanishing Gradient). The network suffers from severe amnesia; it cannot learn dependencies spanning more than 8-10 words!
• If the largest eigenvalue of ( mathbf{W}_{hh} > 1 ), gradients blow up to infinity (Exploding Gradient), causing numerical overflow and NaN parameters.

Long Short-Term Memory (LSTM) Networks (Hochreiter & Schmidhuber, 1997)

To overcome the vanishing gradient bottleneck, Sepp Hochreiter and Jürgen Schmidhuber invented the Long Short-Term Memory (LSTM) cell. The key innovation is the Cell State (( mathbf{c}_t )) — an internal highway that runs straight down the entire chain with only minor linear interactions, regulated by three non-linear Gating Mechanisms:

Gated Recurrent Unit (GRU) (Cho et al., 2014)

The GRU is a streamlined variation of the LSTM that merges the cell state and hidden state into a single vector, using only two gates: a Reset Gate (( mathbf{r}_t )) and an Update Gate (( mathbf{z}_t )). GRUs have ~25% fewer parameters than LSTMs, training faster while achieving comparable empirical performance.

import torch
import torch.nn as nn

class LSTMSequenceClassifier(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):
        super(LSTMSequenceClassifier, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        # Bidirectional LSTM: processes sequences forward and backward
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(hidden_dim * 2, output_dim)

    def forward(self, text_seq):
        embedded = self.embedding(text_seq)  # [batch_size, seq_len, embed_dim]
        # output: all time steps, (hidden_state, cell_state)
        lstm_out, (hn, cn) = self.lstm(embedded)
        
        # Concatenate final forward and backward hidden states
        final_hidden = torch.cat((hn[-2, :, :], hn[-1, :, :]), dim=1)
        logits = self.fc(final_hidden)
        return logits

# Initialize Model
model = LSTMSequenceClassifier(vocab_size=1000, embedding_dim=64, hidden_dim=128, output_dim=2)
# Sample batch: 4 sequences of 10 word token IDs
sample_batch = torch.randint(0, 1000, (4, 10))
predictions = model(sample_batch)

print(f"Batch Input Shape  : {sample_batch.shape}")
print(f"Output Logits Shape: {predictions.shape} (4 samples classified into 2 classes)")