Recurrent Networks, LSTMs, and the Memory Problem

Hook #

CNNs (last course) conquered images — data with spatial structure. But a huge fraction of the world's data is sequential: text (a sequence of words), speech (a sequence of sounds), time series (stock prices, sensor readings), DNA, music. Sequences have a property images don't: order matters and context accumulates — "the dog bit the man" means something different from "the man bit the dog," and to understand a word you often need to remember words from far earlier ("The cat, which had been missing for days, finally came home — it was hungry" — to resolve "it," you must remember "cat"). A plain feedforward network (the MLP) can't handle this — it processes a fixed-size input all at once, with no notion of "before" or "memory." Recurrent Neural Networks (RNNs) were the answer: they process a sequence one element at a time, carrying a hidden state forward that acts as a memory of what came before — so each step's output depends on the current input and the accumulated history. This lesson covers RNNs and their central struggle: the memory problem. Plain RNNs forget — they can't hold onto information across long gaps (the vanishing-gradient problem from the backprop lesson, now unfolding over time), so they lose track of long-range dependencies. The fix was the LSTM (Long Short-Term Memory) and its cousin the GRU — recurrent networks with gates and a cell state that let them deliberately remember and forget, holding information across long ranges. For years, LSTMs were the state of the art for sequences — until the transformer (a later lesson) dethroned them. Understanding RNNs and their memory problem is understanding why the transformer exists, and it's the foundation of all sequence modeling.

What you'll be able to do by the end of this lesson #

• Explain a Recurrent Neural Network (RNN) — a network that processes a sequence one element at a time, maintaining a hidden state (a memory) that carries information forward, so each output depends on the current input and the accumulated past.
• Explain backpropagation through time (BPTT) — training an RNN by "unrolling" it across the sequence and backpropagating gradients through every time step — and why this makes RNNs hard to train on long sequences.
• Explain the long-range dependency problem — plain RNNs forget because gradients vanish (or explode) over time (the vanishing-gradient problem from the backprop lesson, unfolding across many time steps), so they can't connect information across long gaps.
• Explain LSTMs (and GRUs) — recurrent networks with gates (learnable "valves" that control what to remember, forget, and output) and a cell state (a protected memory "conveyor belt") that let them hold information over long ranges — the fix that made sequence learning work.

A quick try before we start #

Read this sentence one word at a time, and notice what your brain does: "The keys that I thought I had left on the kitchen counter this morning turned out to be … in my pocket." To understand "be" and what's missing, you had to remember "keys" from the very start of the sentence — across a dozen intervening words. That's a long-range dependency, and it's the essence of sequence understanding: context from far back matters. Now, how would a machine do this? A plain feedforward network sees a fixed input all at once — it has no way to process "a word at a time" or to remember earlier words. An RNN solves this by processing the sequence step by step, carrying a hidden state forward: it reads "The," updates its hidden state; reads "keys," updates it (now the state "knows" about keys); reads "that," updates again — the hidden state is a running memory that accumulates context, and at each step the network combines the current word with its memory of everything before. Beautiful in principle. But here's the problem, which you can feel: by the time the RNN reaches "be," a dozen words later, has it held onto "keys"? In a plain RNN, often no — the memory of "keys" has faded, drowned out by all the intervening words, because the signal has to survive being repeatedly transformed at every step (and gradients, flowing back through all those steps during training, vanish — the backprop lesson's problem, now stretched across time). The RNN forgets. The LSTM fixes this with an explicit memory (the cell state) and gates that decide "this is important — keep it" (remember "keys") and "this is noise — drop it" — so it can deliberately carry "keys" across the whole sentence. Hold the tension: RNNs give a network memory, but plain RNNs forget over long ranges — and LSTMs' gates are how they learned to remember.

Why this matters here #

This lesson matters because sequence modeling is one of the two great pillars of deep learning (alongside vision), and RNNs/LSTMs are its foundation — the architectures that first made machines process language, speech, and time series, and the necessary background for understanding why the transformer (which replaced them) was such a leap. Sequential data is everywhere and enormously important: all of natural language (the basis of NLP and the LLMs you use), speech recognition and synthesis, machine translation, time-series forecasting (finance, weather, demand), music and audio, and biological sequences (DNA, proteins). Any task where order matters and context accumulates is a sequence task, and for years RNNs and LSTMs were the tools for all of them — they powered the first good machine translation, the first practical speech recognition, and the text-generation demos that hinted at what was coming. Understanding them is understanding the lineage of modern language AI.

The deeper reasons this lesson matters are two, both of which set up the rest of the course. First, the memory problem is the central drama of sequence modeling, and understanding it is essential for understanding the transformer. A plain RNN's fatal flaw — that it forgets over long ranges — is a direct manifestation of the vanishing-gradient problem from the backprop lesson, now unfolding across time rather than across layers: an RNN unrolled over a 50-word sentence is effectively a 50-layer-deep network (backpropagation through time), so gradients flowing back to the early words must survive 50 multiplications and vanish, meaning the RNN can't learn to connect a word at the end to a word at the start. This is why long-range dependencies were so hard, and why the whole history of sequence models is a story of fighting the memory problem. The LSTM's solution — a protected cell state (a memory that flows forward with minimal transformation, so information and gradients survive) plus gates (learnable controls that decide what to write to, keep in, and read from that memory) — was ingenious and worked, holding information across much longer ranges and dominating sequence tasks for years. But even LSTMs had limits (they still process sequentially, still struggle with very long ranges, and — crucially — can't be parallelized because each step depends on the previous). Understanding both the RNN's memory problem and the LSTM's partial fix is exactly what makes the transformer's radical solution (next lessons) comprehensible: the transformer throws out recurrence entirely and lets every position directly attend to every other position — solving the long-range problem and the parallelization problem at once, which is why it dethroned LSTMs. Second, for a working engineer, this lesson provides practical judgment about sequence models: RNNs/LSTMs are still genuinely useful for many tasks (they're lighter than transformers, work well on shorter sequences and smaller data, and are still common in time-series and on-device/streaming applications), so knowing them isn't just history — it's a live tool. And the conceptual takeaways — hidden state as memory, the challenge of long-range dependencies, gating as a mechanism for selective memory — recur throughout deep learning. This lesson is the foundation of sequence modeling and the "before" that makes the transformer's "after" make sense.

The engineer's lens #

The first lens is the RNN — giving a network memory via a hidden state, and the cost of backpropagation through time. A plain feedforward network (the MLP) has a fatal limitation for sequences: it takes a fixed-size input and processes it all at once, with no notion of order, no memory, no way to handle variable-length sequences. The RNN solves this with a simple, powerful idea: process the sequence one element at a time, and carry a **hidden state* forward from step to step.* At each time step, the RNN takes two inputs — the current element (this word) and its hidden state from the previous step (its memory of everything so far) — and produces an output and an updated hidden state (to pass to the next step). The hidden state is thus a running memory: it accumulates context as the sequence unfolds, so the network's response to each element depends on the entire history before it. The "recurrent" name comes from this loop — the same network is applied repeatedly, feeding its output back as input. This elegantly handles variable-length sequences (just keep looping) and order (the hidden state encodes what came before). Training an RNN uses backpropagation through time (BPTT): you "unroll" the RNN across the sequence (picture the loop stretched out into a chain of copies, one per time step), then backpropagate gradients through the whole unrolled chain — which is just the backpropagation from the DL-fundamentals course, applied to this unrolled network. And here is the seed of the memory problem: an RNN unrolled over a long sequence is a very deep network (a 100-element sequence unrolls to a 100-step-deep chain), so BPTT must propagate gradients back through many steps — and that's where the trouble starts. For the engineer, the RNN's core idea (hidden state = memory carried forward, applied step by step) is the foundation of all sequence modeling, and understanding that training it means backpropagating through an unrolled, deep-in-time network is what makes its central weakness — the memory problem — comprehensible.

The second lens is the long-range dependency problem — why plain RNNs forget, the vanishing gradient over time. The RNN's beautiful design has a crippling flaw in practice: plain RNNs can't remember over long ranges — they forget. The reason is exactly the vanishing-gradient problem from the backprop lesson, now unfolding over time instead of over layers. Because backpropagation through time propagates gradients back through every time step, and each step multiplies the gradient by a factor (the chain rule), the gradient reaching the early time steps is a product of many factors — and if those factors are consistently less than 1, the product shrinks toward zero (vanishes) over the many steps. The consequence: the RNN can't learn to connect information across long gaps, because the gradient signal from a late output never reaches the early inputs strongly enough to teach the network "remember that word from 50 steps ago." Concretely, a plain RNN reading "The keys … turned out to be …" loses "keys" by the time it reaches "be" — the memory has faded and the gradient can't teach it to persist. (The mirror problem, exploding gradients, also occurs and makes training unstable — often addressed by gradient clipping, from the calculus course.) This long-range dependency problem is the central obstacle of sequence modeling: sequences constantly have dependencies that span long distances (a pronoun and its referent, a subject and its verb, the beginning and end of a paragraph), and a model that forgets over distance fundamentally can't understand them. For the engineer, recognizing that the RNN's forgetting is the vanishing gradient in the time dimension is the key insight — it connects the sequence-modeling struggle directly to the deep-network training problem you already understand, and it explains why the LSTM (which fights vanishing gradients over time) and eventually the transformer (which eliminates the sequential depth entirely) were necessary. The memory problem is why naive RNNs, despite their elegance, weren't enough.

The third lens is the LSTM and GRU — gates and cell state, deliberate memory that fixed the long-range problem. The LSTM (Long Short-Term Memory) is the ingenious fix that made sequence learning actually work for long ranges, and its design is worth understanding conceptually (Olah's diagrams are the canonical explanation). The core idea is a protected memory pathway — the cell state — that runs through the entire sequence like a conveyor belt, carrying information forward with minimal transformation (so information — and gradients — can flow across many steps without vanishing, because the cell state isn't repeatedly multiplied and squashed the way a plain RNN's hidden state is). Alongside this memory belt, the LSTM has gates — small learnable neural networks that act as valves controlling the flow of information, and there are three: a forget gate (decides what to remove from the cell state — "this old information is no longer relevant, drop it"), an input gate (decides what new information to write to the cell state — "this is important, remember it"), and an output gate (decides what to read from the cell state to produce the current output). Because these gates are learned, the LSTM learns what to remember, what to forget, and when — so it can deliberately carry "keys" across a long sentence (the input gate writes it, the forget gate keeps it, the output gate reads it when "be" arrives). This selective, gated memory is what lets LSTMs handle long-range dependencies that plain RNNs lose, and it dominated sequence tasks (translation, speech, text) for years. The GRU (Gated Recurrent Unit) is a simpler, faster variant that merges some of the gates (fewer parameters, often comparable performance) — a common practical choice when you want gating without the LSTM's full complexity. For the engineer, the LSTM's cell-state-plus-gates design is the concept to hold: a protected memory pathway that resists vanishing, plus learned gates that control remember/forget/output — an elegant, effective solution to the memory problem. But it's worth knowing the limits that motivate the transformer: LSTMs still process sequentially (step by step, so they can't be parallelized — each step waits for the previous, which makes training on huge data slow), and they still struggle with very long-range dependencies (gates help, but information still degrades over enough distance). The synthesis of the lesson: sequences need memory, so **RNNs* carry a hidden state forward step-by-step (trained by backpropagation through time, which unrolls the network deep-in-time); but plain RNNs suffer the long-range dependency problem — they forget, because gradients vanish over time (the backprop lesson's problem in the time dimension); the LSTM (and simpler GRU) fixed this with a protected cell state (a memory conveyor belt that resists vanishing) and learned gates (forget/input/output — deliberate remember/forget/read), dominating sequences for years — but still processing sequentially and struggling with very long ranges, which is exactly what the transformer will solve.* Understanding the memory problem and the LSTM's fix is understanding why the transformer had to be invented.

What to focus on in the resources #

• Karpathy — 'Unreasonable Effectiveness of RNNs' (free) — primary. The classic essay: a character-level RNN generating Shakespeare and code, building the intuition for what a recurrent network does (carry hidden state, predict the next token). The best first exposure — then his makemore videos build RNNs from scratch.
• Olah — 'Understanding LSTM Networks' (free) — primary. The definitive LSTM explanation; the gate-and-cell-state diagrams are the clearest anywhere. Read it to understand how gates let an LSTM remember where a plain RNN forgets — the cell state as a memory "conveyor belt."
• Géron Ch 15 (free notebooks). The practical build — RNNs, LSTMs, GRUs in code on real sequence tasks, covering BPTT and the vanishing-gradient-over-time problem. The hands-on complement.
• Skip on first pass: the precise BPTT gradient equations, the exact LSTM gate formulas, bidirectional/stacked RNNs, and attention (that's the next lesson). Get: RNN = hidden state carried forward step-by-step (memory); BPTT = unroll and backprop (deep-in-time); the long-range dependency problem = RNNs forget because gradients vanish over time (backprop lesson's problem, in time); and LSTM/GRU = protected cell state + learned gates (forget/input/output) for deliberate long-range memory — with the sequential/parallelization limits that motivate the transformer.

Explain it back #

Explain to a colleague why sequences need a special kind of network, why plain RNNs "forget," and how LSTMs fix it. A strong answer: sequential data (text, speech, time series) has the property that order matters and context accumulates — to understand a word you often need to remember words from far earlier (a pronoun and its referent). A plain feedforward network can't handle this (fixed input, no memory), so an RNN processes the sequence one element at a time, carrying a hidden state forward that acts as a running memory — each step combines the current input with its memory of everything before. It's trained by backpropagation through time (BPTT): unroll the recurrent loop across the sequence into a deep chain, then backprop through all the steps. But plain RNNs forget over long ranges — the long-range dependency problem — and the reason is exactly the vanishing-gradient problem from the backprop lesson, now over time: because BPTT multiplies gradients at every step, the gradient reaching early time steps is a product of many factors that shrinks toward zero, so the network can't learn to connect a late word to an early one (it loses "keys" by the time it reaches "be" 12 words later). The LSTM fixes this with a protected cell state — a memory "conveyor belt" that carries information forward with minimal transformation (so it and its gradients resist vanishing) — plus learned gates: a forget gate (drop irrelevant memory), an input gate (write important new info), and an output gate (read the memory for the current output). Because the gates are learned, the LSTM learns what to remember and forget, so it can deliberately hold information across long ranges. The GRU is a simpler, faster variant. LSTMs dominated sequences for years — but they still process sequentially (can't be parallelized) and still struggle with very long ranges, which is exactly what the transformer (next lessons) solves by eliminating recurrence and letting every position attend directly to every other.

Where this connects #

Backward: This is the sequence counterpart to the CNN (last course) — where CNNs exploited spatial structure, RNNs exploit sequential structure (a different inductive bias — from the ViT lesson — matched to a different data shape). The RNN's memory problem is precisely the vanishing-gradient problem from the backpropagation lesson, now unfolding over time (an unrolled RNN is a deep-in-time network) — and gradient clipping (from the calculus course) addresses its exploding counterpart. The LSTM's cell state is another skip-connection-like pathway for gradient flow, echoing ResNet's residual connections from the DL-fundamentals course.

Forward: This lesson sets up the entire rest of the course. The RNN/LSTM's limits (sequential processing, long-range struggle) are exactly what the next lesson's attention mechanism begins to fix (let the model look back at any part of the input directly) and what the transformer (the lesson after) fully solves (attention replaces recurrence, enabling parallelism and unlimited-range dependencies). The hidden-state-as-memory and gating ideas recur throughout. And sequence modeling is the foundation of the NLP and LLM courses ahead — the language models you use are the transformer's answer to the very memory problem this lesson introduced. RNNs and their memory struggle are why the transformer exists.