Ch 38. Remembering without attention

Attention lets every token look back at every earlier token (Ch 9). A recurrent model reads one token at a time and keeps a small memory that it updates as it goes. To predict the next token it has only that memory. It’s like reading a book while keeping notes on a single index card.

Attention lets every token look back at every earlier token (Ch 9). A recurrent model works differently: it reads one token at a time and keeps a small, fixed-size memory, a vector it updates after every token. To predict the next token it has only that memory, never the earlier tokens themselves. It’s like reading a book while keeping notes on a single index card.

The update, written down

The tiny model’s whole memory update is one line. With $h_{t-1}$ the 32-number memory, $x_t$ the character just read, and learned weights $W_h$, $W_x$, $W_y$ and biases $b$, $c$:

This is the design of Elman’s 1990 networks.^[1] The tanh keeps every memory number between −1 and 1, so the memory can’t grow without limit however long the text. Training is the backpropagation from Ch 7, run backwards through the characters one step at a time.

The appeal is cost. A transformer keeps something for every token it has read (Ch 26), so its memory grows with the text. A recurrent model’s memory stays the same size:

The appeal is cost. A transformer’s memory grows with the text, because it keeps something for every token so later tokens can look back: the KV cache from Ch 26. A recurrent model’s memory stays the same size however long the text gets:

The catch is the index card: whatever the model doesn’t note down is gone for good. Early recurrent networks forgot quickly; the LSTM added gates that learn what to keep.^[2] LSTMs were the standard for language until the transformer replaced them.^[3] It trained much faster: a recurrent model reads even its training text one token at a time, while a transformer takes in every position at once.^[3]

The catch is the index card. Everything the model will ever need must be squeezed into those few numbers as it reads, and whatever it doesn’t note down is gone for good. Early recurrent networks were bad at keeping information for long; the LSTM added gates that learn what to keep and what to forget.^[2][2] For years, LSTMs were the standard way to model language, until the transformer, which needs no recurrence at all, replaced them.^[3][3]

The main reason was training speed. A recurrent model can’t work out its memory at step 10 before step 9, so even in training, where the whole text is known in advance, it reads one token at a time. A transformer handles every position of a training text at once, which suits GPUs. (Writing a reply still goes one token at a time; it’s training that runs in parallel.) The first transformer set a new record for a single model on English-to-French translation after 3.5 days on eight GPUs, at a small fraction of the training cost of the best earlier models.^[3][3]

Why plain recurrent networks forget

To learn from a mistake 20 characters later, the error has to travel back through 20 copies of that update, multiplied each time by $W_h$ and by the slope of tanh, which is never more than 1. Over many steps it usually shrinks to almost nothing, or now and then blows up. The LSTM’s fix was a separate cell carried forward by adding to it rather than squashing it, which its authors called a constant error carousel, with gates that learn when to write to it and when to read from it.^[2]

Attention avoids the long chain altogether. For a text of $n$ tokens, a recurrent layer needs $n$ steps one after another, and information from a token $n$ places back passes through $n$ updates to arrive; a self-attention layer needs 1 step, and any earlier token is 1 step away. The price is comparing every pair of tokens, $n^2$ comparisons instead of $n$ updates.^[3]

The size difference is large. At its full 1,024 tokens, GPT-2 small keeps 18,432 numbers per token, 18,874,368 in all; the tiny model here keeps 32.

Recurrent ideas are back in newer designs like Mamba, which learns what to keep as it reads, and whose cost grows only in step with the length of the text.^[4] But a fixed memory still struggles to copy long passages word for word, which a transformer does easily.^[5]

Recurrent ideas are back in newer designs called state-space models. Mamba, from 2023, updates its fixed-size state in a way that depends on the token it is reading, so it can choose what to keep and what to forget, and its cost grows only in step with the length of the text.^[4][4] The fixed memory still has a price: a transformer can copy a long passage it was shown word for word, while a model with a fixed-size state struggles once the passage outgrows its memory.^[5][5]