Ch 8. The context problem

Now “it” is the suitcase. To work that out, you used “trophy”, “suitcase”, “fit”, and the very last word, all at once, without even trying.

Now “it” is the suitcase. You didn’t have to think about it, but to get there you used “trophy”, “suitcase”, “fit”, and the very last word, all at once. Sentences like this were collected into a test for computers, the Winograd Schema Challenge, precisely because they need this kind of whole-sentence understanding.^[1][1]

The obvious fix is a bigger window. But a counting model needs a place in its table for every combination of words, and with longer windows almost all of them never appear in any text.^[1] A neural network shares what it learns between similar words, but a window of fixed size still has a hard edge: anything just outside it is invisible.^[2]

What we want is a model that can look at anything earlier in the text, and decide for itself which parts matter.

The obvious fix is a bigger window. For the counting model, each extra word makes the table vastly bigger. GPT-2’s vocabulary has 50,257 tokens, so there are about 2.5 billion possible pairs of tokens and about 127 trillion possible triples. Most of them never appear in any training text, so a table of counts would be almost entirely zeros.^[2][2]

A neural network like the one in Ch 7 handles that better, because similar words get similar vectors and can share what they learn.^[3][3] But a fixed window still has two problems:

• It has a hard edge. Anything one word outside the window is invisible, however important.
• Every slot is fixed. The network learns “the word 3 back” and “the word 4 back” as separate inputs, while what matters is which word is relevant, wherever it is.

What we want is a model that can look at anything earlier in the text, and decide for itself which parts matter for the word it’s working on.

The fixed window, written down

A window model in the style of Bengio and colleagues feeds the vectors of the last $n$ tokens into its first layer side by side:

where $e_{t-k}$ is the vector of the token $k$ places back, the brackets mean “put the lists end to end”, and each slot gets its own columns of $W$.^[3] So a pattern learned for the word 3 back says nothing about the word 4 back, and a wider window needs more weights. The tiny model from Ch 7 is this design: 3 characters of 8 numbers each make 24 inputs to 64 neurons, and every extra character would add another 8 × 64 = 512 weights.

Modern models still have a limit, called the context window, but it’s much larger. GPT-2 can look at 1,024 tokens at once, and newer models such as Llama 3 at more than 100 times as many.^[3][4]

Modern models still have a limit, called the context window, but it is much larger. GPT-2 can look at up to 1,024 tokens at once.^[4][4] Llama 3, from 2024, can look at up to 128K (about 128,000), more than 100 times as many.^[5][5] Within that window, every token can use any earlier token.

A bigger window isn’t free. Attention compares every token with every earlier one, so its work grows with the square of the window’s length (see Ch 9):

so a full Llama 3 window needs about 15,600 times as many attention scores per head as a full GPT-2 window.

Prices don’t always follow that work. On Anthropic’s price list, Claude Haiku 5.5 charges five times as much per token, input and output, once a prompt is longer than 100,000 tokens, while its other recent models charge one rate across a million-token window.^[6]