Ch 11. Generating text

GPT-2 only does one thing: given some text, it gives a chance to every possible next token. Writing a paragraph is a loop around that: run the model, pick a token, add it to the end, and repeat. Every chat app writes this way. Try it: each press runs the model once. Click any word in the list to pick it instead of the model’s favorite.

GPT-2 only ever does one thing: given some text, it gives a probability for every possible next token. That’s the whole stack from Part IV, run once. Writing a paragraph is just a loop around it:

1. run the model on the text so far,
2. pick a token from its probabilities,
3. add that token to the end of the text,
4. repeat.

Every model in a chat app writes this way. Try it yourself: each press runs the full model once. Click any word in the list to pick it instead of the model’s favorite.

Keep pressing Add the most likely and GPT-2 writes “The cat sat on the floor, and the cat was still asleep.” Go back and pick ” bed” instead, and it gets stuck saying the same thing over and over. Picking at random, weighted by the chances, mostly avoids loops like this.^[1]

Keep pressing Add the most likely and GPT-2 writes “The cat sat on the floor, and the cat was still asleep.” Go back and pick ” bed” instead, and it gets stuck: ”, and the cat sat on the bed, and the cat sat on the bed…“. Always taking the single most likely token often falls into loops like this. Picking at random, weighted by the probabilities (the temperature spinner from Ch 2), mostly avoids them, which is one reason most text generators sample instead.^[1][1]

Greedy isn’t the likeliest text

Written out, greedy decoding takes the top token at each step, and sampling draws one from the tempered probabilities $P_T$:

where $x_1,\dots ,x_t$ is the text so far and $\sim$ means “drawn at random from”. The best token at each step doesn’t add up to the best text, because a text’s chance is the product of its tokens’ chances. After The cat sat on the, greedy takes ” floor” (7.6%), and then a comma (26%): about 2% for the pair. ” edge” starts lower (4.8%), but ” of” follows it 96% of the time, so ” edge of” has about 4.6%, more than twice as likely. Beam search keeps several partial texts at each step to find likelier ones, but the likeliest texts are often the dullest and most repetitive, which is part of why open-ended writing samples instead.^[1]

Apps that run a model can also make a token less likely each time it has been used, a frequency penalty. Here is GPT-2 continuing “The cat sat on the bed”, always taking the most likely token, with no penalty, a small one, and a big one:

Most model APIs also offer a setting aimed straight at loops. A frequency penalty lowers the score of every token by a set amount for each time it has already appeared in the reply, so the more a token has been used, the less likely it gets. A presence penalty is the flat version: one fixed cut for any token already used, however many times.^[2][2] Here is GPT-2 continuing “The cat sat on the bed”, always taking the most likely token after the penalty:

With no penalty, the 32 tokens use only 7 different ones. A small penalty breaks the loop. A big one stops GPT-2 from using any token twice, even ” the”, so the cat sits on “her back”.

With no penalty, the 32 tokens use only 7 different ones. At 0.5, “the cat sat on the bed” comes round once more, then the loop breaks and 25 of the 32 are different. At 5, no token appears twice. Even ” the” is ruled out, so the cat sits on “her back”. A penalty works on tokens, not ideas, so a strong one also pushes out the small words that every sentence needs.

Written down, with $z_i$ token $i$’s score and $c_i$ the number of times it has appeared in the reply so far, the recordings above use

with ${\alpha }_f$ the frequency penalty, and a presence penalty ${\alpha }_p$ subtracts ${\alpha }_p$ once from every token with $c_i>0$. The softmax and the pick follow as usual. How big a penalty needs to be depends on how spread out the model’s scores are, so 0.5 for GPT-2 isn’t the same as 0.5 for another model.

Each token is chosen once, and everything after it builds on it. The model never goes back to fix a word. And every new token means one more full run of the model. Writing is the slow part, and it costs more too: on Anthropic’s price list, a token the model writes costs five times as much as a token it reads.^[2]

That doesn’t mean it can’t look ahead. When researchers traced a large model writing a rhyming poem, it had already picked likely words to end the next line on before it wrote that line.^[3]

Each token is chosen once, and everything after it is built on top. The model doesn’t write a draft or go back and fix an earlier word. If it picks ” bed”, it has to make the best of ” bed”. And each new token costs a full run through all 124 million of GPT-2’s weights, so a 500-token answer means 500 runs. The prompt’s tokens can all go through the model together in one run, but the reply comes out one run per token.^[3][3] Prices fit that: on Anthropic’s price list, every model charges five times as much for a token of the reply as for a token of the prompt.^[4][4]

Choosing one token at a time doesn’t mean thinking only one token ahead, though. Researchers at Anthropic traced Claude 3.5 Haiku writing a rhyming couplet whose first line ended “grab it”. Before it wrote the first word of the second line, it already held two candidate endings, rabbit and habit. When they switched off rabbit, the model wrote a different line, ending “a powerful habit”.^[5][5] The plan lives in the model’s numbers; the text still comes out one token at a time.

A run costs about two operations per weight, a multiply and an add, so GPT-2 small needs about 250 million operations per token, and about 124 billion for those 500 tokens.^[6] Rerunning the model on the whole text for every token would cost far more, since the text grows by one token each time. Real systems keep each earlier token’s keys and values from attention and only run the newest token through the model; Ch 26 shows how.^[3]

The loop has to stop somehow. It stops when the model picks a special “end of text” token, or when it reaches a length limit it was given. The model never sees that limit, so it doesn’t write a shorter answer to fit: the reply just stops, even mid-sentence. To get a short answer, ask for one.

The loop has to end somehow. GPT-2’s vocabulary includes a special token, <|endoftext|> (number 50,256), that means “end of text”. When the model picks it, the program stops.^[6][7] Otherwise, the program stops at a length limit it was given. Chat models work the same way with their own end-of-turn token.

The length limit is the program’s, not the model’s. The model never sees it, so it doesn’t plan a shorter answer: if the limit comes first, the reply is cut off, even mid-sentence, and the API reports that it stopped for length. To get a short answer, ask for one in the prompt (Ch 20). The limit counts only the reply, including a reasoning model’s thinking, and the prompt and reply together must still fit in the context window.^[2][2]