Ch 0. The guessing game

In 1951, Claude Shannon played this game with people, one letter at a time, to measure how predictable English is. Very, it turned out.^[1]

In 1951, Claude Shannon ran an experiment that looks a lot like this one. He showed people the start of a passage and asked them to guess the next letter, again and again, and used how well they did to measure how predictable written English is. The answer: very.^[1][1]

The model above, GPT-2 small, plays the same game. It doesn’t commit to one answer: it gives every possible next word a chance, and spreads its bets across floor, bed, couch, and many more.

A large language model plays the same game. The model above is GPT-2 small, which OpenAI released in 2019.^[2][2][3][3] It read the words The cat sat on the and gave every possible continuation a chance. Notice that it doesn’t commit to one answer. It spreads its bets: floor, bed, and couch are all likely, and no single option gets even one chance in ten.

Chances, written down

The game has an exact form. Write a text as tokens $x_1,x_2,\dots ,x_n$: the chunks GPT-2 really reads, which Ch 4 explains. For each place $i$, the model gives the chance of token $x_i$ given every token before it, written $P(x_i|x_1,\dots ,x_{i-1})$. The chance of a whole text is those chances multiplied together:

where $\prod$ means multiplying, for every place $i$ from 1 to $n$.^[2] The product shrinks fast. After The cat sat on the, floor gets 7.6%, and a comma after floor gets 26%, so the model’s favorite two-token continuation has a chance of only about 2%.

Spreading its bets is normal. Here the 50 likeliest next tokens add up to only about 64%; the other 50,207 share the rest. Shannon measured that spread as entropy, in bits:

where $p_i$ is the chance of outcome $i$. One bit is one fair yes-or-no question; a choice between eight equally likely outcomes is 3 bits.^[4] From his guessing experiments, Shannon estimated that printed English carries about one bit per letter, given the 100 letters before it.^[1]