Ch 2. Probability and sampling

The simplest way to choose is to always take the most likely word. That gives the same text every time, and with big models it tends to be dull and to repeat itself.^[1]

Temperature controls how adventurous the pick is. A low temperature makes the likely words even likelier. A high one evens out the chances, so unusual words get picked more often. At 0, the model always takes the top word.

The simplest choice is to always take the most likely word. That’s called greedy decoding, and it gives the same output every time. With large models, always picking the top word tends to produce dull text that can get stuck repeating itself.^[1][1]

Temperature controls how adventurous the pick is. Below 1, it makes the likely words even likelier. Above 1, it evens the chances out, so unusual words get picked more often. At 0 it becomes greedy: always the top word.

Under the hood, the model gives every word a raw score, and the scores are divided by the temperature before they become probabilities. Try it:

Temperature, written down

Each candidate token $i$ gets a raw score $z_i$, called a logit. Softmax turns the scores into probabilities after dividing each one by the temperature $T$:

where $e$ is the constant 2.718… and the sum runs over all 50,257 tokens, so the $p_i$ add up to 1. For any two tokens, $p_i/p_j=e^{(z_i-z_j)/T}$: a gap in scores counts for more at a low $T$ and for less at a high one. At $T=1$ the model’s own probabilities come out unchanged. As $T$ gets close to 0, all the probability goes to the top score, which is why 0 means greedy. Adding the same number to every score changes nothing; only the gaps matter.

The recorded runs further down divide GPT-2’s raw scores by $T$ and draw from all 50,257 tokens; the widget’s cut to ten is only for the picture. Two other controls, next, cut off the long tail of unlikely tokens instead of flattening it. In the widget they apply after the temperature, top-k first, and what’s left is rescaled to add up to 100%. So the order matters: a higher temperature flattens the chances first, and top-p then needs more words to reach p. OpenAI’s documentation suggests changing the temperature or top-p, not both, since each already decides how far down the list a pick can reach.^[2]

Written down, top-p keeps the smallest set $V$ of most likely tokens with ${\sum }_{i\in V}{p}_i\ge p$, then divides each kept chance by that sum so they add up to 1.

Temperature 0 on a busy server

On a hosted model, temperature 0 isn’t always repeatable. When researchers asked Qwen3-235B for the same 1,000-token answer 1,000 times at temperature 0, they got 80 different answers, all identical for the first 102 tokens. The server groups many people’s requests into one batch, and the size of the batch changes the order in which the model’s numbers are added up. With rounding, a different order gives slightly different scores, so when two tokens are almost tied, a different one can come out on top.^[4]