Ch 14. Guessing ahead

A model writes one token per pass, and each pass reads every one of its weights from memory, which takes most of the time.^[1] But one pass can check many tokens at once: it predicts what comes next after every position, and reading a few more tokens costs a big model about the same as reading one.

A model writes one token per pass through all its blocks (Ch 11), and each pass has to read every weight from memory. For a big model, most of the time goes into that reading, not the arithmetic.^[1][1] So a big model writing 30 tokens reads all its weights 30 times, even for the easy ones, like the next number in “one, two, three”.

Here is the trick that helps. In one pass, a transformer doesn’t only predict the token after the last one: it predicts the next token after every position at once. That is how it reads a whole prompt in one go. Reading a few more tokens costs a big model about as much as reading one, because the weights are read once either way.

So let a small, fast model guess the next few tokens, and let the big model check them all in one pass. It keeps the guesses up to the first one it disagrees with, and puts its own token there. This is speculative decoding.^[1][2] Below, GPT-2 small guesses four tokens at a time for GPT-2 medium:

So let a small, fast model guess. It writes the next few tokens on its own, cheaply. Then the big model reads the text plus all the guesses in one pass, and at each position it compares its own top token with the guess. It keeps the guesses up to the first one it disagrees with, swaps that one for its own token, and drops the rest. If every guess matches, it adds one more token of its own. This is speculative decoding.^[1][1][2][2] Below, GPT-2 small (124 million weights) guesses four tokens at a time for GPT-2 medium (355 million). Both use the same tokens, which the trick needs:

Every pass still adds at least one token. For “The cat sat on the”, GPT-2 medium wrote 30 tokens in 7 passes instead of 30. And the text is exactly what the big model would have written alone, because it checks every token.

Every pass of the big model adds at least one token, its own, so it is never slower in passes than writing alone. When the guesses are good it adds up to five. For “The cat sat on the”, GPT-2 medium wrote 30 tokens in 7 passes instead of 30. Counting (“six, seven, eight, …”) is easy for the small model too: 7 passes. “The capital of France is” took 9, because two of the small model’s first guesses were wrong.

And the text is the same, token for token, as GPT-2 medium writing alone. The big model checks every token and only keeps a guess that is exactly what it would have picked. The drafter only changes how many passes it takes. When the model samples instead of always picking the top token (Ch 2), a slightly different rule accepts or replaces each guess at random so that every token still comes out with exactly the big model’s probabilities.^[1][1][2][2]

The rules, written down

When sampling, the big model’s probabilities are $p$ and the drafter’s are $q$. A guessed token $x$ is kept with chance

and if it’s rejected, the replacement is drawn from $\max (0,p-q)$, rescaled to add up to 1. Tokens the drafter overrates are sometimes thrown back, tokens it underrates are made up for in the replacement, and every token comes out with exactly the chance $p$ gives it.^[1][2]

How much this saves depends on how often guesses are kept. If each guess is kept with chance $\alpha$, independently, and the drafter guesses $k$ tokens, a pass of the big model adds on average

tokens, between 1 and $k+1$.^[1] With $\alpha =0.8$ and $k=4$, that’s about 3.4. The cat run above averaged about 4.3 of a possible 5.

On real hardware this makes big models write about two to three times faster, with identical text.^[1][2] Some model families ship their own small guessers, like Google’s Gemma 4 models with -assistant in the name.^[3]

On real hardware this makes big models write about two to three times faster with identical output.^[1][1][2][2] Some model families ship their own small drafters. Google’s Gemma 4 models have matching drafters named with -assistant, from 78 million to half a billion weights, and the model card reports up to three times faster writing.^[3][3] They guess several tokens ahead for their big model, which is why the card calls them multi-token prediction drafters.