Ch 4. Tokens: the model's alphabet

GPT-2 reads text in chunks called tokens. A token is often a whole word, and sometimes a piece of one. Each token has a number. Type below to see how GPT-2 cuts up your text.

GPT-2 doesn’t read letters one at a time, and it doesn’t read whole words either. It reads tokens: chunks of text that are often a whole common word, sometimes part of a rarer one, and sometimes a single character. Each token has an ID number. Type below to see how GPT-2 splits your text.

Common words are usually one token, and the space before a word usually belongs to it. Rarer words are cut into pieces you can often recognize, like token + ization. GPT-2 has a fixed list of tokens, and every text is made from them.

A few things to notice:

• Most common words are one token, and the space before a word is usually part of the token: ” cat” (with the space) is token 3797.
• Longer or rarer words are split into pieces you can often recognize, like token + ization.
• GPT-2 knows exactly 50,257 different tokens. Every possible text is made of them.^[1][1]

Why token + ization and not tok + enization? The pieces are found automatically, from lots of text. Start with single letters. Find the two neighbors that appear together most often, and glue them into one new piece. Repeat. This is called byte pair encoding. Step through it below on a tiny example.

Why token + ization and not tok + enization? The pieces are learned from text by a simple method called byte pair encoding (BPE), first used for language models by Sennrich and colleagues in 2016.^[2][2] Start with single characters, then repeatedly find the pair of neighbors that appears most often and glue it into a new token.

Step through it below on the tiny example from their paper. Each merge adds one token to the vocabulary, and the text needs fewer tokens to write down.

GPT-2 did the same with lots of web text. Frequent words ended up as one token; rare words are built from a few.

GPT-2 ran the same process on bytes (the numbers from the last chapter) over a large amount of web text, and kept merging until it had 50,257 tokens.^[1][1] Frequent words end up as one token; rare words are built from a few.

GPT-2’s vocabulary, counted

GPT-2’s tokens are this process written out. It starts from the 256 possible values of a byte, so any text at all, in any language or with emoji, can be written down even where no merge applies.^[1] On top of those it learned 50,000 merges, and it adds one special token, <|endoftext|> (number 50,256), that marks the end of a text:^[3]

where $|V|$ is the number of tokens in the vocabulary. Each merge picks the pair of neighboring symbols that occurs most often in the training text:

where $xy$ is any symbol $x$ followed directly by a symbol $y$, $\mathrm{count}(xy)$ is how many times that happens, and the winning pair $(a,b)$ becomes the new token $ab$. GPT-2’s first merge was ” t”: a space followed by t.

The real tokenizer has rules the toy doesn’t. GPT-2 never merges across kinds of characters, such as letters and punctuation, except that a space can join what follows it. Without that rule, common words would use up tokens on variants like dog., dog!, and dog?.^[1] And to split new text, the tokenizer doesn’t count anything: it applies the merges it learned, earliest first.^[2]

Try strawberry in the tokenizer above, with and without a space in front. With a space, as it usually appears in a sentence, it’s a single token. The model never sees the letters inside it, so when asked how many rs are in strawberry, it can’t just look and count.

Try typing strawberry in the tokenizer above, with and without a space in front.

Without a space, GPT-2 splits it into st + raw + berry. With a space, as it usually appears in a sentence, ” strawberry” is a single token, number 41236. The model never sees the letters inside it. Asked how many rs are in strawberry, the model can only answer from what it has learned about token 41236, not by looking at the letters.

Tick GPT-4 and GPT-4o in the tokenizer to compare. Newer models learned bigger lists of tokens, so the same text usually takes fewer of them. A token number only makes sense to the model it belongs to.

The difference is biggest for code. GPT-2 uses a token for nearly every space at the start of a line, so a short program can take almost twice as many GPT-2 tokens as GPT-4 tokens. Even one company’s models can count differently: Anthropic says its newer Claude models use about 30% more tokens for the same text.^[1] So a price per token only tells you so much; count your own text.

Tick GPT-4 and GPT-4o in the tokenizer to compare. Newer models learned bigger vocabularies (called cl100k_base and o200k_base), so the same text usually takes fewer tokens.^[3][4] A token ID only makes sense to the model that uses that tokenizer.

The difference is biggest for code. Paste this five-line function into the tokenizer:

def count_words(text):
    words = text.split()
    if not words:
        return 0
    return len(words)

It takes 44 GPT-2 tokens but 26 with GPT-4’s tokenizer, while the sentence The cat sat on the mat. It looked out of the window and watched the birds fly by. takes 20 in all three. GPT-2 spends a token on nearly every space of indentation, so return 0 alone is 9 tokens; GPT-4’s tokenizer has tokens for runs of spaces and needs 4. Even one company’s models can count differently: Anthropic says its Claude 4.7 and later models produce about 30% more tokens for the same text than earlier ones.^[4][5] So a price per token only compares models that share a tokenizer. To compare costs, count your own text.

How much a model can read at once, and what it costs to use, are counted in tokens, not words. The six nursery rhymes from Ch 1 are 145 words but 195 GPT-2 tokens. Other languages often need many more tokens for the same text.^[2]

How much a model can read at once, and what it costs to use, are counted in tokens, not words. The six nursery rhymes from Ch 1 are 145 words and 195 GPT-2 tokens, about 1.3 tokens per word: punctuation marks are tokens of their own, and rarer words take more than one. Other languages often need many more tokens for the same meaning. In one study, the same text took up to 15 times as many tokens in some languages as in others.^[5][6] That leaves less room in the model’s window and costs more.