Ch 3. Text is just numbers

Every character has an agreed number, set by the Unicode Standard: capital T is 84, and a space is 32. There are numbers for most of the world’s writing systems, and for emoji too. Try typing é or 🐱.^[1]

Every character has an agreed number, set by the Unicode Standard. Capital T is 84, a lowercase h is 104, and a space is 32. Unicode has room for more than a million characters, covering most of the world’s writing systems, plus emoji.^[1][1]

Try typing é or 🐱. They get numbers too, just bigger ones: é is 233 and 🐱 is 128,049.

Computers store those numbers in small pieces called bytes. One byte only holds a number up to 255, so bigger numbers take several bytes. Switch the widget to UTF-8 bytes to see it: é takes two bytes, and 🐱 takes four. GPT-2 reads text as these bytes, so it can read any text at all.^[2]

Computers store numbers in bytes, and one byte can only hold a number from 0 to 255. A common way to store text, called UTF-8, uses one byte for the basic English letters, digits, and punctuation, and two to four bytes for everything else.^[2][2] Switch the widget to UTF-8 bytes to see it: é becomes two bytes, and 🐱 becomes four.

GPT-2, the model used in this site, starts from these bytes. Its designers chose bytes so that the model could read any text at all, in any language, without a special case for each one.^[3][3]

How UTF-8 packs a number into bytes

Unicode numbers its characters from 0 to 1,114,111 (written U+0000 to U+10FFFF).^[1] UTF-8 picks how many bytes a character gets from its number $c$: one byte when $c<128$, two when $c<\mathrm{2,048}$, three when $c<\mathrm{65,536}$, and four for the rest.^[2] For a two-byte character, the bytes are

where $\lfloor \cdot \rfloor$ rounds down and $cmod64$ is the remainder after dividing by 64. For é, $c=233$: 233 is 3 sixty-fours with 41 left over, so the bytes are 195 and 169. Three- and four-byte characters split $c$ the same way, six bits per byte after the first.

The scheme was built for two things. Text in plain English, with every character below 128, is the same byte for byte as the older ASCII code. And every byte after the first in a character starts with the bits 10 (a value from 128 to 191), so a reader dropped anywhere in a stream of bytes can find where the next character starts.^[2]