Ch 13. Fewer bits per number

GPT-2 small has about 124 million weights, and its files store each one in 4 bytes: about 500 MB in all.^[1] The same rule works for any model:

memory for the weights ≈ number of weights × bits per weight ÷ 8 bytes

GPT-2 small has 124,439,808 weights. Its published files store each one as a 32-bit floating-point number: 4 bytes (Ch 3), enough for about seven significant digits.^[1][1] That comes to about 498 MB. The same rule works for any model:

memory for the weights ≈ number of weights × bits per weight ÷ 8 bytes

Does a weight need that much detail? Rounding each number to one of a few allowed values is called quantization. With 8 bits a number can take 256 values; with 4 bits, only 16. Slide GPT-2 small down from 32 bits and watch what it predicts after “The cat sat on the”:

Does a weight need seven digits? Storing numbers with fewer bits, by rounding each one to a small set of allowed values, is called quantization. With 8 bits a number can take 256 different values; with 4 bits, 16; with 2 bits, only 4. To make those few values count, each block of 32 weights shares one scale, set by its largest number, and each weight is stored as a small whole number times that scale. File formats for sharing models, like GGUF’s Q4_0 and Q8_0, work this way.^[2][2] Slide GPT-2 small down from 32 bits and watch what it predicts after “The cat sat on the”:

At 16 and 8 bits nothing visible changes. At 6 and 5 bits the guesses drift, but stay sensible. At 4 bits and below, the model falls apart and writes gibberish.

At 16 and 8 bits nothing visible changes: the same top guess, ” floor”, the same next twelve tokens, and almost the same loss (Ch 7) on a test passage. At 6 and 5 bits the guesses drift: ” bed”, then ” table”, still sensible. At 4 bits and below the model falls apart, repeating “The stack.” or writing gibberish, and its loss shoots up. Look at the dots: as the bits run out, each weight snaps to a coarser grid, and the small weights in a block with one large weight are rounded hardest.

Block rounding, written down

For a block of 32 weights $w_1,\dots ,w_{32}$ whose largest size is $m$, $n$-bit rounding works like this:

where $s$ is the block’s scale (stored once), $q_i$ is the small whole number stored for weight $i$, and ${\hat{w}}_i$ is the value the model uses. Each weight moves by at most $s/2$. In the block the widget shows, $m=0.689$, so at 4 bits the step $s$ is about 0.098, and every weight smaller than 0.049 becomes exactly 0. That’s why the small weights suffer most.

The scale costs 16 bits per block, half a bit per weight, so “4-bit” really means about 4.5 bits per weight: GPT-2 small at 4 bits takes 70 MB, not 62. GGUF’s Q4_0 stores exactly this, 4-bit numbers plus one 16-bit scale per 32.^[2]

This demo rounds in the plainest way. Better methods keep more quality with fewer bits. One lets the model practice with the rounding during training, called quantization-aware training (QAT). Google’s Gemma 4 QAT models use 4-bit weights, and their model card describes their quality as similar to the 16-bit originals.^[2] That is what the qat in a model name means.

This demo uses the plainest method, rounding to the nearest allowed value. Better methods keep more quality at fewer bits. Large transformers have a few unusually large numbers that plain rounding handles badly; treating them separately lets 8-bit versions match 16-bit ones even at 175 billion weights.^[3][3] Another way is to let the model learn to cope with the rounding. With quantization-aware training (QAT), the weights are rounded during training too, so training nudges them (Ch 7) toward values that still work after rounding.^[4][4] Google’s Gemma 4 models come in QAT versions with 4-bit weights, and their model card describes their quality as similar to the 16-bit originals.^[5][5] That is what the qat in a model name means.

In QAT, the forward pass uses the rounded weights $\hat{w}$, so the loss includes the rounding error, while backpropagation treats rounding as if it weren’t there and updates the full-precision weights $w$. Rounding has a slope of 0 almost everywhere, so without that shortcut no weight would ever move.^[4] The outlier method instead finds the few feature dimensions with very large values and multiplies them in 16 bits, and everything else in 8.^[3]