Ch 12. Scale

Everything so far is how today’s largest models work too. What changed is size. A year after GPT-2, GPT-3 used the same design with more than 1,000 times as many weights.^[1] Compare the models on this site with some of the largest ones:

Everything so far, from tokens to attention to the generation loop, is how today’s largest models work too. What changed is size. A year after GPT-2, GPT-3 used the same design with 175 billion weights, more than 1,000 times GPT-2 small, and read 300 billion tokens in training.^[1][1] Compare the models on this site with some of the largest ones that published their numbers:

The bars use a log scale: each step is ten times the one before. The companies behind the biggest commercial models mostly don’t publish these numbers.

Training the biggest models costs tens of millions of dollars, and the price has grown about 2.4 times every year.^[5] Small models have gone the other way: in 2024, a model the size of GPT-2 small, the one on this site, was trained from scratch for about $20.^[6] Using models has gotten cheaper fast as well: the lowest price for a model that reaches a given test score has fallen between 9 and 900 times a year, depending on the test.^[7]

The bars use a log scale: each step is ten times the one before, or the big models wouldn’t fit on the screen. Compute is counted in arithmetic operations; the largest Llama 3 model took about 3.8 × 10²⁵ of them.^[4][4] The companies behind the biggest commercial models mostly don’t publish these numbers.

In dollars, training the most compute-hungry models has gotten about 2.4 times more expensive each year since 2016. For GPT-4 and Gemini, the chips and the staff each cost tens of millions of dollars, and if the trend holds, the largest runs will cost more than a billion dollars by 2027.^[5][5] Small models have gone the other way. In 2024, a model with GPT-2 small’s design and size, the one on this site, was trained from scratch in about 90 minutes on one machine with 8 GPUs, for about $20, and did slightly better than the original on a common-sense test.^[6][6] The full 1.5-billion-weight GPT-2 took 24 hours and $672; in 2019, training it had been a project for a whole team.^[7][7]

Using a model at a given level has gotten cheaper fast too. Over the three years to early 2025, the lowest price for a model that reached a given benchmark score fell between 9 and 900 times a year, depending on the benchmark and the score. Reaching GPT-4’s score on a set of PhD-level science questions got about 40 times cheaper each year. The fastest drops came in the last year of that span, so they may not last.^[8][8]

Counting the compute

Training compute is usually estimated as

operations, where $N$ is the number of weights and $D$ the number of training tokens: about 2 per weight for each token going forward through the model, and 4 more for backpropagation.^[9] For GPT-3, $6\times 175\times {10}^9\times 300\times {10}^9\approx 3.15\times {10}^{23}$, next to the paper’s own count of 3.14 × 10²³.^[1] The $20 copy of GPT-2 small read about 10 billion tokens, so it took about $6\times 124\times {10}^6\times {10}^{10}\approx 7.5\times {10}^{18}$ operations, about 5 million times fewer than the largest Llama 3.^[6]

In 2020, researchers found that a model’s loss, how wrong its guesses are, falls along a smooth, predictable curve as models get bigger, as long as they have enough text to learn from.^[8] A rule like this is called a scaling law. Slide along theirs:

In 2020, researchers trained many language models of different sizes and found that the loss (Ch 7) falls along a smooth, predictable curve as models get bigger, as long as they have enough text to learn from.^[9][9] The same held for more training text and more compute. A formula like this is called a scaling law. Slide along theirs:

Each tenfold increase in size lowers the loss by about the same fraction, so improvements keep coming but each one costs ten times more. Labs use curves like this to predict how a huge model will do before they pay for it. The curve doesn’t say which skills will appear.

Each tenfold increase in size lowers the loss by about the same fraction, so improvements keep coming but each one costs ten times more. A lower loss means the model is less surprised by real text: it gives the right next token a higher probability, on average.

Scaling laws let labs predict how well a huge model will do from cheaper small runs before they spend the money. They also say nothing about which skills appear. The GPT-3 paper found that bigger models were much better at picking up a new task from just a few examples in the prompt.^[1][1]

The curve on the chart is Kaplan and colleagues’ fit for size alone:

where $L$ is the loss in nats per token and $N$ counts the weights outside the token and position tables.^[9] Ten times the weights multiplies the loss by ${10}^{-0.076}\approx 0.84$, about 16% lower, at any size. That’s what a straight line on a log–log chart means.

For the same amount of computing, a smaller model trained on more text can beat a giant one. In 2022, Chinchilla, with 70 billion weights, beat Gopher, a model four times its size, by reading far more text.^[3]

For a fixed amount of compute, should you build a bigger model or train it on more text? In 2022 a team trained Chinchilla with 70 billion weights on 1.4 trillion tokens, using the same compute as their earlier 280-billion-weight Gopher, which had read 300 billion. The smaller model, trained on more text, did better.^[3][3] That’s about 20 tokens of training text per weight. Llama 3’s largest model went further, with about 39 tokens per weight: 405 billion weights and 15.6 trillion tokens.^[4][4]

The Chinchilla team fit the loss to both the weights $N$ and the training tokens $D$:

with $E=1.69$, $A=406.4$, $B=410.7$, $\alpha =0.34$, and $\beta =0.28$. $E$ is the loss no model can beat (the text’s own unpredictability), and the other two terms are what a finite model and a finite amount of text add.^[3] The fit predicts 1.94 for Chinchilla and 1.99 for Gopher. Spending a budget $C\approx 6ND$ to make $L$ as low as possible grows $N$ and $D$ together, which works out to about 20 tokens per weight.