Ch 7. Learning from mistakes

Nobody sets a model’s weights by hand. They are found by training: show the model examples, measure how wrong its predictions are, and nudge every weight so it’s a little less wrong. Then repeat, millions of times.

For a language model, any text is full of examples: at every point, the words so far are the question and the next word is the answer. The counting model learned from the same kind of text in Ch 1, by filling in a table. A network has no table to fill in, so it starts from random weights and improves them a little at a time.

Getting a program’s numbers from examples, instead of writing its rules, is called machine learning. Doing it with many stacked layers of neurons is deep learning, and a large language model is a deep network trained on text. All of it is part of artificial intelligence, a field named in 1955.^[1]

In Ch 6 you set the weights yourself. A real model’s weights are found by training, a loop with three parts:

1. Predict. Run the model on examples whose right answers are known.
2. Measure. Score how wrong the predictions were.
3. Nudge. Change every weight a little, in whichever direction makes the score better.

Then repeat, millions of times. For a language model the examples come free with any text: at every point, the words so far are the question and the word that comes next is the answer.

The counting model in Ch 1 learned from text like this too. Counting is a kind of training: read the text once and fill in a table. A network can’t be filled in like that, because no single weight stands for one pair of words; each weight plays a part in many predictions. So it starts from random weights and improves them by trial and correction.

This is what machine learning means: a program’s numbers come from examples instead of rules someone wrote. In 1959 Arthur Samuel described a checkers program that got better by playing, in a paper titled Some Studies in Machine Learning Using the Game of Checkers.^[1][1] Networks with many stacked layers of neurons, trained this way, are deep learning; GPT-2 stacks 12 blocks of them. A large language model is a deep network trained on text to predict the next token. All of this sits inside the older, wider field of artificial intelligence, named in a 1955 proposal for a summer workshop at Dartmouth, which also covers programs that follow rules people wrote, with no learning at all.^[2][2]

To see every weight move, start with a network far smaller than a language model. It has one input and one output: for each place along a line, it predicts a height. The dots are its examples, and the curve is its prediction. Press Take 1 step to nudge every number once, or Train to watch it learn to follow the dots.

To see every weight move, start with a network far smaller than a language model. It has one input and one output: for each place along a line, it predicts a height. The dots are the examples, and the curve is what the network currently predicts. It has 16 neurons, which squash with tanh, one of the choices in the neuron above, and 49 numbers, all listed under it. They start random, so the curve starts wrong. Press Take 1 step to see every number nudged once, or Train to watch them learn to follow the dots.

Each step is the loop above: predict a height for every dot, measure how far off the curve is, and nudge all 49 numbers. The next two sections take apart the measuring and the nudging.

To improve, the network needs a score for how wrong it is, called the loss. For a language model, the loss is high when the model gave the word that really came next a tiny chance, and low when it gave it a big one. Training means changing the weights so the loss goes down.

To improve, the network needs a score for how wrong it is: the loss. For the curve above, the loss is the average squared distance between the line and the dots. For a language model, it’s how surprised the model was by the word that actually came next: if it gave the right word a high probability, the loss is low; if it gave it a tiny probability, the loss is high.

Training means changing the weights so that the loss goes down.

The loss, written down

The curve fitter’s loss is the mean squared error over its $N$ dots:

where $y_i$ is dot $i$’s height and ${\hat{y}}_i$ is the line’s height at the same place. A language model’s loss is the cross-entropy: the average surprise at each real next token,

where $p(x_i|\dots )$ is the chance the model gave the token that really came next. A chance of 1 costs nothing; a chance near 0 costs a lot. Guessing evenly among $V$ choices gives every one a chance of $1/V$, so the loss is $\ln V$: that’s the dashed line on the tiny model below. Papers often report perplexity, $e^L$: a perplexity of 30 means the model is as unsure as if it were choosing evenly among 30 tokens.^[3]

A real network has millions of weights, so its landscape has millions of directions. A method called backpropagation works out which way to nudge every weight at once.^[2]

For a network with millions of weights, the landscape has millions of directions. The trick that makes this practical is backpropagation: working backwards from the loss through every multiply and add, it finds which way to nudge every weight at once.^[3][4] The networks on this page use it, written out by hand, and use a popular recipe for the step size called Adam.^[4][5]

One step, written down

Gradient descent moves each weight $w$ against the slope of the loss:

where $\frac{\partial L}{\partial w}$ is how fast the loss changes as $w$ changes (the slope in that direction) and $\eta$ is the step size, also called the learning rate. Backpropagation gets every slope from the chain rule, working backwards one operation at a time. For a single neuron with $y=f(z)$ and $z={\sum }_i{w}_i{x}_i+b$:

so a weight’s slope is the slope arriving from above, times how steep the activation is, times the input that weight multiplies.^[4] Adam changes the step: it keeps a running average of each weight’s slope and of its square, and divides the first by the square root of the second, so every weight gets its own step size. The networks here use its usual settings, 0.9 and 0.999 for how much of each average carries over from one step to the next.^[5]

The same recipe trains a language model. The one below reads 3 characters and guesses the next, using the nursery rhymes from Ch 1. Each character gets its own list of numbers, those feed a layer of neurons, and out comes a chance for every character: the whole path from the last chapter, in miniature. It trains entirely in your browser.

The same recipe trains a language model. The one below reads the 3 characters before a gap and predicts the next character, using the six nursery rhymes from Ch 1. It has every part from the path in Ch 6, in miniature:

• each character becomes a vector of 8 numbers, as tokens do in Ch 5;
• the 3 vectors feed a layer of 64 neurons;
• the neurons feed an output that gives a chance for every character (Ch 2).

It’s a scaled-down version of a design from 2003,^[5][6] and it trains entirely in your browser. All of its numbers start random, the character vectors included, and every step nudges every one of them.

At first it writes random letters. As it trains, its loss falls, and real words appear, then pieces of real phrases. Its character lists were learned too. That’s where the token vectors of Ch 5 come from: they are weights like any other.

At first it writes random letters, and its loss sits on the dashed line: the loss you’d get by guessing every character with equal chance. As training goes on, the loss falls and the writing starts to look like the rhymes: real words appear, then fragments of real phrases.

Nothing told it what the characters are. Its character vectors were nudged along with every other weight, just to make the next character easier to predict. That is the answer to where the token vectors in Ch 5 come from: they’re weights, learned the same way.

GPT-2 was trained the same way, just much bigger: it learned from millions of web pages.^[3]

GPT-2 was trained the same way, just much bigger: it predicted the next token across about 40 GB of text from 8 million web pages, nudging all of its 124 million weights to make the right tokens more likely.^[6][7]

Overfitting

A network can keep getting better at its training text while getting worse at text it hasn’t seen, by memorizing the examples instead of learning their patterns. This is called overfitting. To catch it, trainers set aside some text the network never learns from and track the loss on that too: when the training loss keeps falling but the loss on the set-aside text starts to rise, the network has begun to memorize.^[8]

Seeing the same text again and again makes memorizing easier, and the biggest models may soon run short of new text to learn from. So researchers tested repeating it. With models of up to 9 billion weights, training on the same text up to 4 times did about as well as the same amount of new text. With more repeats, each extra pass helped less, until more training added nothing.^[9]

Dropout

Training GPT-2 also used dropout, a guard against overfitting: at each step, some of the numbers flowing through the network are set to zero at random, each with a chance of 0.1, a different tenth every step.^[10] The network can’t rely on any one neuron, so it memorizes its training text less and does better on text it hasn’t seen.^[11] Dropout is only for training: when the model is used, nothing is dropped, so the same input always gives the same probabilities.

The rest of the course keeps this recipe. Attention and the transformer block, in the next Part, are more weights, trained the same way. Today’s biggest models use the same recipe with far more weights and text, and a chat assistant is the same model trained further on conversations.

From here on, every model in the course is weights set by this loop. What changes is how the weights are arranged, how many there are, and which examples they learn from:

• Arranged differently. Attention and the transformer block (Ch 9–10) add new parts to the calculation, and training sets their weights just as it set these.
• Many more of them. Today’s largest models follow the same recipe with billions of weights (Ch 12), and storing them all takes its own tricks (Ch 13).
• Different examples. An assistant is a base model trained further on conversations (Ch 15).

Training also explains some limits. The loss only rewards text that’s likely to come next, not text that’s true, which is one reason models make things up (Ch 16).