Ch 42. Recap: the whole picture

In short: a language model gives a probability for every possible next token (Ch 0–2). Text is split into tokens, each token becomes a list of numbers, and a stack of blocks refines those numbers, with attention letting each token use the ones before it (Ch 3–10). Training sets the numbers inside by making the right next tokens more likely (Ch 6–7), and writing is a loop of predicting and picking (Ch 11). Tuning makes the model an assistant (Ch 15). A prompt is the start of a document for the model to continue (Ch 18–22). A harness around the model adds memory and picks which model answers (Ch 23–27), and lets it use tools safely (Ch 28–34). Other kinds of models change one piece and keep the rest (Ch 36–41).

In one paragraph: a language model gives a probability for every possible next token (Ch 0–2). Text becomes tokens (Ch 3, 4), each token becomes a vector (Ch 5), and a stack of transformer blocks refines those vectors, with attention letting each token use earlier ones (Ch 8–10). The numbers inside are learned by training that nudges millions or billions of weights to make the right tokens more likely (Ch 6–7, 12). Generating text is a loop of predicting and picking (Ch 11). Those weights can be rounded to fewer bits to save memory (Ch 13), and a small model can draft tokens for a big one to check, so it writes the same text faster (Ch 14). Tuning turns the model into an assistant (Ch 15), and reasoning models are tuned to write their working first (Ch 17). Prompting is choosing the document the model continues: the wording, examples, instructions, and layout all shift what comes next, often in ways you have to test to find out (Ch 18–22). A harness wraps the model with prompts, context management, caching, and a choice of which model answers (Ch 23–27). It can also give the model tools and run it in an agent loop, check its answers for made-up facts, and guard against prompt injection (Ch 28–34). Other kinds of models change one piece and keep the rest: encoders read both ways, mixtures of experts use part of themselves per token, recurrent models keep a fixed memory, vision models read patches, diffusion models paint from noise, and decision models predict moves instead of words (Ch 36–41).

The whole model in a few lines

The In depth passages add up to all of GPT-2 small. For tokens $x_1,\dots ,x_n$, each starts as its row of the token table plus its position’s row (Ch 5, 10):

Each of the 12 blocks then updates every vector with attention and an MLP (Ch 9, 10):

The last token’s final vector is scored against every token in the table and turned into probabilities, with temperature $T$ (Ch 2):

Training lowers the cross-entropy by nudging every weight downhill (Ch 7):

Tuning changes the weights, not these lines. Prompts and harnesses change the tokens that go in, and what happens to the ones that come out. The other kinds of models each change one line: the attention mask, the MLP, attention itself, what a token stands for (a patch of a picture, or a step of a game), or how the output is made.