Ch 17. Thinking before answering

Several quirks come from answering straight away. Ask someone for 17 × 24 and they’ll reach for paper. A model’s paper is its own output: every token it writes becomes context for the next one, so writing out the steps gives later tokens something to build on. Just asking big models to show their working helped them with math problems.^[1]

Several of the quirks in Ch 16, like miscounted letters, come from answering straight away: the first token of the answer has to be right with no working before it. Ask someone for 17 × 24 and they’ll reach for paper. A language model’s paper is its own output: every token it writes goes back in as context for the next one (Ch 11), so writing out the steps gives later tokens something to build on. Simply asking large models to show their working step by step, called chain of thought, made them better at math word problems.^[1][1]

A reasoning model does this every time: it writes scratch work before its answer. Here is Qwen3-0.6B, a small model whose thinking can be switched off, on six puzzles:^[2]

A reasoning model does this every time. Before its answer it writes a stretch of scratch work, which chat apps usually hide or fold away. Here is Qwen3-0.6B, a small open model whose thinking can be switched off,^[2][2] on six puzzles, recorded with thinking off and on:

Models are also trained to think. Give them problems whose answers can be checked, and reward whatever they wrote when the answer is right. DeepSeek’s model, trained this way, started writing longer answers and re-checking its steps on its own.^[5]

Models aren’t only asked to think; they are trained to. One recipe: give the model problems whose answers can be checked automatically, like math with a single number as the answer, and reward whatever it wrote when the final answer is right. DeepSeek trained a large model this way. Over training, its answers grew longer on their own, it began to re-check its steps, and its score on a hard math competition rose from 15.6% to 71.0%.^[5][5] The model is still predicting one token at a time. What changed is that it learned which kinds of scratch work tend to lead to right answers.

Rewarding the group

DeepSeek’s method, GRPO, samples a group of $G$ answers to the same problem and scores each with simple rules: a reward $r_i$ for a right final answer, plus one for putting the scratch work inside the expected tags. Each answer is then judged against its own group:

where $A_i$ is answer $i$’s advantage: positive if it did better than the group’s average, in units of how much the group’s rewards spread out. Training raises the probability of every token in answers with a positive advantage and lowers it for the rest, with a limit on how far one step can move and a penalty for drifting from the starting model, as in Ch 15.^[5] Comparing within the group means no separate model is needed to say how good an average answer is.

Small reasoning models usually learn by copying a big one instead, through distillation (Ch 15).^[5]

Small reasoning models usually learn from a big one instead, by distillation (Ch 15). DeepSeek tuned small Qwen and Llama models on 800,000 examples made with its big model, and released them with names like DeepSeek-R1-Distill-Qwen-1.5B. For a 32-billion-weight Qwen model, this worked much better than training it with rewards directly.^[5][5] Qwen3-0.6B, the model above, was also tuned to copy the probabilities of larger Qwen3 models.^[2][2]

Thinking is bought with tokens, and tokens take time and money.

The recommended settings for Qwen3’s thinking mode leave room for up to 32,768 tokens of output, and warn against always picking the most likely token, which can make it repeat itself endlessly.^[6][6] Thinking is bought with tokens, and tokens take time and money.

Qwen3 writes its scratch work between <think> and </think> tags, so a chat app can find it and fold it away; with thinking switched off, the model skips it.^[6]