Part V · Putting it together

Ch 15. From predictor to assistant

A chat is still just text

A model trained on web text, a base model, just continues whatever it’s given. A chat app writes the whole conversation out as one piece of text, with special markers for who’s speaking, and lets the model continue it. That format is the chat template. Here’s the real one the Qwen2.5 chat models use:^[1]

A model trained on web text is a base model: it continues whatever it’s given. So how does a chat app get answers out of one? It writes the conversation out as a single piece of text, in a fixed format with special marker tokens, and lets the model continue it. That format is the chat template. Here is the real one used by the Qwen2.5 chat models:^[1][1]

Try it

What the model actually reads

Computed in your browser

System prompt (empty uses the built-in one) Your next message

Show the raw text

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
The capital of France is Paris.<|im_end|>
<|im_start|>user
And of Italy?<|im_end|>
<|im_start|>assistant
▍

markers are special tokens that only the chat format uses; system, you, and assistant show where each piece came from. The model writes from the cursor on, and stops when it writes <|im_end|>.

Send this exact text to Qwen2.5-0.5B-Instruct and watch it write the reply. Choose a model tier to run a real model in your browser.

The text ends by opening the assistant’s turn, so the likeliest continuation is an answer. When the model writes the end marker, the app stops and shows the reply.

The conversation ends with the marker that opens the assistant’s turn, so the most likely continuation is an answer. When the model writes <|im_end|>, the chat app stops the loop from Ch 11 and shows the reply.

Base model or assistant?

Here are the same prompts given to three models: GPT-2; Qwen2.5-0.5B, a small base model from 2024; and Qwen2.5-0.5B-Instruct, the same model after tuning it to be an assistant.

Try it

Same words, different models

Recorded · GPT-2 small and Qwen2.5-0.5B

Prompt

GPT-2 smallbase model, 2019

How do I boil an egg? You can boil an egg in a pot of water. You can boil an egg in a pot of water. How do I boil an egg in a pot of water? You can boil an egg in a pot

Cut off at 48 tokens.

Qwen2.5-0.5Bbase model

How do I boil an egg? What is the best way to boil an egg? Do those questions have the same meaning? Options are: (1). no. (2). yes. (1).

Stopped by itself.

Qwen2.5-0.5B-Instructtuned, but given the bare text

How do I boil an egg? To boil an egg, follow these steps: 1. **Prepare the Egg**: Make sure your egg is clean and free of any dirt or yolk. If you're using a plastic container to hold the egg, make sure it's sealed

Cut off at 48 tokens.

Qwen2.5-0.5B-Instructtuned, in its chat format

Boiling an egg is a simple and effective way to cook it. Here's how you can do it: 1. **Prepare the Egg**: Make sure your egg is clean and free of any dirt or yolk. If using a plastic

Cut off at 48 tokens.

Each model picks its most likely token every time, so these are repeatable. Shaded text is the prompt; the rest is what the model wrote.

Try your own prompt on GPT-2 and on Qwen2.5-0.5B-Instruct, with and without its chat format. Choose a model tier to run a real model in your browser.

Notice that the base models just continue the text: sometimes they answer, and sometimes they write a quiz or repeat themselves. The tuned model answers and stops, but only when it gets its chat format.

A few things to notice:

Base models continue text. Asked how to boil an egg, the Qwen base model writes another question and a multiple-choice quiz, the way that question might continue on a web page. GPT-2 repeats itself.
Modern base models already know a lot. The Qwen base model answers the capital question and writes a haiku. On the web, a question is often followed by its answer, so an answer is a likely continuation.
The format matters. The tuned model, given the bare question without its chat template, answers and then keeps going: about checking the answer with Python code, or with a stray line of its own template. In its chat format, it answers and stops.

How the tuning works

Turning a base model into an assistant usually takes two more rounds of training, after the long first one. Step through them:

Step through

From predictor to assistant

Made-up example

1 Pretraining
2 Instruction tuning
3 Preference tuning

Trained on: Huge amounts of text from the web, books, and code
Goal: Predict the next token
Example: How do I boil an egg? What is the best way to boil an egg? Do those questions have the same meaning?

A model that continues any text in the style of whatever it resembles. A question might be followed by an answer, or by more questions.

A 2022 paper found that people preferred the answers of a small tuned model over those of a model more than 100 times bigger that it was built from.^[2]

A 2022 paper on InstructGPT describes this recipe. People preferred the answers of a 1.3-billion-weight tuned model over those of the 175-billion-weight GPT-3 it was built from.^[2][2] Qwen2.5’s tuning used over a million examples in the second stage, followed by several stages of the third.^[1][1] Learning from people’s preferences has a catch: people tend to prefer answers that agree with them, so the model learns that too (Ch 16).^[3][3]

The three stages, written down

The second stage is the same next-token loss as pretraining, counted only on the example answers. The third trains a reward model $r$ to score an answer $y$ to a prompt $x$ , from pairs where people preferred $y_{w}$ over $y_{l}$ :

L_{reward} = - \log σ (r (x, y_{w}) - r (x, y_{l}))

where $σ (z) = 1 / (1 + e^{- z})$ turns the gap between the two scores into a chance. The loss is small when the preferred answer scores clearly higher. Then the assistant itself is trained to earn high scores without drifting far from the tuned model it started as:

maximize r (x, y) - β \log \frac{π (y | x)}{π_{SFT} (y | x)}

where $π$ is the model being trained, $π_{SFT}$ is the model after the second stage, and $β$ sets how far it may drift. Without that last term, the model learns to please the reward model in ways people wouldn’t.^[2]

Tuning for your own job

Tuning is just more training, and it works for any job: fine-tuning a model on your own examples teaches it your task and wording. Two tricks make it cheaper. LoRA trains a small add-on instead of changing every weight.^[3] Distillation trains a small model to copy a big one.^[4]

Tuning is just more training, the same measure-and-nudge loop as in Ch 7, and it works for any job, not only chat. Fine-tuning a model on your own examples, such as support questions with the replies you’d want, teaches it your task, format, and wording. Two tricks make it cheaper:

Train a small add-on. Changing every weight of a big model takes memory for all of them and a full copy of the model for each job. LoRA leaves the weights alone and trains two thin tables next to some of the big tables of weights; their product is added to the big table. GPT-2 small turns a token’s vector into its query with a 768 × 768 table: 589,824 weights. Two thin tables, 768 × 8 and 8 × 768, hold 12,288 numbers, about 2% as many. On GPT-3, LoRA trained 10,000 times fewer numbers than changing every weight, needed a third of the memory, and did about as well.^[4][4]
Learn from a bigger model. Distillation trains a small model, the student, to copy a big one, the teacher. The student can learn from the teacher’s whole list of probabilities (Ch 2), not just from which token came next, so it also learns which other answers were nearly right.^[5][5]

In symbols, LoRA replaces a frozen table $W$ (here $d \times d$ ) with

W^{'} = W + \frac{α}{r} B A

where $B$ is $d \times r$ , $A$ is $r \times d$ , $r$ is the small rank (8 above), and $α$ is a fixed scale. Only $A$ and $B$ are trained. $B$ starts at all zeros, so training starts from exactly the original model.^[4]

Distillation softens both models’ probabilities with a temperature $T$ above 1, so the near misses show, and trains the student’s $q^{(T)}$ to match the teacher’s $p^{(T)}$ :

L = - T^{2} \sum_{i} p_{i}^{(T)} \log q_{i}^{(T)}

where the $T^{2}$ keeps the slopes at the same size whatever $T$ is.^[5]

Recipe card

Recap

A chat is written out as one text in a chat template, and the model continues it.
A base model just continues text; tuning teaches it to answer in the chat format and stop.
Tuning has two usual stages after pretraining: learning from example answers, then from people’s preferences between answers.
Fine-tuning trains a model further for your own task. LoRA trains a small add-on instead of every weight; distillation trains a small model to copy a big one’s probabilities.

Preference tuning trains a reward model with $- \log σ (r_{w} - r_{l})$ ; LoRA adds a trained $B A$ to a frozen $W$ .

Next question: even tuned assistants sometimes make things up or trip over simple questions. Why?

Sources

Qwen Team et al. (2024). Qwen2.5 Technical Report. arXiv.
Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.
Edward J. Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
Geoffrey Hinton et al. (2015). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop.

Sources

Qwen Team et al. (2024). Qwen2.5 Technical Report. arXiv.
Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.
Mrinank Sharma et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024.
Edward J. Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
Geoffrey Hinton et al. (2015). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop.

Sources

Qwen Team et al. (2024). Qwen2.5 Technical Report. arXiv.
Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.
Mrinank Sharma et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024.
Edward J. Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
Geoffrey Hinton et al. (2015). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop.