Ch 20. Instructions, roles, and the system prompt

Chat models have a slot at the top of the conversation for a system prompt: instructions for the whole chat. In a chat app, its makers write it and you never see it. Chat models were trained to follow instructions like these, so they usually do.^[1] But what kind of instructions help?

The chat template (Ch 15) has a slot ahead of the conversation for a system prompt: instructions that apply to the whole chat. In a chat app, its makers write it and you never see it; when you use a model directly, you write it yourself. Chat models were tuned on examples of an assistant following instructions, and on people’s ratings of which replies were better.^[1][1] So instructions there tend to be followed. But what kind of instructions help?

The specific instructions worked: two short sentences and an everyday example. The persona, “a world-famous physicist”, made the answer more than twice as long, and no better.

The specific instructions worked: two short sentences and an everyday example, shorter than the default answer. The persona, “a world-famous physicist”, made the answer more than twice as long and no better: it added claims that are muddled, like most blue light being “scattered away from us towards the edges of the sky”.

Telling the model it was “a brilliant mathematician who never makes mistakes” didn’t help either: it got 3 of 8 sums right, against 5 of 8 without it. A large study of 162 personas found the same: on average, personas didn’t make answers more accurate.^[2]

The sums tell the same story. Telling the model it is “a brilliant mathematician who never makes mistakes” did not make it one: it got 3 of the 8 right, against 5 of 8 with the default system prompt. A large study found the same across 162 personas and 2,410 factual questions: adding a persona did not improve accuracy on average, and which persona helped on which question was largely random.^[2][2]

Real system prompts often hold dozens of rules. Here Qwen2.5-1.5B-Instruct wrote 12 short notes, each under 1, 2, 4, 6, 8, and then 10 rules that a program can check, like “Don’t use any commas”:

System prompts in real apps often hold dozens of rules. Here Qwen2.5-1.5B-Instruct wrote 12 short notes, each under 1, 2, 4, 6, 8, and then 10 rules, all of which can be kept at once and checked by a program: start with “hello”, use no commas, end with “see you soon”, and so on. Each note starts the list at a different rule, so every rule comes both early and late:

More rules didn’t make it worse at each one: with ten rules it kept 88 of 120, about three in four. But every rule it misses spoils the note. With one rule, 7 of the 12 notes kept it; with ten, not one note kept them all. Much larger models slip too as rules pile up: the best of 20 kept only about two thirds of 500 rules.^[3] So check every rule that matters, and split a long list into steps (Ch 31).

More rules didn’t make the model worse at each one: with a single rule it kept it in 7 of the 12 notes, and with ten it kept 88 of 120, about three in four. What fell is the chance that a note kept every rule: 7 of 12 with one rule, 3 with four, 1 with eight, and none with ten. The misses were mostly in rules that fight the model’s habits, like using no “the” or adding a question, and they came from all over the list. A benchmark of 500 rules found that much larger models slip too as rules pile up: the best of 20 kept only 68% at the full 500, and models favored rules near the start of the list.^[3][3] So a long list of rules needs checking, one rule at a time, and it often works better split into steps (Ch 31).

Why “all of them” falls so fast

If each rule is kept with chance $p$, and the rules are kept or missed independently, all $k$ are kept with chance

With $p=88/120\approx 0.73$, ten rules give ${0.73}^{10}\approx 4\%$: about half a note in 12, and the recording got none. A rule that’s kept 90% of the time still leaves only ${0.9}^{10}\approx 35\%$ of outputs with all ten right.

One sequence, ranked by training

In the template from Ch 15, the system prompt sits between <|im_start|>system and <|im_end|>, and the user’s message follows in the same way. Leave the system prompt out and Qwen’s template fills one in: “You are Qwen, created by Alibaba Cloud. You are a helpful assistant.” Either way, the network gets one sequence of tokens. Nothing in its layers treats the system part differently; any extra weight it gets was learned in tuning.

That’s a weakness when other text in the sequence gives orders. OpenAI researchers argued that treating system prompts and untrusted text as equals is one of the main things that lets prompt injection work (Ch 33). They trained GPT-3.5 on generated examples of an instruction hierarchy, with system messages ranked above users’ and users’ above tool outputs, teaching it to ignore lower-ranked instructions that conflict. It became much harder to attack, even in ways it hadn’t been trained on, while losing little on ordinary tasks.^[4]