Ch 22. Where prompts break

Back in Ch 16, small changes in wording changed an answer. Real prompts have many choices like that: which examples, in what order, laid out how, and where the important part goes. None of them should matter. Here are three, each changed on its own:

Back in Ch 16, small changes in wording changed an answer. Prompts for real tasks have many such choices: which examples, in what order, laid out how, and where the important part goes. None of them should matter to the meaning. Here are three, each changed on its own:

Shuffling the same four examples changed how many of eight reviews GPT-2 small labeled right: from all 8 to just 4, no better than a coin. Much larger models do this too, and an order that works for one model may not work for another.^[1]

Shuffling the same four examples changed how many of the eight reviews GPT-2 small labeled right from 8 of 8 to 4 of 8, which is no better than a coin. Researchers found the same effect in much larger models: some orders do nearly as well as a fully trained classifier and others are no better than random guessing, and a good order for one model doesn’t carry over to another.^[1][1]

Why order matters

The orders that went badly have something in common. All five that scored 4 of 8 ended with a positive example, and all twelve that ended with a negative one scored 6 or more. Models tend to favor labels that appear near the end of the prompt, and labels that were common in their training text. One fix, contextual calibration, measures that lean directly: give the prompt a test input with no content, like “N/A”, see how far the answer tilts away from even, and correct every real answer by the same adjustment. It raised GPT-3 and GPT-2’s average accuracy by up to 30 points and made the choice of prompt matter less.^[2]

Writing the examples as “Review: … Sentiment: …” got 7 of 8 right; an arrow between review and label got 4. Formatting alone makes big differences in larger models too.^[2]

Writing the same examples as “Review: … Sentiment: …” got 7 of 8 right; an arrow between review and label got 4. A study of open models found differences of up to 76 percentage points from formatting alone, and the gap didn’t go away with bigger models, more examples, or instruction tuning.^[2][3]

With 30 lines of locker numbers, Qwen2.5-1.5B-Instruct almost always found the right one when it was the first or last line. Every miss came with it somewhere in the middle. Studies of large models found the same thing.^[3]

With 30 lines of locker numbers, Qwen2.5-1.5B-Instruct found the right one almost every time when it was the first or last line. All of its 8 misses came with the line somewhere in the middle. Large studies show the same U shape: models use information at the start and end of a long prompt better than information in the middle.^[3][4]

When Qwen missed, it missed completely. In all 8 misses, the right number got under a 2% chance, while 76 of the other 80 answers got over 99%. The model wasn’t unsure about those lines; it put nearly all its probability on some other answer.

Adding a sentence that has nothing to do with the task can matter too. Here are 24 short word problems for Qwen2.5-1.5B-Instruct, as written and with one extra sentence, like “Her friend baked 30 cookies.“:

Adding a sentence that has nothing to do with the task can matter too. Here are 24 short word problems for Qwen2.5-1.5B-Instruct, each asked as written, then with one irrelevant sentence added, like “Her friend baked 30 cookies.“:

As written, it solved all 24; with the extra sentence, 18. Most of the misses used the extra number: it added the friend’s 30 cookies to Nina’s and answered 63 instead of 33. Much larger models do this too.^[4] Telling it to ignore anything irrelevant didn’t help this small model.

As written, it solved all 24. With the extra sentence it solved 18, and 5 of its 6 misses used the irrelevant number: for Nina’s cookies it took the 48 she baked, subtracted the 15 she gave away, and added her friend’s 30, answering 63 instead of 33. Researchers found the same with much larger models: accuracy on grade-school math problems dropped sharply when one irrelevant sentence was added. One fix they found was to say that irrelevant information may be ignored.^[4][5] For this small model it didn’t help: 17 of 24.

People often ask the question first and add details as they go. Top models do much worse that way: they guess too early and don’t recover.^[5] Here are the same 24 problems, asked first as the bare question, with the two facts following in two more messages:

People often start a chat with the question and add details as they go. A large study found that top models did 39% worse on average when a task’s details arrived over several turns instead of in one message: they made assumptions early, tried to answer too soon, and didn’t recover.^[5][6] Here are the same 24 problems, asked first as the bare question, with the two facts following in two more messages:

In one message it solved all 24; over three turns, 18. It answered as soon as it had the first fact, then built on that early answer. Told only that Omar runs laps of a 400-meter track, it answered 10; told he runs 4 laps, it answered 40 instead of 1,600.

In one message it solved all 24; over three turns, 18. Asked the bare question, it usually asked for the details, in 20 of the 24 chats. But as soon as it had the first fact it answered anyway, in 21 of the 24, and when the second fact came it built on that early answer. Told only that Omar runs laps of a 400-meter track, it answered 10; told he runs 4 laps, it answered 40, four times its own guess, instead of 1,600. Every earlier turn stays in the prompt (Ch 24), including the model’s own wrong answers.