Ch 32. Making things up

As Ch 16 showed, a model can make up a fact and state it as confidently as a true one. Bigger models do it less, but nothing in a reply marks the made-up parts. In 2023, two New York lawyers filed a court brief citing six court decisions that ChatGPT had invented. When they asked it whether the cases were real, it said they were. The judge fined them $5,000.^[1]

References are a common place for this. Asked for short literature reviews, ChatGPT made up more than half of its references, and the newer GPT-4 made up about one in five.^[2]

As Ch 16 showed, a model writes likely text, not checked facts. When it doesn’t know an answer, it can still write a confident one: a hallucination. Bigger, tuned models do this less often, but nothing in a reply marks which parts are made up. In 2023, two New York lawyers filed a court brief citing six decisions that ChatGPT had invented, quotes and all. When they asked it whether the cases were real, it said they were. The judge fined them $5,000 and ordered them to write to each judge falsely named as an author.^[1][1]

References are a common place for this. Two researchers asked ChatGPT for short literature reviews on 42 topics and checked all 636 references in them: 55% of those from GPT-3.5 and 18% of those from GPT-4 didn’t exist. Of the real ones, 43% and 24% had substantive errors.^[2][2] A reference list looks the same either way.

One way to catch a guess is to ask the same question several times. If the model knows the answer, it gives the same one each time; if it’s guessing, the answers scatter.^[3] Below, a small chat model answered each of 11 questions ten times. Three of them ask about a book, a chemist, and a town invented for this chapter.

With sampling on (Ch 2), the same question can get a different answer each time, and that is useful. When a model knows a fact, most of its probability sits on one answer, so the samples agree. When it’s guessing, the probability is spread thin, so they scatter. SelfCheckGPT turned this into a check that needs nothing but the model itself: sample several answers and see whether they agree.^[3][3] Below, Qwen2.5-1.5B-Instruct answered each of 11 questions ten times, sampled with its usual settings. Three of them ask about a book, a chemist, and a town invented for this chapter.

Asked who wrote a book that doesn’t exist, the model names nine different authors. On the questions it knows, all ten answers match. A check that trusts an answer only when 8 of 10 agree flags 4 of the 5 wrong answers and none of the right ones. It misses the Tour de France: all ten answers say Eddy Merckx, who won it five times, but not in 1987. Agreement shows the model is sure, not that it’s right.

A common tip is to turn the temperature down (Ch 2). At temperature 0 the model gets the same six right and the same five wrong. For the invented book it now says “Helen Macdonald” every time, and nothing shows that it’s guessing.

Asked who wrote a book that doesn’t exist, the model names nine different authors in ten tries. On the six questions it gets right, all ten samples agree. A harness that passes an answer on only when at least 8 of 10 samples agree flags 4 of the 5 wrong answers and none of the right ones. The one it misses is the Tour de France: all ten samples say Eddy Merckx, who won it five times, but not in 1987, when Stephen Roche did. When a model has learned something wrong, it says it every time, so agreement shows the model is sure, not that it’s right. The check also costs ten calls instead of one; Ch 27 got a cheaper signal from the model’s confidence in a single answer.

A common tip is to turn the temperature down. At temperature 0, taking the likeliest token every time (Ch 2), the model gets the same six right and the same five wrong. For the invented book it says “Helen Macdonald”, a name none of the ten samples gave, and it says it every time. A lower temperature doesn’t make a guess true; it makes the guess repeatable, and hides the scatter that gave it away. The line under the table shows each answer at temperature 0.

Measuring the scatter

The spread can be put in one number: the entropy of the answers. If $n_i$ of the $N$ samples give answer $i$,

which is 0 when all ten agree and $\ln 10\approx 2.30$ when all ten differ. The invented book scores 2.16; the 1995 Nobel Prize, with 6 votes for Saul Bellow, 3 for Svetlana Alexievich, and 1 for “Aleksievich”, scores 0.90. Counting the two spellings as one answer brings it down to 0.67. Semantic entropy does exactly that: it groups the samples by meaning, using a second model to judge whether two answers say the same thing, before taking the entropy.^[4] The widget only groups answers that match word for word.

Tick the box to add one sentence to the system prompt: if you’re not sure, say “I don’t know”. Now the model declines every time for the invented book and town, and keeps all its right answers. But it still makes up a year for the invented chemist, and it still says Merckx.

Tick the box to add one sentence to the system prompt: if you are not sure of the answer, say “I don’t know”. The model now declines all ten times for the invented book and the invented town, and half the time for the 1995 Nobel Prize. It keeps all six right answers. But it still makes up a prize year for the invented chemist, and it still says Merckx: a model can only decline when something in it registers that it doesn’t know.

Why do models guess at all? Training rewards likely text, true or not, and facts that appear in few training documents are the ones models most often get wrong.^[4] And most tests of models are graded like school exams: a right answer scores a point, and a wrong answer or “I don’t know” scores nothing. A guess never scores less than admitting doubt, so a model tuned to do well on those tests learns to guess.^[5]

Part of the cause is training: as in Ch 16, the loss rewards likely text, true or not, and a fact the training text mentions only once leaves no pattern to learn from. Rarity matters well beyond once. Across several question sets and models up to 176 billion weights, accuracy on a question rose with the number of training documents about its subject, and retrieving documents at answer time made the count matter less.^[4][5] That fits the widget: all four well-known answers come back right, and two of the four less well-known ones do.

Researchers at OpenAI and Georgia Tech argue that the rest comes from how models are tested. Most benchmarks grade like a multiple-choice exam: 1 point for a right answer and 0 for a wrong one, and “I don’t know” also scores 0. Under that scoring a guess never does worse than admitting doubt, so a model tuned to climb those leaderboards learns to guess. Of the ten widely reported benchmarks they looked at, nine gave no credit at all for “I don’t know”.^[5][6] Their proposed fix is to score mistakes below “I don’t know”, and to say so in the instructions.

Where guessing stops paying

Their proposal tells the model: answer only if you are more than $t$ confident, since a mistake costs $t/(1-t)$ points, a right answer earns 1, and “I don’t know” earns 0. If the model’s chance of being right is $p$, answering is worth

points on average, which is above 0 exactly when $p>t$. With $t=0.9$ a mistake costs 9 points, so only an answer that is more than 90% likely to be right is worth giving. Ordinary exam grading is $t=0$, where any guess beats “I don’t know”.^[6]

They give a bound for training too. For facts with no pattern to them, like people’s birthdays, a base model should be expected to get at least as large a share wrong as the share of those facts that appear exactly once in its training text: if 20% of birthdays appear once, expect at least 20% of its answers about birthdays to be wrong.^[6]

Made-up answers can also open a door for attackers. Models that write code sometimes tell you to install software packages that don’t exist: in one study, at least 5% of the packages suggested by commercial models and 22% from open models were made up.^[6] Anyone can publish a package under an unused name, so an attacker can publish harmful code under a name a model invented, and wait for someone to install it.

Made-up answers can also open a door for attackers. When models that write code suggest software packages to install, some of the packages don’t exist. Across 16 models and 576,000 code samples, on average at least 5.2% of the packages suggested by commercial models and 21.7% of those from open-source models were made up: 205,474 different invented names in all.^[6][7] Package registries let anyone publish under an unused name. So an attacker can publish harmful code under a name a model invented, and wait for a person, or an agent running install commands (Ch 29), to install it.

• Give it the facts. A model that answers from text put into its prompt (Ch 25) has less to guess. Even so, legal research tools that look up cases before answering still made things up 17% to 33% of the time.^[7]
• Let it decline, as the box above does.
• Ask more than once when a wrong answer would be costly.
• Check one claim at a time, in a new chat. A model that has given a wrong answer tends to defend it with more wrong claims, but asked about each claim on its own, it often spots the mistake.^[8]
• Check what can be checked, with code or a person: that a court case exists, that a paper is real, that a package is real.

No single fix removes the problem, but several help:

• Put the facts in the prompt. A model answering from text the harness retrieved (Ch 25) has less to guess. It still isn’t immune: in a preregistered study, legal research tools from LexisNexis and Thomson Reuters that retrieve cases before answering, some advertised as free of hallucinations, made things up 17% to 33% of the time.^[7][8]
• Let it decline, as the system prompt above does, and when you test it (Ch 35), don’t score “I don’t know” like a wrong answer.
• Ask more than once when a wrong answer would be costly, and treat disagreement as a warning. Don’t turn the temperature down to make answers look consistent.
• Check claims one at a time, in a fresh call. Once a model has committed to a wrong answer, it tends to justify it with more false claims, ones it can recognize as false on their own: shown its own wrong claims in a separate session, ChatGPT spotted 67% of them and GPT-4 87%.^[8][9] So an explanation the model writes after its answer defends the answer more than it checks it.
• Check what can be checked with code: that a cited case is in a court database, that a cited paper turns up in a library search, that a package is in the registry, that a number matches a tool’s result (Ch 28).

Scoring a long answer

A long answer is rarely all right or all wrong, so one verdict for the whole thing hides most of what matters. FActScore splits an answer into atomic facts, short claims that can each be checked, and reports the share that a reliable source supports. People checked biographies written by commercial models this way, and only 58% of ChatGPT’s facts were supported. Checking every claim by hand is slow, so the authors also built an automatic version, which retrieves from a knowledge source and asks a strong model about each claim, and estimated the score with less than a 2% error.^[10] It’s the same advice as above, made into a measurement: check claims one at a time.