Ch 25. Search and memory

In Ch 24, retrieval found the cat rhyme by meaning. Search engines have long worked another way: a keyword search looks for the question’s own words. Each finds things the other misses. Try a few questions about the same chat:

In Ch 24, retrieval found the cat rhyme by meaning. Search engines have long worked another way: a keyword search scores each text by the question’s own words it contains, after dropping common ones like “the” and “which”. The usual method, BM25, gives more credit for words that are rare among all the texts and for using them more often, and less for long texts.^[1][1] It is exact: it finds a name, or a code like an error number, that an embedding blurs. Anthropic’s example is a search for “Error code TS-999”, where embeddings may find pages about error codes in general and miss the one about TS-999.^[2][2] But it knows nothing of meaning: “kitty” doesn’t match “cat”. Try a few questions about the same chat:

Words found Sam, Leo, and the crown, which meaning missed, but found nothing for “kitty”, which meaning got right. So many systems run both searches and combine the two lists, giving each text points for how high it comes in each.^[1] Here that picked the right exchange for 4 of the 5 questions; words alone got 4 and meaning alone 2.

Words found Sam, Leo, and the crown, which meaning missed: names carry little meaning, and the crown pulled meaning toward Humpty Dumpty and the king’s men. Meaning found “kitty”, a word in no exchange. So many systems run both searches and merge the two lists by rank: each text gets points for how high it comes in each list, so one near the top of either does well.^[3][3] Anthropic found that adding keyword search to embeddings missed fewer of the right passages.^[2][2] Here the merged list put the right exchange first for 4 of the 5 questions, words alone for 4, and meaning alone for 2. It still missed Leo: meaning ranked the star rhyme first and words second, which beat Leo’s rhyme, first by words but only third by meaning.

Scoring words and ranks

For a question with words $q$, BM25 scores a text $D$ as

where $f(q,D)$ counts the word in the text, $|D|$ is the text’s length in words, avgdl the average length, and $k_1=1.2$ and $b=0.75$ are the usual settings.^[1] The count saturates: a word used five times scores less than five times one use. The weight $\text{idf}(q)=\ln (1+\frac{N-n+0.5}{n+0.5})$, for a word in $n$ of the $N$ texts, is highest for words in only one; the 1 inside the logarithm keeps a word found in every text from counting against a match.

Reciprocal rank fusion merges the lists using only ranks, so it needn’t compare BM25 scores with cosines: each text scores $\sum 1/(k+r)$ over the lists that rank it, at rank $r$, with $k=60$.^[3] For Leo’s question the star rhyme gets $1/62+1/61\approx 0.0325$ and Leo’s rhyme $1/61+1/63\approx 0.0323$.

All of that happens inside one chat. A new chat starts with an empty prompt, and the model remembers nothing of the last one. Apps with a memory feature save notes outside the model and put them into the next chat’s prompt.^[2] Here the harness ends the long chat by asking the model to write notes, in two ways:

All of that happens inside one conversation. A new chat starts with an empty prompt, and the model, whose weights don’t change as you talk, remembers nothing of the last one. Memory features fill the gap the way the strategies in Ch 24 do: the harness saves notes outside the model and puts them into the next chat’s prompt. ChatGPT, for example, saves things you ask it to remember and details it picks up itself, and later began drawing on past chats too.^[4][4] Here the harness ends the long chat by asking the model to write notes, in two ways:

Asked openly, the model wrote only about the last child, Mei, and made up the rest, such as plans for an astronomy camp. With those notes, the new chat couldn’t answer any question. Asked for each child and their rhyme, it did better but mixed up Sam’s and Mei’s rhymes, so the new chat said Sam loves “Twinkle, twinkle” without any doubt. A wrong note is repeated in every chat after, which is why apps let you see and delete what they remember.^[2]

Without notes, the new chat knows nothing and says so. Asked openly for notes about the user, the model wrote only about the last child, Mei, and filled the rest with things that never came up: an interest in astronomy, books about space, an astronomy camp next summer. With those notes it still couldn’t answer any of the three questions. Asked for each child and the rhyme they love, it got three right but gave Sam the star rhyme, which was Mei’s, gave Mei Leo’s clock rhyme, and left out Humpty Dumpty. The new chat then said Sam loves “Twinkle, twinkle”, with no sign of doubt.

So what gets written down matters, and a wrong note is worse than a wrong reply: it is repeated in every chat after. That’s why apps let you see what is remembered, delete it, turn memory off, or chat without it, and try not to save sensitive details, such as health, unless asked.^[4][4] Some systems go further and let the model decide what to move in and out of its prompt as it works, the way a computer moves data between fast memory and disk.^[5][5]

What notes cost

The notes were 193 tokens from the open request and 221 from the specific one, against 616 for the whole chat. With the specific notes, the new chat’s prompt for Omar’s question grew from 25 tokens to 254. Every later chat pays for its notes, so they have to stay short, and notes saved after every chat pile up into the same problem as a long chat. The same three answers apply: drop old notes, summarize them, or retrieve only those that match the question.