Ch 24. Managing the context

Every message and every reply is added to the prompt, and the model re-reads all of it on every call. But a model can only read so much at once: its context window. Long prompts are also slower and cost more. So the harness has to choose what the model sees.

Every message and every reply is added to the prompt, and the model re-reads all of it on every call (Ch 23). But a model can only read so much at once: its context window (Ch 8). Long before that limit, long prompts also get slower and more expensive. So the harness has to choose what the model sees.

• Drop the oldest. Simple, but here the first rhyme was the one you asked about. Without it, the model couldn’t say which child loves it.
• Summarize. The harness asks the model to shrink old turns into a short note. The note kept “Lily”, so the answer was right, but summaries can lose details that turn out to matter.
• Retrieve. The harness stores old turns outside the prompt and puts back the ones closest in meaning to the question.^[1] The cat rhyme came back, and the answer was right.

• Drop the oldest. Simple, and often fine. But here the first rhyme was the one you asked about. With it gone, the model said the conversation didn’t mention it. Here it admitted the gap; a model can just as easily fill one with a confident guess (Ch 16).
• Summarize. The harness asks the model to condense the old turns into a short note, then keeps the note instead. The summary kept “Lily”, so the answer was right. Summaries lose detail, though, and you can’t always tell in advance which details will matter.
• Retrieve. The harness keeps the old turns stored outside the prompt, and when a question comes in, finds the ones most similar in meaning, using embeddings like the map in Ch 5, and puts them back.^[1][1] The cat rhyme was the closest match to the question, so it came back and the model answered correctly.

Picking by similarity

The harness turned the question and each stored exchange into 384 numbers with MiniLM, the embedding model from Ch 5, and scored each exchange by the cosine of the angle between its vector $v$ and the question’s vector $u$:

The cat rhyme scored 0.49 and the other five 0.35 to 0.45, so it came first. None scored very low, because every exchange shares “nursery rhyme” and “loves”: close in meaning isn’t always the same as useful. The summary was cheaper still, shrinking the 149-token first exchange to a 30-token note, but it only worked because the note happened to keep the name you later asked about. It also kept a mistake: the model’s first reply said a cat catches a mouse, which the rhyme never says, and the summary repeated it.

The window isn’t all room for the prompt: the reply has to fit in it too, so the harness leaves space for it. And models don’t use a full window well. In one test, only half of the models that claimed to read 32,000 tokens still did well at that length on harder tasks.^[2] So a shorter prompt can be better even when everything fits.

The window isn’t all room for the prompt. The reply is written into the same window, and so is any thinking the model does first, so the harness has to leave space for them. Anthropic’s API, for example, counts the system prompt, every message, the tool definitions and results, and the output. If the prompt alone is too long, the API refuses it with an error, “prompt is too long”; chat apps instead drop the oldest turns for you, the first of the three ways above.^[2][2] The 600-token budget here counts only the prompt, to keep the example small.

A window that fits everything isn’t always used well, either. RULER tested 17 models that each claimed to read at least 32,000 tokens. Nearly all of them found a single fact hidden in a long text, but on harder tasks, such as following a chain of references or counting things across the text, only half still did well at 32,000.^[3][3] Anthropic calls this decline “context rot”: as the token count grows, accuracy and recall get worse.^[2][2] So trimming the context can help even when everything fits.