Ch 35. Did it work?

When an agent gets a job wrong, its last message rarely says why. So harnesses keep a trace: a log of every model call and every tool call in the run, in order. Here are the traces of seven runs recorded for earlier chapters, such as Ch 28 and Ch 34. The darker a number, the bigger that call’s prompt.

When an agent fails, its final message rarely says why. Did it search for the wrong thing, get a bad tool result, or follow something it shouldn’t have? To find out, harnesses keep a trace: a record of every model call and tool call in a run, in order, often with how long each took and how many tokens it used. Here are the traces of seven runs recorded for earlier chapters, from the chapters Ch 23 to Ch 34. A darker cell means a bigger prompt.

With no guardrails, the summary run never answers. Its trace shows why: the page told it to delete the files, and it did, then fetched the page again and started over. Its 8 calls read 4,179 tokens in all, and every one of them costs money.

With no guardrails, the summary run ends with no answer, which on its own says little. The trace shows the page’s hidden instruction, three deletions, a second fetch of the same page, and a fourth attempt at a file that is gone. Each call reads its whole prompt again, so the 8 calls read 4,179 tokens in all. Prompts are most of what an agent costs: in τ-bench, a test introduced below, they were 95.9% of the price of a GPT-4o agent.^[1][1] Anthropic says adding full tracing in production is what let them see why their agents failed to find obvious information.^[2][2]

A trace explains one run. To know whether a change to the prompt, the tools, or the model made things better, you need an eval: a fixed list of tasks, each with a way to check the result, run again after every change. Some results code can check exactly. Does the answer contain 7006652? Are all the user’s files still there?

A trace explains one run. To know whether a change to the prompt, the tools, or the model made things better or worse, you need an eval: a fixed set of tasks, each with a way to check the result, run again after every change. That includes changes you didn’t make: a provider can update the model behind the same name, and a prompt that worked can stop working (Ch 45). It needn’t be big: Anthropic started with about 20 tasks drawn from real use, because early changes moved results so much that a few cases showed it.^[2][2]

Where the right answer is known, code can check it: does the answer contain 7006652? For an agent that changes things, it can also check the end state, what the world looks like afterwards, rather than the exact steps, since an agent may reach the same result by different paths.^[2][2] Are all the user’s files still there?

Code checks are fast and never change their mind. They catch the multiplication done in the model’s head (7,050,952 is wrong) and the run that deleted every file. Only 2 of the 7 runs pass both checks.

The two code checks catch the multiplication done without a calculator (7,050,952 is wrong), the reply that never finishes the sentence, the “summary” that mentions no cat facts, and the run that deleted every file. The run with “ask before deleting” keeps the files but never answers, so it fails too. Only 2 of the 7 runs pass both checks.

Code can only check answers it can match, like a number. For free-form answers, like a summary, teams use an LLM as a judge: another model reads the task and the answer and says pass or fail. Here the grader is the same small model that did the tasks. On its own, it passes all five answers, including the wrong product. Shown a correct answer, it catches the wrong product but still passes two answers that are wrong.

String matching only works when the right answer has one fixed form. A good summary can be worded a thousand ways, so teams also use an LLM as a judge: a model given the task, the answer, and often a list of criteria or a correct answer, that says pass or fail. Here the grader is Qwen2.5-1.5B-Instruct, the same small model that ran the tasks. On its own it passes all five answers, agreeing with the code checks on only 2. Shown a correct answer, it fails the wrong product, though its reason has the two numbers the wrong way round, and still passes the reply that never finishes the sentence and the summary with no cat facts: it mixes up the correct answer with the one it is grading. That is 3 of 5.

Strong graders do much better. GPT-4, judging which of two chat answers was better, agreed with people over 80% of the time, as often as people agree with each other. It also favored the answer shown first, longer answers, and, to a degree, its own answers.^[3][3] So a grader needs testing too, against answers people have graded. Anthropic grades research reports with one model call and a rubric: are the claims accurate, the sources good, the task fully covered, the tools used sensibly? People still test by hand, and caught what the grader missed, such as agents choosing pages written to rank in search over better sources.^[2][2]

A test set only holds the tasks someone thought of. Once people use an agent, traces from real runs show the failures nobody wrote a test for, and each one can become a new test. And the same task can go differently from one run to the next, so it’s worth running each test more than once.

A test set only holds the tasks someone thought to write down. Once people use an agent, traces of real runs show the failures nobody wrote a test for, and each can become a new test. Watching needn’t mean reading people’s conversations: Anthropic monitors patterns, such as which decisions agents make and how they interact, without looking at what individual users wrote.^[2][2]

The recordings here use greedy decoding, so a rerun gives the same trace. Real agents usually sample, so the same task can succeed once and fail the next time. τ-bench, a test of agents serving simulated customers that checks the final state of a database much as the files are checked above, runs each task several times. GPT-4o solved about 61% of its shop tasks on a single try, but fewer than 25% on all of 8 tries.^[1][1]

Public test sets, called benchmarks, compare agents on the same tasks. GAIA has 466 questions that are simple for people but need web browsing and tools. People got 92% right; GPT-4 with plugins got 15%.^[1] A benchmark score says how a model does on those tasks, not on yours, which is why Ch 45 suggests trying your own prompts.

A benchmark also only works if the model hasn’t already seen its questions, and models learn from text on the web, where benchmarks are published. Researchers wrote 1,205 new math problems like those in GSM8k, a popular benchmark, and kept them private. Some models scored up to 8 points lower on the new problems, a sign they had partly memorized the old ones. The strongest models scored about the same.^[2] Your own tests can wear out the same way if you keep changing the prompt until they pass, so keep a few aside for a final check.^[3]

Public test sets, benchmarks, compare agents on the same tasks. GAIA has 466 questions that are simple for people but need reasoning, web browsing, handling more than text, and tools. Its authors found people got 92% right and GPT-4 with plugins 15%.^[4][4] A benchmark says how an agent does on its tasks, with its tools, graded its way. Your own tasks may differ on all three, which is why Ch 45 ends with trying your own prompts, and why teams keep an eval of their own. A benchmark also assumes the model hasn’t seen its questions, and models learn from web text, where benchmarks are published. To test this, Scale AI had people write 1,205 new grade-school math problems to match GSM8k, a widely used benchmark, and kept them private. Some models scored up to 8 percentage points lower on the new problems, and the Phi and Mistral families did worse on them at almost every size, while GPT-4, Claude, and Gemini scored about the same on both. Models more likely to write out GSM8k’s own questions tended to drop more, a sign that they had partly memorized them.^[5][5] A model fitted to its test rather than to the task is called overfit (Ch 7).

Your own eval can wear out the same way. Change the prompt until every test passes, and the tests start to measure how well the prompt fits them rather than how it does on new tasks. Machine learning practice keeps a second set of tests aside, used only to check the final version, and refreshes both sets with new examples from time to time.^[6][6]

Once versus every time

If a task succeeds independently with probability $p$ on each try, the chance it succeeds on all of $k$ tries is $p^k$, which falls fast: at $p=0.6$, one try succeeds 60% of the time but eight in a row only ${0.6}^8\approx 1.7\%$ of the time. Real tasks vary, some always passing and some never, which is why τ-bench’s measured drop is gentler than that. τ-bench calls this measure pass^k. From $n$ tries of a task with $c$ successes it estimates it as

averaged over tasks: the chance that $k$ tries picked from the $n$ are all successes.^[1] It is the opposite of pass@k, the chance that at least one of $k$ tries succeeds, which suits tasks where a test can pick out the good try, like code with unit tests.