Part VIII · Agents

Ch 29. The agent loop

A loop around the loop

Some tasks need several tools in a row, where each step depends on the last. Take this one: “Find which nursery rhyme mentions a cat, and count its words.” The model has to search, read the result, then count. An agent handles this with one more loop: the harness calls the model, runs the tool it asks for, adds the result, and calls it again, until the model answers without asking for a tool. Step through a real run:

Step through

The agent loop

Recorded · Qwen2.5-1.5B-Instruct

Task Find which nursery rhyme mentions a cat, and count its words.

Model · 301 tokens in
<tool_call> {"name": "search_rhymes", "arguments": {"query": "cat"}} </tool_call>

Model call 1 of at most 8, reading 301 tokens. It writes a tool call, then stops.

So there are two loops. Inside, the model writes each reply one token at a time, as in Ch 11. Around that, the harness keeps calling the model, running tools, and adding results until the model is done.

There are two loops here. Inside each model call is the token loop from Ch 11: predict, pick, append. Around it is the harness loop: call the model, run a tool, add the result, repeat. This pattern of interleaving a model’s reasoning with actions was popularized by the ReAct paper.^[1][1]

The model never saw the rhyme until the search returned it, and it never counted anything: it copied the rhyme into a call to the word counter, then reported the counter’s answer, 30 words.

The harness makes the rules

This run worked partly because of rules in the harness, not the model:

It lets the model ask for only one tool at a time.
Its instructions tell the model to use a tool for counting instead of guessing.
It stops after 8 model calls, so a confused agent can’t go on forever.

Each call also re-reads the whole conversation so far, which grows with every step.

This run worked partly because of choices in the harness, not the model:

One tool at a time. The harness stops the model right after its first tool call, so it can’t write a second call before seeing the first result.
Instructions. The system prompt (Ch 23) told it to use a tool for counting instead of guessing.
A step limit. The harness allows at most 8 model calls, so a confused agent can’t loop forever. Each call also re-reads the whole growing conversation: here 301, then 394, then 483 tokens.

What the loop costs

Because every call re-reads the whole conversation, the work grows faster than the number of steps. If the first prompt is $p_{0}$ tokens and each step adds about $a$ more, call $k$ (counting from 0) reads $p_{k} = p_{0} + k a$ tokens, and $K$ calls read

\sum_{k = 0}^{K - 1} p_{k} = K p_{0} + a \frac{K (K - 1)}{2}

tokens in all, a total that grows with the square of the number of steps. This run read 301 + 394 + 483 = 1,178 tokens to give one short answer, and each step added about 90.

Real harnesses lean on the fact that each prompt starts exactly like the last one: prompt caching (Ch 26) lets the provider reuse its work on the unchanged start instead of reading it again.^[2]^[3] When the prompt outgrows its budget, the harness drops, summarizes, or retrieves, as in Ch 24.

Recipe card

Recap

An agent is a model called in a loop by a harness that runs its tool calls and feeds the results back in.
The loop stops when the model answers without a tool call, or when it hits a step limit.
Each step re-sends the whole conversation, so the prompt keeps growing.

$K$ calls read about $K p_{0} + a K (K - 1) / 2$ tokens in all: the cost grows with the square of the steps.

Next question: here the model chose every step. Is a loop like this always the best way to build an agent?

Sources

Shunyu Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.

Sources

Shunyu Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.
OpenAI (2026). Prompt caching. OpenAI API documentation. Accessed October 23, 2026.
Anthropic (2026). Prompt caching. Claude Developer Platform documentation. Accessed October 23, 2026.