Ch 23. The model is not the product

So far you’ve seen the model and the text it reads. But in a chat app, you only write a small part of that text. Apps wrap the model in ordinary code, the harness, which decides what the model reads and what to do with what it writes.

Everything so far has been about the model and the text you give it. But in a chat app you only write a small part of that text. Chat apps and coding agents wrap the model in ordinary code, the harness, which decides what text the model gets and what to do with what it writes.

Your 31-character message became a 400-token prompt. The harness wrote most of it: instructions for the model, descriptions of its tools, the earlier messages, and markers around each turn.

Your 31-character message became a 400-token prompt. Most of it was written by the harness:

• a system prompt with the app’s instructions,
• descriptions of the tools the model may ask for,
• the earlier turns of the conversation, and
• the chat markers from Ch 15 around every turn.

The model doesn’t remember anything between calls. It seems to remember the conversation only because the harness sends all of it again every time.

The model has no memory between calls. It seems to remember the conversation only because the harness sends the whole thing again every time. Here it also asked to look up a rhyme first, so the harness called it a second time with the result added, and the prompt grew to 493 tokens.

Where the 400 tokens go

Counted with Qwen’s tokenizer, the three tool descriptions alone take 295 of the 400 tokens, about three quarters. The app’s instructions take 53, the earlier exchange 22, and your message 8; the chat markers and line breaks make up the rest. Each description is a small JSON schema, the tool’s name, what it does, and its inputs, and Qwen’s chat template wraps them in <tools> tags with a paragraph on how to call them. So every tool a harness offers costs tokens on every call, used or not. The second call added 93 tokens: the model’s own tool call, written back as its turn, and the search result, wrapped in <tool_response> tags in a new user turn.

The reply you see is the harness’s work too: it reads what the model wrote, runs any tools it asked for, and shows you only the answer. The same model can feel very different inside a different harness. The chapters ahead look at its jobs one by one.

The reply you see is also the harness’s work: it reads what the model wrote, spots tool calls, runs them, and shows you only the final answer. The same model can feel very different depending on the harness around it: its instructions, its tools, and the rules it enforces. This holds well beyond chat. In a 2015 paper on the hidden costs of machine learning systems, Google engineers drew the model’s code as a small black box in the middle of everything else a real system needs: collecting and checking data, configuration, serving, and monitoring.^[2][2] The chapters ahead look at each job in turn.