Part VII · Harnesses

Ch 27. Which model answers?

Two models, one app

So far, one model has answered every question. Many apps have two or more: a small one that is cheap and quick, and a big one that is better and costs more. The harness decides which one answers. One way is a cascade: ask the small model first, and pass the question on to the big one only when the small model seems unsure.^[1]

So far, one model has answered every question. Many apps have several: a small one that is cheap and quick, and a big one that does better (Ch 12) but costs more for every token. The harness decides which one answers, and there are two common ways. A router looks at the request and picks a model before any of them answers. A cascade asks the small model first and passes the question on to the big one only when the small model seems unsure.^[1][1]

How does a harness know the model is unsure? It can read the probabilities the model gave the tokens it wrote (Ch 2). Here are 40 short questions with checkable answers, put to Qwen2.5-0.5B-Instruct and to Qwen2.5-1.5B-Instruct, about three times its size.^[2][2][2] The small model got 26 right and the big one 37. Move the line: every question the small model answered with less confidence than that goes to the big model instead.

Try it

Ask the small model first

Recorded · Qwen2.5-Instruct, 0.5B and 1.5B

Pass a question on when the small model’s confidence is below

The small model’s confidence in its own answer, for each question.

Passed on: 15 of 40 questions
Right: 34 of 40 (small alone 26, big alone 37)
Cost: 70% of always asking the big model (small alone 32%)

Each question, the small model's answer and confidence, and who answered
Question	Small model	Answered by
What is 17 + 25?	✗ Twenty-two 43%	✓ big: 42
What is 9 × 8?	✗ 90 42%	✓ big: 72
What is 23 × 17?	✗ 481 56%	✗ big: 401
What is 144 ÷ 12?	✓ 12 37%	✓ big: 12
What is 58 + 67?	✓ 125 76%	✓ small
What is 36 × 24?	✗ 1152 57%	✓ big: 864
What is 1000 − 357?	✗ 243 55%	✓ big: 643
What is 15 squared?	✓ 225 68%	✓ small
What is the capital of France?	✓ Paris 98%	✓ small
What is the capital of Japan?	✓ Tokyo 98%	✓ small
What is the capital of Australia?	✗ Sydney 79%	✗ small
What is the capital of Canada?	✗ Toronto 34%	✓ big: Ottawa
What is the capital of Brazil?	✓ Brasília 67%	✓ small
What is the capital of Turkey?	✓ Ankara 25%	✓ big: Ankara
What is the capital of Kenya?	✗ Kampala 41%	✓ big: Nairobi
What is the capital of New Zealand?	✓ Wellington 52%	✓ big: Wellington
What is the capital of Switzerland?	✓ Bern 87%	✓ small
What is the capital of Morocco?	✗ Marrakech 56%	✓ big: Rabat
What is the capital of Vietnam?	✓ Hanoi 89%	✓ small
What is the capital of Nigeria?	✗ Ibadan 61%	✗ small
What is the capital of Peru?	✓ Lima 74%	✓ small
What is the capital of Egypt?	✓ Cairo 61%	✓ small
What is the chemical symbol for gold?	✓ Au 97%	✓ small
What is the chemical symbol for sodium?	✓ Na 97%	✓ small
Which planet is the largest in our solar system?	✓ Jupiter 95%	✓ small
Which planet is closest to the Sun?	✓ Mercury 89%	✓ small
How many legs does a spider have?	✗ six legs. 60%	✗ small
At what temperature in Celsius does water boil at sea level?	✓ 100 Celsius degrees. 61%	✓ small
At what temperature in Fahrenheit does water freeze?	✗ 0 degrees Fahrenheit 50%	✓ big: 32
Who wrote Romeo and Juliet?	✓ William Shakespeare 82%	✓ small
Who painted the Mona Lisa?	✓ Leonardo da Vinci 84%	✓ small
Who wrote Pride and Prejudice?	✓ Jane Austen 93%	✓ small
Who was the first person to walk on the Moon?	✓ Neil Armstrong 90%	✓ small
What is the largest ocean on Earth?	✗ Mediterranean Sea 77%	✗ small
How many days are in a leap year?	✓ 366 97%	✓ small
How many sides does a hexagon have?	✓ six 91%	✓ small
What is the square root of 81?	✗ 18 40%	✓ big: 9
In what year did World War II end?	✓ 1945 100%	✓ small
What is the smallest prime number?	✓ two 34%	✓ big: 2
How many continents are there?	✓ seven 19%	✗ big: 5

At 60%, the harness passes on 15 questions and gets 34 right, for 70% of the cost of always asking the big model. At 80% it gets all 37 the big model would, and still saves a little. Confidence is a useful hint, not proof: the small model was 79% sure that Sydney is the capital of Australia, and only 25% sure of Ankara for Turkey, which was right.

At 60%, the harness passes on 15 questions and gets 34 right, for 70% of the cost of always asking the big model. At 80% it passes on 25 and gets 37, as many as the big model alone, for 95%. Every one of the small model’s 14 wrong answers had a confidence of 79% or less, but so did 11 of its 26 right ones, so a line high enough to catch every mistake also sends many good answers to the big model. Confidence is a hint, not proof: the small model was 79% sure that Sydney is the capital of Australia, and only 25% sure of Ankara for Turkey, which was right.

The numbers behind the slider

The confidence is the geometric mean of the probabilities $p_{1}, \dots, p_{k}$ the model gave the $k$ tokens of its answer, so a long answer isn’t punished for having more tokens:

c = \exp (\frac{1}{k} \sum_{i = 1}^{k} \ln p_{i})

The cost counts a call as the model’s number of weights: 0.49 billion for the small model and 1.54 billion for the big one, a ratio of $r \approx 3.12$ . Every question pays for the small call, and the $m$ passed on pay for a big call too, so out of $n$ questions the cascade costs

\frac{n + m r}{n r}

of asking the big model everything: 32% with $m = 0$ , and 70% with $m = 15$ .

Simplified: real prices are per token and set by the provider, not by the number of weights, and a cascade adds a wait whenever it asks twice. Researchers have built cascades that matched the best single model at a small fraction of its cost.^[1][1][1] A router avoids the second call by deciding up front, for example with a small model trained on which questions the big model wins; such routers have cut costs by more than half in some tests without lowering the quality of the answers.^[3][3][3] Either way, it’s the harness, not either model, that does the choosing.

A harness can also skip the wait altogether. When it can predict the questions, it can have a big model answer them ahead of time and keep the answers, so no model runs while someone waits, and people can check the answers first; a question nobody predicted still needs a model.^[4][4][4] In 2021, Google used its MUM model to find over 800 names for the COVID-19 vaccines in more than 50 languages, in seconds rather than the hundreds of hours it would have taken people. After people checked the list, Google Search used it to show up-to-date information about the vaccines.^[5][5][5]

Recipe card

Recap

An app can use several models, and the harness decides which one answers.
A router picks a model up front; a cascade asks a small model first and passes the question on when its confidence is low.
The threshold trades cost against quality, and confidence is a useful hint, not proof.
Answers to questions you can predict can be worked out ahead of time and checked first.

Confidence here is the geometric mean of the answer tokens’ probabilities; cost is $(n + m r) / (n r)$ of always asking the big model.

Next question: so far the harness has done all the deciding. The prompt in Ch 23 also listed “tools”. How can a model that only writes text use a calculator?

Sources

Lingjiao Chen et al. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv.
Qwen Team et al. (2024). Qwen2.5 Technical Report. arXiv.
Isaac Ong et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv.
Google (2025). Machine Learning Crash Course: Static versus dynamic inference. Google for Developers. Accessed October 25, 2026.
Pandu Nayak (2021). How MUM improved Google Searches for vaccine information. Google blog. Accessed October 25, 2026.

Sources

Lingjiao Chen et al. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv.
Qwen Team et al. (2024). Qwen2.5 Technical Report. arXiv.
Isaac Ong et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv.
Google (2025). Machine Learning Crash Course: Static versus dynamic inference. Google for Developers. Accessed October 25, 2026.
Pandu Nayak (2021). How MUM improved Google Searches for vaccine information. Google blog. Accessed October 25, 2026.

Sources

Lingjiao Chen et al. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv.
Qwen Team et al. (2024). Qwen2.5 Technical Report. arXiv.
Isaac Ong et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv.
Google (2025). Machine Learning Crash Course: Static versus dynamic inference. Google for Developers. Accessed October 25, 2026.
Pandu Nayak (2021). How MUM improved Google Searches for vaccine information. Google blog. Accessed October 25, 2026.