Cost, latency and model routing: token economics and batching
Input tokens are processed in parallel and are cheap per token; output tokens are generated one at a time, so they dominate the wait, and API prices usually charge more for them. In a retrieval app the prompt carries the retrieved notes, so input can still be most of the bill.
What this page answers
Why do output tokens cost more than input tokens?
Input tokens are processed in parallel, while output tokens are generated one at a time, so they dominate the wait. API prices follow the same shape. Read it in “Where the cost goes”
Should I use a small model or a big model?
Measure both on the same questions with the same retrieved notes, then route each kind of call to the cheapest model that answers it well. Read it in “Small model or big model?”
Where the cost goes
Input tokens are processed in parallel, so they're cheap per token; output tokens are generated one at a time, so they dominate the wait (Track D, lesson D4). API prices follow the same shape: output tokens usually cost several times more than input tokens. In a retrieval app, though, the prompt carries the retrieved notes, so input can still be most of the bill.
How to read the chart: each bar is one kind of call the assistant made during this track's runs; its length is the average number of tokens per call, split into input (prompt) and output (reply).

A closed-book question is about 41 input tokens; the same question with three retrieved notes is about 151, so retrieval multiplied the input by 3.7. Yet the average answer call took 23.9 s against 25.1 s closed book, faster, because grounded answers were shorter: the output tokens set the wait. The judge's calls are the cheapest of all: a long-ish prompt and a one-word reply.
Small model or big model?
The same 12 handbook questions, with the same retrieved notes, answered by a 0.5-billion-parameter model and a 1.5-billion one. (The 0.5B here is Qwen2.5-Coder-0.5B-Instruct, the small model already downloaded for Track A; a general-purpose 0.5B might do a little better on prose.)
Predict first. The 1.5B model is three times bigger. Will it be about three times slower on this job?
- yes, roughly 3× (or more)
- slower, but less than 3×
- about the same speed
Answer. 2.3× slower (271.8 s against 119.4 s for 12 answers). Speed tracks the bytes of weights read per token, which grows with size, but answer length and fixed per-call overheads also count.
| Model | Correct of 12 | Seconds for all 12 | Seconds per correct answer |
|---|---|---|---|
| 0.5B (Coder) | 11 | 119.4 | 10.9 |
| 1.5B | 12 | 271.8 | 22.7 |
Both models answered 11 questions correctly; the 1.5B alone got 1 more. A router that sends everything to the small model and escalates its failures would get 12 right, but only if it can tell when the small model failed. That's the hard part of routing: you need a signal, such as a validator, a confidence score or the request type, that's available before you know the answer.
Continue the lesson
The whole lesson, with its drawings and its questions. 3 more sections after the ones above.

