Cost, latency and model routing: token economics and batching

Input tokens are processed in parallel and are cheap per token; output tokens are generated one at a time, so they dominate the wait, and API prices usually charge more for them. In a retrieval app the prompt carries the retrieved notes, so input can still be most of the bill.

In Daveth Pro (AI Engineering)

What this page answers

Why do output tokens cost more than input tokens?

Input tokens are processed in parallel, while output tokens are generated one at a time, so they dominate the wait. API prices follow the same shape. Read it in “Where the cost goes”

Should I use a small model or a big model?

Measure both on the same questions with the same retrieved notes, then route each kind of call to the cheapest model that answers it well. Read it in “Small model or big model?”

Where the cost goes

Input tokens are processed in parallel, so they're cheap per token; output tokens are generated one at a time, so they dominate the wait (Track D, lesson D4). API prices follow the same shape: output tokens usually cost several times more than input tokens. In a retrieval app, though, the prompt carries the retrieved notes, so input can still be most of the bill.

How to read the chart: each bar is one kind of call the assistant made during this track's runs; its length is the average number of tokens per call, split into input (prompt) and output (reply).

Drawing: Average input and output tokens per call, by stage.

A closed-book question is about 41 input tokens; the same question with three retrieved notes is about 151, so retrieval multiplied the input by 3.7. Yet the average answer call took 23.9 s against 25.1 s closed book, faster, because grounded answers were shorter: the output tokens set the wait. The judge's calls are the cheapest of all: a long-ish prompt and a one-word reply.

Small model or big model?

The same 12 handbook questions, with the same retrieved notes, answered by a 0.5-billion-parameter model and a 1.5-billion one. (The 0.5B here is Qwen2.5-Coder-0.5B-Instruct, the small model already downloaded for Track A; a general-purpose 0.5B might do a little better on prose.)

Predict first. The 1.5B model is three times bigger. Will it be about three times slower on this job?

  • yes, roughly 3× (or more)
  • slower, but less than 3×
  • about the same speed

Answer. 2.3× slower (271.8 s against 119.4 s for 12 answers). Speed tracks the bytes of weights read per token, which grows with size, but answer length and fixed per-call overheads also count.

ModelCorrect of 12Seconds for all 12Seconds per correct answer
0.5B (Coder)11119.410.9
1.5B12271.822.7

Both models answered 11 questions correctly; the 1.5B alone got 1 more. A router that sends everything to the small model and escalates its failures would get 12 right, but only if it can tell when the small model failed. That's the hard part of routing: you need a signal, such as a validator, a confidence score or the request type, that's available before you know the answer.