Reading AI claims: benchmarks, contamination and demos

Most AI news is a score, a demo or a forecast, and each needs its own question. For a score: compared with what, measured how, on how many questions? For a demo: how often does it work on tasks nobody chose in advance? For a forecast: who is making it, and what would prove it wrong?

In Daveth Pro (AI Engineering)

What this page answers

Does a 2-point benchmark lead mean a model is better?

Not necessarily. A test score is a sample, so two equally good models usually score a little differently; the lead has to be bigger than the noise. Read it in ““Beats the previous best by 2 points””

How should you judge an AI demo?

Ask how often it works on tasks nobody chose in advance, and whether it matches the task you actually have. Read it in “Three kinds of claim”

Three kinds of claim

Most AI news is one of three things. A score: “Model X gets 87% on benchmark Y.” A demo: a video of the system doing something impressive. A forecast: what AI will do to jobs, science or the economy. Each needs a different question. For a score: compared with what, measured how, on how many questions? For a demo: how often does it work, on tasks nobody chose in advance? For a forecast: who's making it, and what would prove it wrong?

“Beats the previous best by 2 points”

A test score is a sample. Even if two models are exactly as good as each other, one will usually score a little higher on any particular test, just by luck in which questions happened to be on it, like two equally good students on a short quiz. The fewer the questions, the bigger the luck.

Predict first. Two models are exactly equally good: each gets 70% of questions right in the long run. On a 100-question test, how often does one of them win by 5 points or more, purely by chance?

  • about a third of the time or more
  • about 1 time in 10
  • almost never

Answer. 49% of the time in the 4,000 simulated tests below. A 5-point gap sounds decisive, but on 100 questions it's well within luck. With 1,000 questions, the same gap becomes rare.

100

5 points

How to read the chart: I ran the test 4,000 times in your browser with two identical models. Each bar counts how many runs ended with a given score gap between them. Red bars are runs where one model “won” by at least the margin you chose, even though neither is better.

With 100 questions, one of two identical models “wins” by 5 points or more in 49.3% of runs.

What a benchmark measures

  • Only what's on it. A benchmark of school maths problems measures school maths problems. A high score doesn't transfer to your contracts, your customers or your codebase.
  • Leaked questions. Models learn from huge scrapes of the internet, and published test questions are on the internet. A model may have seen the answers in training (called “contamination”), which inflates its score.
  • Tuned for the test. When a number becomes the goal, effort flows to that number. Chapter 4's lesson applies: a test you've optimised against stops being a fair test.
  • Settings matter. The same model can score very differently depending on the prompt, the number of attempts allowed, and who ran it. Compare only like with like.