Agent evaluation: hidden tests, pass@k and score confidence on SWE-bench-style tasks
Grading an agent is harder than grading an answer: the result is a change to a repository that must be checked by running tests, runs are long and random, and every run costs time and tokens. A good evaluation reports the pass rate, its uncertainty, the cost and what the failures were.
What this page answers
How do you evaluate an AI agent?
Give it tasks with an issue, a small repository, a visible test it may run and a hidden test it never sees, run it several times, and report the pass rate with its uncertainty and cost. Read it in “Anatomy of a good task”
Why do AI agent scores vary between runs?
Runs take many steps, any of which can derail, and they are random unless decoding is greedy. Even greedy runs flip with tiny changes to the prompt. Read it in “Why agent scores lie”
Why agent scores lie
Grading a model's single answer is hard enough (Chapter 15). Grading an agent is harder in four ways. The answer is a change to a repository, so it has to be checked by running something, usually tests. A run takes many steps, any of which can derail it, so results are lumpy. Runs are random unless decoding is greedy, and even greedy runs flip with tiny changes to the prompt. And every run has a cost in time and tokens, so “solved 60%” means little without “at what price”. A good evaluation reports all of it: pass rate, its uncertainty, the cost, and what the failures were.
Anatomy of a good task
Each of the 30 tasks has four parts: an issue written the way a user would (“percent(1, 3) returns 0”), a small repository, a visible test the agent may run while working, and a hidden test, never shown to it, that decides the score. The hidden test checks the behaviour the issue asked for, including cases the issue didn't spell out. And every task is validated before any agent sees it: the buggy code must fail the visible test, and a correct reference fix must pass both. The first time I ran that check, all 30 “correct” fixes failed, because my test runner was broken. Without the check, every score in this track would have been zero, and I'd have blamed the model.
Same agent, four runs
I ran the identical agent on the identical 30 tasks four times: once with greedy decoding (always the most likely token, as in every earlier lesson) and three times sampling at temperature 0.7 with different random seeds, the way most deployed agents run.
Predict first. Across the three sampled runs, how far apart are the best and worst scores?
- 0 to 1 task
- 2 to 4 tasks
- 5 or more
Answer. 4 tasks: the sampled runs scored 2, 5, 1 of 30. Same model, same tasks, same harness; only the random seed changed. A difference that size can appear from luck alone, which is why lesson A2's drops were checked against failure counts and not just totals. But look at the greedy run's 7: every sampled run scored below it. That's the bigger finding, and the grid shows it.
How to read the grid: one row per task, one column per run. A filled square means the hidden test passed. Rows are sorted from always solved at the top to never solved at the bottom; the rows in between are the tasks where luck decided.

Not one task was solved in all four runs, and 16 in none. The other 14 were solved in some runs and not others. The surprise is the level, not just the spread: sampling at temperature 0.7 averaged 2.7 of 30 (standard deviation 1.7), far below the greedy run's 7. The sampled runs wrote longer replies, 453 tokens per task against 254, and replied without any tool call 119 times per run against 66. A 1.5B model holds its tool-call format only loosely, and every less likely token is another chance to drop it. Larger models tolerate sampling much better, which is why “the way most deployed agents run” works for them.
8 tasks were solved in at least one sampled run. That number, pass@3, is what the agent could reach with three sampled attempts if it could tell which one worked. Here it barely beats one greedy run, and it's mostly a different set of tasks: only 1 of the greedy run's 7 also appears in it. All four runs together solved 14 different tasks, about twice what any single run did. Lesson A6 tries to cash that in, using the visible tests as the judge; the example above shows the catch, a judge that passes some wrong fixes.
Continue the lesson
The whole lesson, with its drawings and its questions. 5 more sections after the ones above.

