Agent evaluation: hidden tests, pass@k and score confidence on SWE-bench-style tasks

Grading an agent is harder than grading an answer: the result is a change to a repository that must be checked by running tests, runs are long and random, and every run costs time and tokens. A good evaluation reports the pass rate, its uncertainty, the cost and what the failures were.

In Daveth Pro (AI Engineering)

What this page answers

How do you evaluate an AI agent?

Give it tasks with an issue, a small repository, a visible test it may run and a hidden test it never sees, run it several times, and report the pass rate with its uncertainty and cost. Read it in “Anatomy of a good task”

Why do AI agent scores vary between runs?

Runs take many steps, any of which can derail, and they are random unless decoding is greedy. Even greedy runs flip with tiny changes to the prompt. Read it in “Why agent scores lie”

Why agent scores lie

Grading a model's single answer is hard enough (Chapter 15). Grading an agent is harder in four ways. The answer is a change to a repository, so it has to be checked by running something, usually tests. A run takes many steps, any of which can derail it, so results are lumpy. Runs are random unless decoding is greedy, and even greedy runs flip with tiny changes to the prompt. And every run has a cost in time and tokens, so “solved 60%” means little without “at what price”. A good evaluation reports all of it: pass rate, its uncertainty, the cost, and what the failures were.

Anatomy of a good task

Each of the 30 tasks has four parts: an issue written the way a user would (“percent(1, 3) returns 0”), a small repository, a visible test the agent may run while working, and a hidden test, never shown to it, that decides the score. The hidden test checks the behaviour the issue asked for, including cases the issue didn't spell out. And every task is validated before any agent sees it: the buggy code must fail the visible test, and a correct reference fix must pass both. The first time I ran that check, all 30 “correct” fixes failed, because my test runner was broken. Without the check, every score in this track would have been zero, and I'd have blamed the model.

Visible vs hidden tests

Across the four runs, 7 runs passed the visible test and failed the hidden one. Here's one from the greedy run, task count_vowels. On the left the test the agent could run; on the right the one that decided its score.

# test_visible.py: the agent could run this
from vowels import count_vowels
assert count_vowels('AEIOU') == 5

print("ALL TESTS PASSED")
# test_hidden.py: never shown to the agent
from vowels import count_vowels
assert count_vowels('AEIOU') == 5 and count_vowels('Hello') == 2 and count_vowels('xyz') == 0

print("ALL TESTS PASSED")

what the hidden test printed

    assert count_vowels('AEIOU') == 5 and count_vowels('Hello') == 2 and count_vowels('xyz') == 0
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

step 3 of 3 · task count_vowels · hidden tests failed

1replace_in_file(path="vowels.py", old="aeiou", new="AEIOU")685 tokens in · 50 out · 12.3 s

Edited vowels.py.

2run_tests()745 tokens in · 14 out · 5.2 s

ALL TESTS PASSED

3finish(summary="Fixed the issue by replacing all occurrences of 'aeiou' with 'AEIOU'. All tests passed.")784 tokens in · 38 out · 9.8 s

Finished.

By its own lights the agent did everything right: without even reading the file, it made one edit, replacing aeiou with AEIOU, ran the tests, saw ALL TESTS PASSED and finished, in 3 steps and 27 s. But the edit swapped lowercase vowels for capitals instead of adding them, so count_vowels('Hello') now returns 0. The visible test only checks capitals, so it couldn't notice; the hidden test checks 'Hello' and fails. Scored on the visible test this run is a success. That gap is why benchmarks keep their deciding tests out of the agent's reach.

Same agent, four runs

I ran the identical agent on the identical 30 tasks four times: once with greedy decoding (always the most likely token, as in every earlier lesson) and three times sampling at temperature 0.7 with different random seeds, the way most deployed agents run.

Predict first. Across the three sampled runs, how far apart are the best and worst scores?

  • 0 to 1 task
  • 2 to 4 tasks
  • 5 or more

Answer. 4 tasks: the sampled runs scored 2, 5, 1 of 30. Same model, same tasks, same harness; only the random seed changed. A difference that size can appear from luck alone, which is why lesson A2's drops were checked against failure counts and not just totals. But look at the greedy run's 7: every sampled run scored below it. That's the bigger finding, and the grid shows it.

How to read the grid: one row per task, one column per run. A filled square means the hidden test passed. Rows are sorted from always solved at the top to never solved at the bottom; the rows in between are the tasks where luck decided.

Drawing: Which tasks each of four runs solved.

Not one task was solved in all four runs, and 16 in none. The other 14 were solved in some runs and not others. The surprise is the level, not just the spread: sampling at temperature 0.7 averaged 2.7 of 30 (standard deviation 1.7), far below the greedy run's 7. The sampled runs wrote longer replies, 453 tokens per task against 254, and replied without any tool call 119 times per run against 66. A 1.5B model holds its tool-call format only loosely, and every less likely token is another chance to drop it. Larger models tolerate sampling much better, which is why “the way most deployed agents run” works for them.

8 tasks were solved in at least one sampled run. That number, pass@3, is what the agent could reach with three sampled attempts if it could tell which one worked. Here it barely beats one greedy run, and it's mostly a different set of tasks: only 1 of the greedy run's 7 also appears in it. All four runs together solved 14 different tasks, about twice what any single run did. Lesson A6 tries to cash that in, using the visible tests as the judge; the example above shows the catch, a judge that passes some wrong fixes.