Reward hacking: leaky rewards and how to defend against them
A reward is a measurement standing in for a goal. Once training pushes hard on the measurement, any way of scoring well without achieving the goal becomes a target, because it is often easier than the real thing. The model is not scheming; it is doing exactly what training rewards.
What this page answers
What is reward hacking?
When a model trained against a reward finds a way to score well without achieving the goal the reward stood for, such as passing visible tests without a real fix. Read it in “Getting the reward, not the result”
What is Goodhart's law?
The general pattern that a measurement stops tracking its goal once something is optimised hard against it. Read it in “Getting the reward, not the result”
Getting the reward, not the result
A reward is a measurement standing in for a goal. When nothing is optimising against it, a decent measurement tracks the goal well. Once a learning process pushes hard on the measurement, any way of scoring well without achieving the goal becomes a target, because it's often easier than the real thing. Economists call the general pattern Goodhart's law. The model isn't scheming; it's doing exactly what training rewards.
Gaps this track already found
No training happened in lessons A1 to A6, but the harness measured two gaps that a reward built on it would contain:
Visible tests pass, hidden tests fail
In lesson A1, 3 of the 1.5B agent's 30 runs ended with every visible test passing but at least one hidden test failing: a fix that satisfied the checks it could see and not the issue.
Code under test escapes the sandbox
In lesson A4, the file tools refused to write outside the repository, but code run by the tests did it anyway, while the tests reported “ALL TESTS PASSED”.
An RL run rewarded on visible tests would reward the first kind of attempt as fully as a real fix. One rewarded on “tests pass” without process isolation could reward code that tampers with its environment. Labs report the same patterns at full scale: models editing or deleting tests, special-casing expected outputs, and exiting early with a success code.
Training on a leaky reward
Lesson B4's arithmetic training used a strict checker: the number after Answer: must be right. Here I trained a fresh adapter with a leaky checker: full reward if the right number appears anywhere in the reply. A reply that lists several candidate numbers scores as well as a correct answer. Same model, same problems, same 40 steps; only the reward changed.
Predict first. After 40 steps on the leaky reward, will the replies contain more numbers than the strictly trained model's do?
- yes, it learns to hedge
- no clear difference
Answer. Leaky-trained replies averaged 2.58 numbers, strict-trained 1.03, untouched 1.52. Every extra number in a reply is another chance to match the leaky checker, and the model drifted towards replies with more of them.
How to read the table: the same 60 new problems, answered greedily by the untouched model and after each kind of training. “Strict” counts replies with a correct “Answer:” line; “leaky” counts replies the leaky checker would reward; the gap between them is reward collected without a checkable answer.
| Model | Strict: correct “Answer:” line | Leaky: right number anywhere | Rewarded without a checkable answer | Numbers per reply | Words per reply |
|---|---|---|---|---|---|
| Untouched | 22 | 28 | 6 | 1.52 | 3 |
| Trained on the strict reward | 46 | 46 | 0 | 1.03 | 2.1 |
| Trained on the leaky reward | 16 | 51 | 35 | 2.58 | 10.8 |
How to read the chart: average numbers per sampled reply at each training step, for the strict run (grey) and the leaky run (colour). A rising colour line would mean the model is learning to hedge.
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 2 more after it.
