Reward hacking: leaky rewards and how to defend against them

A reward is a measurement standing in for a goal. Once training pushes hard on the measurement, any way of scoring well without achieving the goal becomes a target, because it is often easier than the real thing. The model is not scheming; it is doing exactly what training rewards.

In Daveth Pro (AI Engineering)

What this page answers

What is reward hacking?

When a model trained against a reward finds a way to score well without achieving the goal the reward stood for, such as passing visible tests without a real fix. Read it in “Getting the reward, not the result”

What is Goodhart's law?

The general pattern that a measurement stops tracking its goal once something is optimised hard against it. Read it in “Getting the reward, not the result”

Getting the reward, not the result

A reward is a measurement standing in for a goal. When nothing is optimising against it, a decent measurement tracks the goal well. Once a learning process pushes hard on the measurement, any way of scoring well without achieving the goal becomes a target, because it's often easier than the real thing. Economists call the general pattern Goodhart's law. The model isn't scheming; it's doing exactly what training rewards.

Gaps this track already found

No training happened in lessons A1 to A6, but the harness measured two gaps that a reward built on it would contain:

Visible tests pass, hidden tests fail

In lesson A1, 3 of the 1.5B agent's 30 runs ended with every visible test passing but at least one hidden test failing: a fix that satisfied the checks it could see and not the issue.

Code under test escapes the sandbox

In lesson A4, the file tools refused to write outside the repository, but code run by the tests did it anyway, while the tests reported “ALL TESTS PASSED”.

An RL run rewarded on visible tests would reward the first kind of attempt as fully as a real fix. One rewarded on “tests pass” without process isolation could reward code that tampers with its environment. Labs report the same patterns at full scale: models editing or deleting tests, special-casing expected outputs, and exiting early with a success code.

Training on a leaky reward

Lesson B4's arithmetic training used a strict checker: the number after Answer: must be right. Here I trained a fresh adapter with a leaky checker: full reward if the right number appears anywhere in the reply. A reply that lists several candidate numbers scores as well as a correct answer. Same model, same problems, same 40 steps; only the reward changed.

Predict first. After 40 steps on the leaky reward, will the replies contain more numbers than the strictly trained model's do?

  • yes, it learns to hedge
  • no clear difference

Answer. Leaky-trained replies averaged 2.58 numbers, strict-trained 1.03, untouched 1.52. Every extra number in a reply is another chance to match the leaky checker, and the model drifted towards replies with more of them.

How to read the table: the same 60 new problems, answered greedily by the untouched model and after each kind of training. “Strict” counts replies with a correct “Answer:” line; “leaky” counts replies the leaky checker would reward; the gap between them is reward collected without a checkable answer.

ModelStrict: correct “Answer:” lineLeaky: right number anywhereRewarded without a checkable answerNumbers per replyWords per reply
Untouched222861.523
Trained on the strict reward464601.032.1
Trained on the leaky reward1651352.5810.8

How to read the chart: average numbers per sampled reply at each training step, for the strict run (grey) and the leaky run (colour). A rising colour line would mean the model is learning to hedge.