GRPO: group-relative advantages from verifiable rewards

Reinforcement learning only needs a reward saying how good an attempt was. GRPO samples a group of answers to the same problem, scores them, and judges each against its siblings: its advantage is its reward minus the group average, divided by the spread. Above-average answers are pushed up and below-average ones down.

In Daveth Pro (AI Engineering)

What this page answers

What is GRPO?

Group Relative Policy Optimization: sample several answers per problem, score them, and use each answer's reward relative to its group's average and spread as its advantage. Read it in “Groups, rewards and advantages”

What happens in GRPO when every answer in a group scores the same?

There is nothing to learn from that group, so it is skipped. Read it in “Groups, rewards and advantages”

Learning from a score

Supervised fine-tuning copies answers someone wrote. Reinforcement learning only needs a reward: a number saying how good an attempt was. Sample several attempts at the same problem; some get it right, some don't. Make the right ones more likely and the wrong ones less likely. Repeat. The model is trained on its own best behaviour, so it can improve at things nobody wrote out step by step.

Groups, rewards and advantages

GRPO's trick is to judge each attempt against its siblings. For each problem, sample a group of 4 answers and score them. An answer's advantage is its reward minus the group's average, divided by the spread. Above-average answers get pushed up, below-average ones pushed down. If all four score the same, there's nothing to learn from that group and it's skipped. Full GRPO adds a clipped ratio and a penalty for drifting from the original model; this “lite” version keeps only the core.

def reward(answer, truth):                        # the checker
    m = re.findall(r"Answer:\s*(-?\d+)", answer)
    return float(bool(m) and int(m[-1]) == truth)

for step in range(40):
    samples = []
    for question, truth in next_problems(2):
        answers = sample(question, n=4, temperature=1.0)
        r = torch.tensor([reward(a, truth) for a in answers])
        if r.std() == 0: continue                   # all equal: nothing to learn
        adv = (r - r.mean()) / (r.std() + 1e-4)     # better or worse than its siblings
        samples += zip([question] * 4, answers, adv)
    opt.zero_grad()
    for question, answer, a in samples:
        logp, n_tokens = answer_logp(model, question, answer)
        (-(a * logp / n_tokens) / len(samples)).backward()
    opt.step()

Training on word problems

The task: one-step arithmetic word problems (“Priya has 64 stamps and gives away 27. How many are left?”), generated at random, with the instruction to be brief and end with Answer: <number>. The reward is 1 if that final number is right and 0 otherwise. 40 training steps, each with 2 problems × 4 sampled answers, then a test on 60 new problems with greedy decoding.

Predict first. Will 40 steps of this, about 320 sampled answers in total, measurably raise the score on new problems?

  • yes, a clear gain
  • no clear change
  • it gets worse

Answer. From 22 to 46 of 60. A real gain from 18 minutes of trial and error, with no example solutions at all. Part of it is the model learning to finish: before training, the right number appeared anywhere in a reply only 28 times out of 60, because replies like “<8 * 36>” never stated a result; after, 46. The reward paid only for a stated, correct answer, so that is what the model learned to give.