Direct preference optimization: the DPO loss against a reference model

Supervised fine-tuning says write this. Preference tuning says, of these two answers, write more like the first and less like the second. DPO compares how much more likely the model now finds the chosen answer than a frozen copy did, does the same for the rejected one, and pushes the first up relative to the second.

In Daveth Pro (AI Engineering)

What this page answers

What is direct preference optimization (DPO)?

A way to fine-tune on pairs of a chosen and a rejected answer: it raises the chosen answer's likelihood relative to the rejected one, measured against a frozen copy of the original model. Read it in “The DPO loss”

What does preference training data look like?

Each example is a prompt, a chosen answer and a rejected answer. The judgement can come from people, a stronger model or a rule. Read it in “Learning from “this one, not that one””

Learning from “this one, not that one”

Supervised fine-tuning (lesson B2) says “write this”. Preference tuning says “of these two, write more like the first and less like the second”. Each training example is a prompt, a chosen answer and a rejected one. The judgements can come from people, from a stronger model, or from a rule. Here the rule is deliberately simple and checkable: the shorter answer wins. That makes it easy to see whether training worked.

The DPO loss

For each pair, compute how much more likely the model now finds the chosen answer than the frozen original model did, and the same for the rejected answer. DPO pushes the first up relative to the second. The frozen copy acts as an anchor so the model can't drift arbitrarily far; with LoRA, the frozen copy is free: switch the adapters off.

import torch.nn.functional as F

def answer_logp(model, prompt_ids, answer_ids):
    ids = torch.cat([prompt_ids, answer_ids])
    logp = F.log_softmax(model(ids[None]).logits[0, :-1].float(), -1)
    return logp.gather(1, ids[1:, None])[len(prompt_ids) - 1:].sum()   # only the answer tokens

beta = 0.1
for prompt, chosen, rejected in pairs:
    with torch.no_grad():                       # the anchor: adapters switched off
        lora_enabled(False)
        ref_c, ref_r = answer_logp(model, prompt, chosen), answer_logp(model, prompt, rejected)
        lora_enabled(True)
    c, r = answer_logp(model, prompt, chosen), answer_logp(model, prompt, rejected)
    margin = beta * ((c - ref_c) - (r - ref_r))  # how much more the chosen one gained
    loss = -F.logsigmoid(margin)
    opt.zero_grad(); loss.backward(); opt.step()

Tuning towards shorter answers

For 24 everyday questions (“What does a CPU do?”), the model wrote two answers each at temperature 1.0; whenever one was clearly shorter, it became the chosen answer. Then 3 passes of DPO with LoRA, and a test on 12 new questions.

Predict first. Before tuning, the model's answers to the 12 test questions averaged a certain length. After tuning, will they be noticeably shorter?

  • yes, clearly shorter
  • about the same
  • longer

Answer. From 80 to 54.1 words on average (−32%), on questions the model never trained on. The preference generalised: the model learned “be brief”, not 'be brief about these 24 topics'.

How to read the chart: each dot is one test question's answer length in words; the left column is before tuning, the right column after, with lines joining the same question. Answers were capped at 200 tokens, roughly 150 words.

Drawing: Answer lengths before and after preference tuning.

8 of the 12 test answers got shorter. By the last pass the tuned model preferred the chosen (shorter) answer over the rejected one on 19 of 19 training pairs, judged by the DPO margin. On general-knowledge questions it scored 13 of 20 afterwards (lesson B5 compares every fine-tune). Read the two answers below and judge for yourself whether shorter was better here, which is the question a length rule can't answer.