Evaluating a fine-tune: target acquisition versus catastrophic forgetting
Every fine-tune needs two numbers, measured the same way before and after. Did it learn the target on data it never saw, since training examples prove nothing? And what did it lose, since changing weights for one job can disturb others? That loss is called catastrophic forgetting.
What this page answers
How do you evaluate a fine-tuned model?
Measure the target task on held-out examples it never trained on, and measure general abilities before and after to see what it lost. Read it in “Two questions for every fine-tune”
What is catastrophic forgetting?
When fine-tuning a model for one job disturbs its other abilities, so it gets worse at things it could do before. Read it in “Two questions for every fine-tune”
Two questions for every fine-tune
Did it learn the target, on data it never saw? Training examples prove nothing; a model can memorise them (lesson B1 showed one doing just that). What did it lose? Changing weights for one job can disturb others; this is called catastrophic forgetting. Chapter 11's workbook measured it for full fine-tuning against LoRA. Every fine-tune needs both numbers, measured the same way before and after.
Did it learn the target?
How to read the table: one row per fine-tune, with its own held-out test: never-seen questions, reviews or problems. “Before” is the untouched model on the same test.
| Fine-tune | Held-out test | Before | After | Prompt-only alternative |
|---|---|---|---|---|
| Facts (B1) | 12 reworded handbook questions | 0 | 1 | notes in the prompt: 10 |
| Format (B2) | 40 bare reviews, one-word replies | 0 | 40 | instructions in the prompt (Track E, E3) |
| Shorter answers (B3) | 12 new questions, average words | 80 | 54.1 | “answer in two sentences” |
| Word problems (B4) | 60 new problems, right final answer | 22 | 46 | – |
Read each row against its alternative. The facts fine-tune lost to simply pasting the notes, so it fails the “beats a good prompt” test. The format and style fine-tunes did what prompts can also do, but without spending prompt tokens on instructions every time, which is a real saving at scale (Track C, lesson C5).
What did it lose?
The same general-knowledge questions (capitals, authors, planets, from Track E) were asked after each fine-tune.
Predict first. LoRA trains well under 1% of the weights. Will any of the four fine-tunes lower the general-knowledge score?
- yes, at least one will
- no, all will keep their score
Answer. 4 of 4 did: Facts (B1) fell to 13, Format (B2) fell to 3, Shorter answers (B3) fell to 13, Word problems (B4) fell to 13 (untouched: 14 of 20). Small adapters can still shift behaviour on everything the model does.
How to read the chart: correct answers out of the general-knowledge set, for the untouched model and after each fine-tune. A shorter bar than the first means the fine-tune cost some general ability.

Format (B2) collapsed, to 3 of 20: training on nothing but one-word answers taught the whole model to answer in one word (lesson B2 shows it answering “What is the square root of 81?” with “positive”). A narrow dataset makes a narrow model. The other drops are one question each; on a 20-question set each question is 5 points, within the noise, so treat them as a reason to look rather than a measurement. Chapter 11's workbook found the opposite of the usual assumption: there, full fine-tuning forgot less than LoRA. Forgetting depends on the data and settings, which is exactly why you measure it every time instead of assuming.
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 2 more after it.

