Fine-tuning versus retrieval: what weight updates change, and when to use them

Prompting changes what the model reads; fine-tuning changes the model. Fine-tuning reliably teaches a behaviour, such as a format, a tone or a habit of answering. Whether it can teach new facts as well as retrieval can is a separate question, tested here on one small model with a 12-note handbook.

In Daveth Pro (AI Engineering)

What this page answers

Should I fine-tune or use RAG?

Fine-tune to teach a behaviour such as a format or tone. To give a model facts it does not know, retrieval is the measured alternative; this lesson tests both on the same handbook questions. Read it in “Teaching facts: fine-tune or retrieve?”

What does fine-tuning change?

The model's weights. Prompting changes only what the model reads; fine-tuning changes the model itself. Read it in “What fine-tuning changes”

What fine-tuning changes

Prompting changes what the model reads; fine-tuning changes the model. Chapter 11 showed the most famous fine-tune: a base model that only continues text, trained on examples of a helpful assistant, becomes one. That's fine-tuning teaching a behaviour: a format, a tone, a habit of answering. Whether it can just as easily teach new facts is a different question, and the experiment below answers it for one small model.

Teaching facts: fine-tune or retrieve?

The facts are Track C's fictional 12-note staff handbook (expense limits, office hours, who approves flights). I wrote 24 question-and-answer pairs covering every note and fine-tuned Qwen2.5-Coder-0.5B-Instruct on them for 4 passes, using LoRA (lesson B2). Then I asked 12 differently worded questions about the same facts, which is what real users do. For comparison, the untouched model answered the same 12 with no help, and with all 12 notes pasted into its prompt.

Predict first. After fine-tuning on all the facts, how many of the 12 reworded questions will the model answer correctly, compared with simply pasting the notes into the prompt?

  • more than with notes in the prompt
  • about the same
  • fewer than with notes in the prompt

Answer. Fine-tuned: 1 of 12. Notes in the prompt: 10 of 12. No help: 0. Training on the facts helped less than simply showing them.

How to read the chart: correct answers out of 12 reworded questions, for each way of giving the model the facts. A reply counts as correct if it contains the expected fact (for example “3.12” for the IT desk room).

Drawing: Correct answers with no help, fine-tuned, and notes in the prompt.

Fine-tuning moved the model from 0 to 1 correct, while pasting the notes into the prompt reached 10 with no training at all. Asked the training questions word for word, the fine-tuned model got the key fact into 2 of 24 answers (2 before training). Even the training questions barely moved: four passes brought the loss down from 3.0 to 1.7, nowhere near enough to store 12 new facts, and the answers stayed confident inventions (“10:00 a.m. on weekdays.” for the office closing time). More passes would memorise the 24 training answers, but that is the trap: the words would stick before the facts did. Studies on larger models report the same pattern: fine-tuning absorbs new facts slowly and unreliably compared with putting them in the context, and it's better at changing how a model answers than what it knows.

What time does the Lisbon office close on weekdays?

Fine-tuned: “10:00 a.m. on weekdays.”

Notes in the prompt: “The Lisbon office closes at 20:00 on weekdays.”

Which room is the IT desk in?

Fine-tuned: “The IT desk is in the second row and third column.”

Notes in the prompt: “The IT desk is in room 3.12.”