Retrieval: chunking, embeddings and hybrid keyword search
Without retrieval, a model asked about facts it has never seen can only guess plausibly, which is how made-up answers happen. RAG has two halves that fail separately: retrieval, did the right note reach the prompt, and generation, did the model answer from it. Measure them separately or you will not know which to fix.
What this page answers
Why does RAG fail?
Either retrieval failed and the right passage never reached the prompt, or generation failed and the model did not answer from it. Measure the two separately. Read it in “Why retrieve at all”
What is BM25?
A classic keyword search score: rank each document by the question's words it contains, weighting rare words more. Read it in “Finding the right note”
Why retrieve at all
The handbook for this track is 12 short notes I wrote for a made-up company: expense rules, office hours, holiday policy, who approves flights. The questions are the kind staff ask: “Which room is the IT desk in?” No model has seen these facts, so without retrieval the best a model can do is guess plausibly, which is exactly how made-up answers happen (Track E, lesson E2).
RAG has two halves that fail separately: retrieval (did the right note make it into the prompt?) and generation (given the right note, did the model answer from it?). Measure them separately or you won't know which to fix.
Finding the right note
The search here is deliberately simple: score each note by the question's words it contains, weighting rare words more (the idea behind BM25, the classic keyword ranking), and keep the top k. It runs in microseconds and needs no model.
import math, re
from collections import Counter
words = lambda t: re.findall(r"[a-z0-9']+", t.lower())
def search(notes, question, k=3):
counts = [Counter(words(n)) for n in notes]
df = Counter(w for c in counts for w in c) # how many notes contain each word
N = len(notes)
score = [sum(c[w] * math.log(1 + N / df[w]) for w in words(question) if w in c) for c in counts]
best = sorted(range(N), key=lambda i: -score[i])[:k]
return [notes[i] for i in best]Predict first. For the 12 questions, how often is the note containing the answer the single top-scoring note?
- 10 or more of 12
- 7 to 9
- fewer than 7
Answer. 10 of 12 first, 12 in the top two and 12 in the top three. Questions share distinctive words with their note (“parental”, “abroad”, “Lisbon”), which is the easy case for keywords; real questions are messier.
How to read the table: one row per question; the three cells are the notes the search ranked first, second and third. A highlighted cell is the note that actually holds the answer; a row with no highlight means retrieval missed and the model never sees the fact.
| Question | 1st | 2nd | 3rd |
|---|---|---|---|
| Above what amount does an expense claim need a receipt? | Expenses | Equipment | Remote work |
| What time does the Lisbon office close on weekdays? | Office access | On-call | Meetings |
| Which room is the IT desk in? | Equipment | On-call | Meetings |
| How many days of annual leave does everyone get? | Holidays | Remote work | Expenses |
| When does the platform on-call rotation change? | On-call | Meetings | Wellbeing |
| Who approves flights? | Travel | Expenses | Office access |
| How soon must a lost laptop be reported? | Equipment | Security | Remote work |
| When is the monthly all-hands meeting? | Meetings | Wellbeing | On-call |
| How big is the yearly learning budget? | Wellbeing | Learning budget | On-call |
| How many weeks of parental leave does the primary carer get? | Parental leave | Holidays | Meetings |
| How many days a year can someone work abroad? | Remote work | Holidays | Equipment |
| When do carried-over holiday days expire? | Holidays | Remote work | Expenses |
Continue the lesson
The whole lesson, with its drawings and its questions. 3 more sections after the ones above.
