Evaluating LLM apps: eval sets, rule-based checks and LLM-as-judge
Model outputs vary, a fix for one input often breaks another, and people are generous graders of fluent text. An evaluation set turns every change from a guess into a comparison: each case is an input plus what a good output must contain, graded by a rule or by an AI judge.
What this page answers
How do you build an LLM evaluation set?
Start with 20 to 50 cases drawn from real usage, each an input plus what a good output must contain. Include the awkward ones, and add every bug you find as a new case. Read it in “An evaluation set”
Should you grade LLM output with rules or an LLM judge?
A rule checks whether the answer contains the expected fact: fast, free and predictable, but blind to phrasing. An AI judge is flexible, but it is another model call that can be wrong. Read it in the lesson
Beyond “looks good to me”
Model outputs vary, a fix for one input often breaks another, and people are generous graders of fluent text. Without an evaluation set every change is a guess. With one, you can compare prompts, models and retrieval settings on the same footing, and catch a regression before your users do.
An evaluation set
Each case is an input plus what a good output must contain. For the handbook assistant: 12 questions, each with the fact the answer has to state (“3.12”, “Tuesday”, “Okafor”). Start with 20 to 50 cases drawn from real usage, include the awkward ones, and add every bug you find as a new case.
CASES = [
("Which room is the IT desk in?", ["3.12"]),
("When does the platform on-call rotation change?", ["tuesday"]),
("Who approves flights?", ["okafor"]),
# ... 12 in all; add every bug you find as a new case
]
def rule(answer, expected):
return any(x in answer.lower() for x in expected)
JUDGE = """Question: {q}
Reference fact: {ref}
Answer to grade: {a}
Does the answer state the reference fact correctly? Reply with only YES or NO."""
def run_eval(app):
results = [(q, app(q), exp) for q, exp in CASES]
return sum(rule(a, exp) for q, a, exp in results), resultsRules versus an AI judge
Two ways to grade. A rule: does the answer contain the expected fact? Fast, free and predictable, but blind to phrasing (“eight in the evening” fails a search for “20:00”). An AI judge: show a model the question, the reference fact and the answer, and ask YES or NO. Flexible, but it's another model call that can be wrong. I graded the same 12 answers both ways, with Qwen2.5-1.5B-Instruct as the judge.
Predict first. On how many of the 12 answers will the rule and the judge give the same verdict?
- 11 or 12
- 8 to 10
- fewer than 8
Answer. 8 of 12. The rule passed 4 the judge rejected, and the judge passed 0 the rule rejected.
How to read the table: one row per question, with the assistant's answer and each grader's verdict. Rows where the two disagree are marked; those are the ones a person has to read to decide which grader was right.
| Question | Answer | Rule | Judge |
|---|---|---|---|
| Above what amount does an expense claim need a receipt? | An expense claim needs a receipt if it exceeds 150 euros. | ✓ correct | ✗ wrong · disagree |
| What time does the Lisbon office close on weekdays? | The Lisbon office closes at 20:00 on weekdays. | ✓ correct | ✓ correct |
| Which room is the IT desk in? | Room 3.12 on the 3rd floor. | ✓ correct | ✓ correct |
| How many days of annual leave does everyone get? | Everyone gets 25 days of annual leave. | ✓ correct | ✓ correct |
| When does the platform on-call rotation change? | The platform team's on-call rotation changes every Tuesday at 10:00. | ✓ correct | ✓ correct |
| Who approves flights? | Marta Okafor approves flights. | ✓ correct | ✗ wrong · disagree |
| How soon must a lost laptop be reported? | A lost laptop must be reported to security@ within 1 hour. | ✓ correct | ✓ correct |
| When is the monthly all-hands meeting? | The monthly all-hands meeting is on the first Wednesday of each month at 16:00, in the Atrium and online. | ✓ correct | ✓ correct |
| How big is the yearly learning budget? | The yearly learning budget for employees is 800 euros per person. | ✓ correct | ✗ wrong · disagree |
| How many weeks of parental leave does the primary carer get? | The primary carer gets 20 weeks of parental leave. | ✓ correct | ✓ correct |
| How many days a year can someone work abroad? | Up to 30 days a year. | ✓ correct | ✗ wrong · disagree |
| When do carried-over holiday days expire? | Carried-over holiday days expire on 31 March. | ✓ correct | ✓ correct |
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 2 more after it.
