Evaluating LLM apps: eval sets, rule-based checks and LLM-as-judge

Model outputs vary, a fix for one input often breaks another, and people are generous graders of fluent text. An evaluation set turns every change from a guess into a comparison: each case is an input plus what a good output must contain, graded by a rule or by an AI judge.

In Daveth Pro (AI Engineering)

What this page answers

How do you build an LLM evaluation set?

Start with 20 to 50 cases drawn from real usage, each an input plus what a good output must contain. Include the awkward ones, and add every bug you find as a new case. Read it in “An evaluation set”

Should you grade LLM output with rules or an LLM judge?

A rule checks whether the answer contains the expected fact: fast, free and predictable, but blind to phrasing. An AI judge is flexible, but it is another model call that can be wrong. Read it in the lesson

Beyond “looks good to me”

Model outputs vary, a fix for one input often breaks another, and people are generous graders of fluent text. Without an evaluation set every change is a guess. With one, you can compare prompts, models and retrieval settings on the same footing, and catch a regression before your users do.

An evaluation set

Each case is an input plus what a good output must contain. For the handbook assistant: 12 questions, each with the fact the answer has to state (“3.12”, “Tuesday”, “Okafor”). Start with 20 to 50 cases drawn from real usage, include the awkward ones, and add every bug you find as a new case.

CASES = [
    ("Which room is the IT desk in?",            ["3.12"]),
    ("When does the platform on-call rotation change?", ["tuesday"]),
    ("Who approves flights?",                    ["okafor"]),
    # ... 12 in all; add every bug you find as a new case
]

def rule(answer, expected):
    return any(x in answer.lower() for x in expected)

JUDGE = """Question: {q}
Reference fact: {ref}
Answer to grade: {a}

Does the answer state the reference fact correctly? Reply with only YES or NO."""

def run_eval(app):
    results = [(q, app(q), exp) for q, exp in CASES]
    return sum(rule(a, exp) for q, a, exp in results), results

Rules versus an AI judge

Two ways to grade. A rule: does the answer contain the expected fact? Fast, free and predictable, but blind to phrasing (“eight in the evening” fails a search for “20:00”). An AI judge: show a model the question, the reference fact and the answer, and ask YES or NO. Flexible, but it's another model call that can be wrong. I graded the same 12 answers both ways, with Qwen2.5-1.5B-Instruct as the judge.

Predict first. On how many of the 12 answers will the rule and the judge give the same verdict?

  • 11 or 12
  • 8 to 10
  • fewer than 8

Answer. 8 of 12. The rule passed 4 the judge rejected, and the judge passed 0 the rule rejected.

How to read the table: one row per question, with the assistant's answer and each grader's verdict. Rows where the two disagree are marked; those are the ones a person has to read to decide which grader was right.

QuestionAnswerRuleJudge
Above what amount does an expense claim need a receipt?An expense claim needs a receipt if it exceeds 150 euros.✓ correct✗ wrong · disagree
What time does the Lisbon office close on weekdays?The Lisbon office closes at 20:00 on weekdays.✓ correct✓ correct
Which room is the IT desk in?Room 3.12 on the 3rd floor.✓ correct✓ correct
How many days of annual leave does everyone get?Everyone gets 25 days of annual leave.✓ correct✓ correct
When does the platform on-call rotation change?The platform team's on-call rotation changes every Tuesday at 10:00.✓ correct✓ correct
Who approves flights?Marta Okafor approves flights.✓ correct✗ wrong · disagree
How soon must a lost laptop be reported?A lost laptop must be reported to security@ within 1 hour.✓ correct✓ correct
When is the monthly all-hands meeting?The monthly all-hands meeting is on the first Wednesday of each month at 16:00, in the Atrium and online.✓ correct✓ correct
How big is the yearly learning budget?The yearly learning budget for employees is 800 euros per person.✓ correct✗ wrong · disagree
How many weeks of parental leave does the primary carer get?The primary carer gets 20 weeks of parental leave.✓ correct✓ correct
How many days a year can someone work abroad?Up to 30 days a year.✓ correct✗ wrong · disagree
When do carried-over holiday days expire?Carried-over holiday days expire on 31 March.✓ correct✓ correct