Calling models well: JSON schema outputs, validation and retry

Most AI features in real software are a model call buried in ordinary code, and each call can fail in ways a chat never shows: malformed output, slow responses, wrong facts, smuggled instructions. Treat the model as an unreliable component: ask for structured output, validate it, and retry when it fails.

In Daveth Pro (AI Engineering)

What this page answers

How do you get reliable JSON from an LLM?

Ask for an exact schema, validate the reply in code, and retry with the error when it fails. This lesson measures how each step changes the success rate on invoice extraction. Read it in “Getting data back, not prose”

Why treat an LLM as an unreliable component?

Because each call can return malformed output, respond slowly, state a wrong fact or follow an instruction smuggled in through the input, so the code around it has to check. Read it in “From chatbot to product”

From chatbot to product

Most AI features in real software are a model call buried in ordinary code: classify this ticket, pull the fields from this invoice, answer from these documents, draft this reply. Each call can fail in ways a chat never shows: malformed output, a slow response, a wrong fact, an instruction smuggled in through the input. This track treats the model as an unreliable component and builds the engineering around it: structured output (this lesson), retrieval (C2), evaluation (C3), guardrails (C4), cost and speed (C5), tracing (C6) and agents (C7).

Getting data back, not prose

The task: pull the vendor, date and total out of 20 invoice snippets written in all sorts of styles (“12/05/2025 (day/month)”, “Grand total 59.99”) and return JSON your code can store. I asked Qwen2.5-1.5B-Instruct three ways: just “as JSON”; with an exact schema and one worked example; and with schema, example and a validator that rejects bad replies and asks again, up to twice, quoting the error.

Predict first. Asked simply for “JSON”, how many of the 20 replies could a program parse into the three required fields?

  • 15 or more
  • 8 to 14
  • fewer than 8

Answer. 10 of 20. Of the other 10: 7 kept the date in its original format (“June 2, 2025”, “22/11/2025”), 2 wrote the total as a string (“380 EUR”), 1 renamed a field (“total_payable”). With a schema and example: 20; with validation and retries: 20.

How to read the chart: for each way of asking, the outlined bar is the share of replies that parsed into valid JSON with the right fields and types; the solid bar is the share where all three values were also correct.

Drawing: Valid and correct JSON replies for three prompting strategies.

The schema and example did most of the work: valid replies went from 10 to 20, and fully correct ones from 10 to 19. The retry loop never fired: with the schema and example every reply was already valid, so it cost nothing here. Keep it anyway; it's insurance for the inputs you haven't seen yet. Notice the gap between the outlined and solid bars: valid JSON with a wrong value. The one remaining error was “Thanks for your order! Kettle & Co., 12/05/2025 (day/month).…”, returned as 2025-12-05: a format-perfect reply with the day and month swapped, even though the invoice said “day/month”. A validator can only check form; checking content needs the evaluations in lesson C3.