Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
At senior level, an evaluation isn't a pile of test cases. It's a design that answers a specific question with evidence you'd stake a release on. This module covers how to start from that question, keep tuning data apart from the data you judge on, score with anchors that reviewers apply consistently, and break errors into dimensions so each fix targets a real cause.
Start with the question. "Is v2 better?" is too vague. "Does v2 extract purchase-order fields at least as accurately as v1, on all supplier formats, without more format failures?" tells you what to measure, on which data, and what counts as a pass.
Control the comparison. Same cases, model, settings, and rubric for both versions. Only the prompt changes. Use cases that represent production: the real mix of formats, lengths, and difficulty.
Keep tuning and holdout apart. You'll look at failures and adjust the prompt. The cases you study while adjusting are tuning data. Hold back a separate set that you only score, never study. Otherwise your prompt fits the test, and the score stops predicting production.
Score dimensions separately. An output can be correct but unfaithful (a true fact the source doesn't contain), faithful but wrongly formatted, or well formatted and wrong. Score correctness, source faithfulness, and format as separate dimensions. One blended number hides which problem you have.
Anchor the scale. For each score level, describe an observable example: "3: all required fields present and match source; 2: one non-critical field missing; 1: any critical field wrong or invented." Have two reviewers score the same 20 outputs, compare, discuss disagreements, refine the anchors, and repeat until agreement is high. For live scoring, send disagreements to a third reviewer to adjudicate.
Precision and recall for extraction. Precision is correct values returned divided by all values returned. Recall is correct values returned divided by all values that should have been returned. A prompt that returns only fields it's sure of can have high precision and poor recall. Report both, by field.
Check confidence against correctness. If the model labels outputs "high confidence," check whether high-confidence outputs are actually more often right. If not, the label is decoration and shouldn't drive routing.
Look at subgroups. A 95% average can hide a supplier format, language, or document type where accuracy is 70%. Break results down by meaningful groups and report sample sizes.
Name errors, then fix causes. Classify failures: omission, substitution, invention, format, wrong unit. A cluster of omissions on one layout points to an input problem. Inventions point to missing permission to say "unknown."
Repair examples by rule. If a few-shot example contradicts your rule, fix the example to match the rule, and make the example set cover your real input types. Then rerun the holdout.
Wen Li owns a purchase-order extraction prompt at a medical supply distributor. It extracts PO number, supplier, line items, quantity, unit, and ship date. A teammate reports v2 is "much better: 100% valid JSON versus 91%."
Weak approach. Ship v2 on parse rate.
Better approach. Wen sets the question first: is v2 at least as accurate as v1 on fields, across supplier formats? She uses 150 tuning POs for adjustments and 100 holdout POs she never inspects during tuning.
Before:
Eval: v1 vs v2, parse success. v1 91%, v2 100%. Ship v2.
After:
Eval: v1 vs v2, holdout n=100, same model and settings.
Dimensions scored separately: format, field correctness,
source faithfulness.
Format: v1 91%, v2 100%.
Ship date field, v2: 70 returned, 63 correct, 90 required.
Precision 63/70 = 90%. Recall 63/90 = 70%.
Ship date field, v1: 84 returned, 76 correct, 90 required.
Precision 90%. Recall 84%.
Subgroup: v2 recall on scanned POs (n=22) is 41%.
Error type: omissions on scanned POs; 4 inventions on
date-free POs.
Confidence label: "high" outputs 88% correct, "low" 85%.
Not useful for routing.
Decision: do not ship. Fix example set (no scanned PO examples;
one example shows an invented date). Re-test on holdout.
Why this is stronger. Format success didn't hide a recall drop. Separate dimensions and per-field metrics found the real problem, and the subgroup breakdown located it. The error taxonomy pointed to the cause: an example set missing scanned layouts and containing an example that broke the "unknown" rule.
Setup. Take 20 outputs from any extraction or summarization prompt you use. In a spreadsheet, write a three-level anchored scale for correctness, with one concrete description per level.
Task. Score all 20. Then either ask a colleague to score them independently with your anchors, or set the sheet aside and rescore them the next day without looking at your first scores. Count agreements. For each disagreement, note which anchor was unclear and rewrite it.
What a good result looks like. You can report your agreement rate as a number. Each rewritten anchor would have settled the disagreement it came from. If you scored extraction, you can also report precision and recall for at least one field.