Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
A single evaluation tells you how a system performed once. An evaluation program tells you whether it's still performing, where it fails, whether a change actually helped, and whether it's safe to release. This module covers how to run evaluation as an ongoing practice and how to turn its results into release decisions that hold up, including when the answer is "don't ship."
Monitor continuously. Inputs change and so do models. Sample production outputs every week, score them against the same rubric, and track the trend. A slow slide in quality is easy to miss without a chart.
Small gains need skepticism. If version B gets 74 of 80 right and version A gets 73, that's one case. Look at which cases changed in each direction, whether the sample represents real traffic, and whether reviewers knew which version they were scoring. Report the uncertainty. Don't call a one-case difference a gain.
Break results down. An overall score can rise while a small, important group gets worse. Report results per subgroup with sample sizes, and say plainly when a sample is too small to be confident. A slice with 15 cases can show a problem worth investigating. It can't prove a rate.
Test outside the comfort zone. Most test sets look like development data. Add edge cases and out-of-distribution inputs, such as unusual formats, languages, or requests the system wasn't designed for, and check that it fails safely.
Control experiments. When comparing zero-shot to few-shot, or persona to no persona, hold everything else constant, define quality criteria in advance, and never select examples or tune wording using the final test cases. Few-shot examples should include hard and boundary cases, not just easy ones. Many personas turn out to add nothing once tested this way.
Right answer, wrong path. A model can get the right label for the wrong reason. Test with counterfactuals: change the one fact that should flip the answer and see whether it flips. Ask for cited evidence and check it. Don't rely on the model's explanation of its reasoning, which is generated text, not a record.
Gate releases. Define in advance which regressions block release: any increase in critical errors, any failure on a protected subgroup beyond a threshold. If a release trips a gate, it ships only with a documented mitigation reviewed by the accountable owner, and a fresh test.
Measure cost per success. Divide total cost by tasks that met the quality bar. Include model calls, retries, human correction time, and downstream errors. A cheaper model that misses a hard quality bar doesn't ship, whatever its token price.
Jonah Weiss runs evaluation for an airline's rebooking assistant. Version B is a few-shot prompt with a "senior agent" persona on a cheaper model. Developers report it scores 74 of 80 versus 73 for version A, at 40% lower model cost. They want to ship it before the holiday peak.
Weak approach. Ship B. It's more accurate and cheaper.
Better approach. Jonah's release review asks for more.
Before:
Release review: B 74/80, A 73/80. B 40% cheaper. Ship B.
After:
Release review: B vs A (reviewers blinded to version)
Overall: B 74/80, A 73/80. B wins 4 cases, loses 3. Difference is
within noise for this sample.
Subgroup set (separate from the 80): passengers requesting
wheelchair or mobility assistance (n=15): B 10/15, A 14/15. B dropped the assistance request in 4
rebookings. Small sample; pattern investigated.
Cause: few-shot examples had no assistance cases. Persona test: B
without persona scored identically. Persona removed.
Counterfactual: changing "connecting" to "nonstop" flipped the
right answer for A in 9/10 tests, B in 6/10. B's correct
answers partly lucky.
Edge/OOD: group bookings over 9 passengers; both fail safely to a
human agent.
Cost per successful rebooking (incl. agent correction time):
A $0.71, B $0.83.
Gate: critical error (dropped assistance request) increased.
Release blocked. Mitigation: add 4 assistance examples; rerun on a
fresh 40-case assistance set.
Why this is stronger. The one-case gain was treated as noise. The subgroup failure surfaced despite the higher average. The persona was tested and removed. Counterfactuals showed some correct answers were lucky. And cost per success, including human correction, reversed the apparent saving.
Setup. Take two versions of a prompt you've used, or write a second version of one. Prepare 20 test inputs, including 5 from a subgroup or edge type that matters.
Task. Shuffle the outputs so the version labels are hidden, then score them. If you can, ask a colleague to score them the same way without knowing which is which; otherwise rescore them yourself the next day without looking at your first scores. Record overall scores and subgroup scores separately. For 3 cases, change one fact that should change the answer and check whether it does.
What a good result looks like. You report the overall difference along with the number of cases that changed each way. Subgroup results are reported separately with their sample size. At least one counterfactual result tells you something the overall score didn't.