Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
Leaders get asked whether an AI system works and whether it's worth more investment. A demo, a dashboard trend, or a vendor benchmark can all suggest yes while the operating reality says otherwise. This module covers how to design evaluations that support enterprise decisions: controlled, representative, honest about uncertainty, and governed so the evidence stays trustworthy over time.
A metric that rose after a release didn't necessarily rise because of it. Seasonality, a new help-center page, a policy change, or a shift in who contacts you can all move the number. Online metrics also arrive late: a resolved ticket that reopens in 14 days counts as a success today. To attribute a gain, you need a comparison, such as a holdout group or a staggered rollout, and an outcome window long enough to capture delayed effects. A polished demo on chosen examples tells you only that those examples worked.
Write down the question, the population, the metrics, the minimum sample, the safety bars, and when you'll stop. Deciding to stop the moment results look good inflates false wins. Report results with intervals, not just point estimates. Some metrics are balanced trade-offs, such as speed versus thoroughness. Some are hard bars, such as zero unsafe advice in a defined category, that no gain elsewhere can offset. Cost-quality experiments need the same structure: the cheaper configuration ships only if it clears every hard threshold.
Version the benchmark, labels, rubric, and settings. Keep a protected holdout that's never used for tuning. Audit labels, especially on few-shot examples, because a mislabeled example teaches the error. When inputs or outcomes drift, such as new products, new user groups, or changed policies, refresh the benchmark and record the change. Rubrics drift too. Periodically have reviewers score fresh cases independently, adjudicate disagreements, and update anchors.
Results on one model, one language, and one user group don't transfer automatically. Test combinations: models crossed with user segments crossed with languages. For languages, use native-speaker reviewers checking task outcomes, not just translation quality. For multimodal inputs, check that text and image readings agree. When switching providers, define shared behavioral requirements, write adapters for differences in message and tool formats, and run the same test suite on each. When behavior differs across models, diagnose with versioned, controlled runs: same inputs, settings recorded, one variable changed.
Maya Lindgren leads AI operations for a consumer electronics company with support in English, Spanish, and French. Leadership saw a demo of a new assistant configuration on Provider B, and the CSAT dashboard rose 5 points in the month after the English pilot. They want to switch all three languages to Provider B next quarter.
Weak approach. Cite the CSAT rise and the demo, and green-light the switch.
Better approach. Maya writes an evaluation plan before any decision.
Before:
CSAT up 5 points since pilot. Demo strong. Recommend full
switch to Provider B.
After:
Question: Does Provider B config v2 match or beat current config
on resolution, safety, and cost across EN/ES/FR and both customer
tiers?
Design: 50/50 randomized assignment, 6 weeks, stratified by
language and tier. Same help-center content in both arms.
Primary: resolved with no repeat contact within 14 days.
Hard bars: 0 unsafe battery or electrical guidance (reviewed);
refund-policy accuracy at least 98%.
Cost: ship only if cost per resolved case is no higher than
current, and all hard bars pass.
Stopping: fixed at 6 weeks or 12,000 conversations per arm;
no interim decisions except safety stop.
Benchmark v4: 900 cases, 300 per language, including a protected
holdout of 180 (60 per language); native-speaker reviewers
calibrated on 40 fresh cases.
Screenshot cases: check image reading agrees with text summary.
Adapters: provider-specific tool-call format; shared test suite.
Note: English CSAT rise coincided with new returns page;
not attributed to pilot.
Why this is stronger. The CSAT rise is set aside as uncontrolled. Delayed outcomes are captured by the 14-day window. Safety bars can't be traded for speed or cost. Stopping rules prevent cherry-picking a good week. Languages and tiers are tested in combination, with calibrated native reviewers, and provider differences are isolated through adapters and a shared suite.
Setup. Find a claim in your organization that an AI change improved something, such as faster handling, higher satisfaction, or fewer errors.
Task. Write down three other changes that happened in the same period and could explain the result. Then draft a one-paragraph evaluation design that would separate the AI effect: comparison group, outcome window, hard bars, and stopping rule.
What a good result looks like. Your alternative explanations are specific and plausible. Your design names a comparison, a window long enough for delayed outcomes, and a stopping rule set in advance.