Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
A prompt that worked last month can fail today without anyone touching it. Inputs change, models get updated, and someone's quick fix for one case breaks three others. This module covers how to protect behavior that already works, how to find out what changed when it stops working, and how to test that a prompt holds up under the variation real users bring.
Regression tests are your memory. Every behavior you rely on should be a saved test case with a known-good result: the refund question that needs the policy citation, the angry customer who must get an escalation, the empty input that must return "needs info." Before any change ships, the whole suite runs. A fix that helps one case and breaks two is caught before users see it.
Drift has three sources.
Watch production samples, not just test results. A weekly sample of 50 real outputs scored against your rubric will catch input drift that a static suite misses.
One factor at a time, written down. When something breaks, reproduce it with the saved input, model, and settings. Change one thing. Rerun the suite. Log the date, the change, the reason, and the before and after scores. A regression log turns "it got worse sometime in March" into "v4.2 on March 11 moved the escalation rule below the examples."
Portability is never assumed. A prompt tuned on one model is a hypothesis on another. Rerun the full suite after any model change, including minor version updates, and review the diffs, not just the totals.
Vary the phrasing. Users ask the same thing many ways, with typos, in fragments, in mixed languages. Write paraphrases of key test cases, and add edge cases: very long inputs, empty fields, contradictory requests.
Long conversations lose state. In a long chat, early instructions or facts get diluted or truncated. For multi-turn tools, keep critical state, such as the customer's plan and the open case number, in a structured summary that's re-sent each turn, rather than relying on the transcript.
Trim carefully. Removing context saves tokens and can improve focus. But a line that looks redundant may be what holds one edge case together. Remove in small batches and run the suite after each.
Devon Clarke maintains a reply-suggestion prompt for a 40-agent support team at a regional internet provider. Agents paste the customer's message and get a draft reply. Last Tuesday, the vendor updated the underlying model. By Thursday, agents noticed that outage-credit replies no longer cited the credit policy, and multi-message chats sometimes forgot the customer's plan tier.
Weak approach. Devon adds "ALWAYS cite the credit policy" in capitals, removes a paragraph he thinks is redundant to make room, and checks it on two tickets. Both look fine. The next week, cancellation requests stop triggering the retention handoff.
Better approach. Devon has a 60-case regression suite, so he starts there.
Before:
Change log: added "ALWAYS cite policy", removed paragraph 4. Tested
on 2 tickets. Looks good.
After:
Regression log, Oct 14
Change: none to prompt. Model updated by vendor Oct 8.
Repro: v5.1 prompt, new model, suite of 60. 51/60 pass (was 58/60).
Failures: 6 outage-credit (no citation), 3 multi-turn (plan tier
lost after turn 4).
Step 1: move credit rule from line 3 to directly after the policy
excerpt. Rerun: 57/60. Outage cases fixed. Multi-turn still failing.
Step 2: add {{account_state}} block (plan tier, open case) re-sent
each turn. Rerun: 60/60.
Added 10 paraphrased outage requests to suite. All pass.
Released v5.2. Rollback: pin the previous model version if the
vendor allows it. Do not roll the prompt back to v5.1 on the new
model; that pairing already failed 9 of 60.
Why this is stronger. The failure was reproduced on fixed cases, so Devon knew the model update caused it, not his edits. Each change was isolated and measured. The multi-turn fix manages state explicitly. The suite grew to cover phrasing variation, and nothing was trimmed without a full rerun.
Setup. Take a prompt you use more than once a week. In a spreadsheet, write 10 test inputs: 6 typical, 2 paraphrases of the same request, and 2 edge cases (empty or very long). Add a column for the expected key behavior.
Task. Run all 10 and record pass or fail. Then make one edit to the prompt, such as deleting a sentence you think is redundant. Rerun all 10. Add a log row: date, change, reason, before score, after score.
What a good result looks like. You have two scored runs on identical inputs and a log row that explains the difference. If anything broke, you can name the case and the change that caused it.