Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
Every production AI system will fail. The question is whether the failure is small, visible, and fixed for good, or large, silent, and repeated. Resilience is designed before the incident. This module covers how to price failure, set escalation thresholds against real losses and capacity, threat-model and red-team realistically, and turn each incident into a permanent control.
Track cost per completed outcome, not per call. Retries, timeouts, human rework, and incident handling all belong to the outcome they served. A workflow that looks cheap per call can be expensive per resolved case.
An escalation threshold trades missed cases against review load. Estimate the cost of each miss and each false escalation, and the review capacity you actually have. Choose the threshold with the lowest expected cost that fits capacity. Then add hard rules for harms that money doesn't capture: some cases always escalate regardless of score.
A retrieval-augmented agent reads documents, calls tools, and writes outputs. An attacker who controls a document can try to steer tool calls (injection) or get the agent to send data somewhere it shouldn't (exfiltration). Map assets, attacker-controlled inputs, trust boundaries, and tool permissions. Red-team multi-step paths with realistic goals, in an isolated environment with synthetic data, not just a list of override phrases.
Providers update models and change behavior. Pin versions where possible, monitor release notes, run your regression suite on any change, and keep a tested fallback with documented limits.
An unchanged prompt can behave differently because of runtime settings: maximum input or output tokens, temperature, retrieval index version, tool timeouts, or a new default model. Check the full configuration.
Each incident gets an owner, a root cause, a new regression test, and a control change. A runbook names who's on call, how to detect the failure, how to recover, and when to escalate.
Andre Kowalski runs operations AI for a property management company with 18,000 residential units. An agent triages maintenance requests, retrieves building manuals, creates work orders, and enriches requests with vendor availability. In March, a tenant's long message ended with "and there's a gas smell near the stove." The agent filed it as routine.
Weak approach. Add "ALWAYS treat gas as an emergency" to the prompt and close the incident.
Better approach. Andre runs a structured review.
Before:
Incident: gas request misfiled. Fix: added gas warning to prompt.
Closed.
After:
Incident review: MR-0311 (owner: A. Kowalski)
Root cause: prompt unchanged, but platform update lowered max input
from 8,000 to 2,000 tokens. Tenant message truncated silently;
"gas smell" was cut.
Controls added:
1. Ingestion length check; truncated messages flagged to dispatcher
with "content not processed" note.
2. Keyword pre-check (gas, smoke, CO, sparking, flooding) before
the model; matches always escalate.
3. Vendor-availability enrichment failure now completes triage
with "vendor data unavailable" instead of timing out.
4. Work-order tool responses schema-validated before dispatch.
5. Tenant phone and unit access codes removed from model input.
Threshold (30-day costs): missed emergency about $40,000; false
escalation about $8 of on-call time; capacity 120/day.
T1: 80/day, 1.0 misses/mo = $19,200 + $40,000 = $59,200
T2: 115/day, 0.4 misses/mo = $27,600 + $16,000 = $43,600
T3: 140/day, exceeds capacity
Selected T2, plus keyword rule above.
Tests: 12 truncation and emergency cases added to regression suite.
Runbook updated: detection, rollback, on-call contacts.
Red team: next quarter, test attacker-edited manual pages steering
work-order tool.
Why this is stronger. The review found that the runtime configuration changed, not the prompt, so a prompt warning wouldn't have helped. Truncation is now detected and disclosed. The threshold balances expected losses within real capacity, and the gas rule covers harm that expected cost understates. The failure became tests, controls, and a runbook entry with an owner.
Setup. Choose an AI workflow you run or depend on. Write down two failure types: a missed case and a false alarm.
Task. Estimate a rough cost for each and your daily review capacity. Calculate expected monthly cost at two different escalation levels. Then list every runtime setting besides the prompt that could change behavior.
What a good result looks like. Your calculation shows its formula and stays within capacity. You've named at least one case that should always escalate regardless of cost. Your settings list includes at least four items beyond the prompt text.