Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
Governance is the set of rules, owners, and checks that decide what an AI system may do and how anyone would know if it stepped outside those limits. A prompt can describe policy, but it can't enforce it. This module covers how to put enforcement where it belongs, how to argue that a system is safe enough with evidence, and how to audit it so the argument stays true.
The model proposes. Something else decides. A model can draft a refund, a purchase request, or an account change. Whether that action is allowed should be checked by trusted application logic against identity, role, amount limits, and current policy. If a manipulated input persuades the model to propose something out of bounds, the proposal fails at the enforcement layer.
Layers must be independent. Three prompt warnings that all rely on the model behaving are one layer, not three. Real layers fail for different reasons: input filtering, model instructions, schema validation, a policy engine, a human check for high-impact cases, and monitoring. Ask of each pair: can one failure defeat both?
Write the safety case. State the intended use and limits, the hazards, the claims you're making ("no payment can exceed the policy limit"), the control behind each claim, the test evidence for each, and the residual risks. Name who accepts those residual risks and what would invalidate the argument.
Review the impact on people. For systems that affect customers or staff, examine privacy, accessibility, error impact by group, and recourse: can someone harmed by an output reach a person who can change it?
Match obligations to the domain. An internal drafting tool needs light controls. A system influencing credit, employment, health, or legal outcomes carries sector rules and higher evidence bars. Scale governance to the stakes.
Name owners and keep a trail. Every production prompt has an owner, a defined set of people who can change it, and a log of who changed what and why. For advisory outputs, name who is accountable for accepting the advice.
Formalize precedence. Define, in configuration, the order of authority: system policy, then user request, then retrieved content, which is never instructions. For policy documents, newer versions supersede older ones, and every policy carries an expiration or review date. Expired knowledge triggers a refresh, not silent reuse.
Give reviewers what they need. When a case escalates, show the input, the proposed action, the evidence, the policy version applied, and why it escalated. A reviewer who sees only a recommendation can only agree.
Audit independently. Someone outside the team that built the system samples real cases on a documented schedule and method and records findings.
Bound self-modification. If a system tunes its own prompts, examples, or memory, changes go through the same versioning, testing, staged rollout, and rollback as human edits.
Gabriel Osei heads AI risk at a property and casualty insurer. A claims assistant proposes payment amounts for small water-damage claims under $5,000. It went to pilot on the basis of "the prompt instructs the model never to exceed policy limits."
Weak approach. Add more warnings to the prompt and a line asking the model to confirm compliance.
Better approach. Gabriel commissions a safety case.
Before:
Add to the system prompt: "IMPORTANT: Never exceed policy limits.
Double-check compliance. Confirm the payment is within limits."
After:
Safety case brief: claims-payment assistant v1.3
Owner: claims automation lead. Residual-risk acceptance: VP Claims.
Claims:
C1 No payment exceeds the policy limit or deductible rules.
Control: rules engine checks every proposal against policy
record (not model text). Test: 400 boundary cases, 0 breaches.
C2 Retrieved documents cannot change payment rules.
Control: retrieved text is data-only; rules engine reads policy
from system of record. Test: 60 injection cases.
C3 Contested or ambiguous claims reach an adjuster with evidence.
Control: escalation shows photos, estimate, policy section and
version, and reason. Test: adjuster review of 50 escalations.
Precedence: policy system of record > claims guidelines (expire
annually) > model output.
Residual risks: estimate fraud from altered photos. Mitigation:
monthly independent audit of 2% sample by internal audit team.
Prompt edits: owner + one reviewer; logged.
Invalidated by: new claim type, rules-engine bypass, guideline
expiry without renewal.
Why this is stronger. Each claim has a control that doesn't depend on the model's cooperation, and each control has test evidence. Residual risk is named and owned. The audit is independent and documented. And the brief says what would make the argument stop holding.
Setup. Pick an AI workflow in your organization that can take or propose an action.
Task. Write three safety claims about it. For each, name the control that enforces it, whether that control depends on the model, and what test evidence exists. Then name who accepts the residual risk.
What a good result looks like. At least one claim is enforced by something other than the prompt. Any claim that depends only on model instructions is flagged as a gap. A named person, not a team, accepts residual risk.