Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
AI systems in an enterprise sit between two groups of people: the users they serve and the staff who oversee and build them. Both relationships need design. This module covers how much autonomy to give a system and where people step in, how to learn from human overrides, how to personalize without overreaching, and how to measure whether your own practitioners can actually do the work.
Autonomy should follow risk and reversibility. A system that drafts internal summaries can act freely, since mistakes are cheap and easy to fix. A system that sends customer notices should have sampled review. A system that commits money, changes records, or affects someone's rights should propose, not act, unless the action is small, reversible, and well tested. Raise autonomy only with evidence, and keep the ability to lower it quickly.
Route what the system shouldn't decide. Define in advance what goes to a person: low confidence on a consequential case, a request outside the system's scope, conflicting evidence, or a policy exception. Routing rules belong in application logic with logged reasons.
Overrides are data, not verdicts. When reviewers change outputs, record what they changed and why. Patterns in overrides reveal model gaps, unclear policy, or reviewer inconsistency. But a reviewer can be wrong too, especially under time pressure or with weak evidence. Sample overrides for second review before using them to retrain or retune.
Adapt to confirmed needs. A new hire and a 20-year specialist need different explanations. Let users choose their level, or confirm it from role and stated preference. Don't infer expertise from writing style, age, or name. Facts and safety limits stay the same at every level. Only depth and terminology change.
Personalization needs permission and limits. Before a system remembers anything about a user, define what types of information it may store, for what purpose, for how long, and how the user can see, correct, or delete it. Tell users plainly. Exclude speculative inferences ("seems unmotivated") and sensitive information the purpose doesn't require. A pilot showing better satisfaction doesn't, on its own, justify wider memory.
Measure practitioners by what they can do. If your organization trains staff to build and run AI workflows, assess observable work: given an unfamiliar task, can they write a grounded prompt, design an evaluation, spot an injection risk, and document the result? Use practical tasks scored against anchored rubrics by calibrated assessors. Prompt counts, usage hours, and quiz recall are activity, not capability. If you claim training improved outcomes, compare against a baseline and account for who chose to take it.
Priyanka Rao leads the internal AI program at a 9,000-person engineering consultancy. A knowledge assistant answers staff questions about design standards and project procedures. The team wants to add memory so it "learns each user over time," and leadership wants to show the training program is working.
Weak approach.
Before:
Remember everything useful about each user and adapt answers to
their skill level. Infer expertise from how they write. Report
prompt volume per trained employee as the training success metric.
Better approach.
After:
Memory policy v1 (owner: P. Rao; review: privacy office)
- May store: user-stated role, preferred detail level, active
project codes. Purpose: relevant answers. Expiry: 180 days or
project close.
- Must not store: inferred skill, performance, health, or personal
traits; client confidential data.
- Disclosure: shown at first use; "What I remember" panel with
edit and delete. Deletion removes downstream copies within 7 days.
- Detail level: user selects Novice/Standard/Expert; default
Standard. Safety notes and standard citations never vary.
Routing: questions on structural safety factors or code compliance
with confidence below threshold go to the discipline lead with the
question, retrieved sections, and reason.
Overrides: discipline leads' corrections logged with reason; 10%
sampled monthly for second review before any prompt change.
Competency: trained staff complete a practical task on an
unfamiliar brief, scored by two calibrated assessors on grounding,
evaluation design, risk handling, and documentation. Outcome
claims compare trained and not-yet-trained teams on rework rate.
Why this is stronger. Memory is scoped, disclosed, user-controlled, and expiring, with inferences excluded. Expertise is chosen, not guessed. High-stakes questions route to an accountable person with evidence. Overrides feed improvement only after a second look. Competency is measured on observable work, and outcome claims have a comparison group.
Setup. Pick an AI system in your organization that adapts to users or keeps history, or one you're considering.
Task. Write a five-line memory policy: what may be stored, what must not be, purpose, expiry, and how users see and correct it. Then write one routing rule that sends a consequential uncertain case to a named role.
What a good result looks like. Nothing inferred about a person's ability or character is storable. A user could find, correct, and delete what's held about them. The routing rule names a trigger, a destination, and the evidence passed along.