Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
Organizations rarely fail at AI because they picked the wrong model. They fail because they funded the wrong problems, underestimated what errors would cost, or built something people wouldn't use. This module is about the decisions that come before architecture: which use cases to pursue, how to compare unlike opportunities, and when the right answer is to redesign or say no.
Start with the problem, not the model. For each proposal, write down: the decision or task it changes, who owns that outcome, who is affected, what data it needs, what rules govern that data and decision, and what happens today without AI. Architecture choices come after this map, because the map determines what's allowed.
Compare on four axes. Proposals arrive with different units: hours saved, revenue, risk reduced. Put each on the same frame:
A high-value, low-readiness idea may be a data project first and an AI project later.
Count the whole cost. Model fees are usually the smallest line. Add integration, human review time, correction of errors, incident handling, monitoring, retraining staff, and eventual retirement. A workflow that saves 2 minutes per case but needs 3 minutes of checking on 40% of cases has a different value than the pitch deck suggests.
Weight errors by harm. Two systems with a 3% error rate are not equally risky. A wrong label on an internal document costs a re-file. A wrong denial on a customer's hardship request can cost them housing. Ask who is harmed by each error type, whether they'll know, whether it can be reversed, and how often the costly kind happens. Frequency alone hides the errors that matter.
Adoption is evidence. Watch people use a prototype in their real setting. Do field staff with gloves and poor signal abandon it? Does it work with a screen reader? Do experienced staff trust it too much or not at all? Low adoption is often a design problem, not a training problem, and mandating use hides it.
Some proposals should stop. If a use case needs data you can't lawfully use, real-time facts the system can't access, or accuracy you can't verify, redesign the scope or decline. Write down the reason so the idea isn't resubmitted unchanged next quarter.
Teresa Gallagher chairs the AI steering group at a 3,500-employee regional utility. Five proposals are competing for one implementation team this year, including a call-center assistant that drafts responses to billing disputes and a tool that summarizes field-inspection reports for engineers.
Weak approach. Each sponsor presents projected hours saved. The billing-dispute assistant wins on volume: 40,000 disputes a year, about 6 minutes saved each. The group funds it.
Better approach. Teresa asks each sponsor to answer a short brief before ranking.
Before:
Rank these five proposals by estimated hours saved.
After:
For each proposal, return:
1. Task and outcome owner; affected people; governing rules and data
permissions.
2. Net annual value as a range: savings minus review time, error
correction, incident handling, monitoring, and run cost. State
assumptions.
3. Worst credible error, who bears it, whether they would notice,
and whether it can be reversed. Estimated frequency.
4. Readiness: data available and permitted (yes/no/partial), named
owner, team capacity.
5. Adoption evidence from at least 5 observed users, including
accessibility.
6. Any mandatory requirement not met. If so: redesign option or
recommend decline.
Do not rank. The steering group ranks.
The answers change the picture. The billing assistant's drafts need agent review on most disputes because account histories are incomplete, cutting net savings by more than half. A wrong draft could tell a customer in arrears that a shutoff is scheduled when it isn't, or the reverse. That error is rare but severe and hard for the customer to detect. Agents in the pilot also stopped using it within a week because it didn't work with their existing screen layout. The inspection summarizer saves less time on paper, but errors are caught by engineers who read the source reports, data is ready, and engineers kept using it.
The group funds the inspection summarizer and sends the billing assistant back for redesign: drafts for low-risk dispute types only, with shutoff-related cases excluded until account data is fixed.
Why this is stronger. Governance and data permissions were mapped before anyone talked about models. Value was net of review and failure costs. Error harm was weighed by severity and detectability, not just rate. Adoption was observed, not assumed. And the redesign is documented with its reason.
Setup. List three AI ideas your organization has discussed or could plausibly pursue. Open a spreadsheet with columns for the six items in Teresa's brief.
Task. Fill in each row honestly, using ranges and "unknown" where you lack data. For each, describe the worst credible error in one sentence and who would bear it.
What a good result looks like. At least one idea looks different after the exercise than before it, either less valuable once failure cost is counted or riskier once error harm is described. Any idea with a failed mandatory requirement has a written redesign option or a decline recommendation.