Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
A workflow that's affordable at 50 runs a day can be expensive at 50,000, and one that answers in two seconds in testing can take twenty in production. Cost and latency are design constraints, not afterthoughts. This module shows how to decide which model each step needs, which calls are worth making at all, and how to cut tokens without cutting the controls that keep output correct.
Larger models usually cost more per token and respond more slowly. Smaller models are cheaper and faster, but may miss more. The right comparison is total task cost: model cost plus the cost of the errors it produces. If a cheaper model saves $300 a month but creates 90 hours of manual correction, it's the expensive option.
So choose per step. Measure each candidate model on your test set for that step, with a quality bar defined in advance. Use the cheapest model that clears the bar. Different steps in one workflow can use different models.
Latency follows the same logic. Know the deadline that matters to the user. A background batch job can wait. A live chat reply can't. Sequential calls add their latencies together. Parallel calls don't.
Multi-step workflows accumulate calls. Before merging or removing any, measure what each one contributes:
Consolidation can save money and time. It can also hide a stage you needed to inspect. Decide with measurements.
A token is the unit models process and providers usually bill. Common sources of waste:
What not to cut: the schema, the rules that handle edge cases, the boundary examples, and the delimiters around untrusted input. Trim in small batches and rerun your regression suite after each.
Malik Johnson runs automation for claims intake at a regional property insurer. About 20,000 claim emails arrive each month. The current workflow makes four calls per email, all on the larger model. The rates below are illustrative. Check your own provider's pricing.
Before:
Call 1 classify claim type 1,800 in / 20 out (larger)
Call 2 extract fields 2,500 in / 300 out (larger)
Call 3 summarize for adjuster 2,500 in / 250 out (larger)
Call 4 draft acknowledgment 1,200 in / 200 out (larger)
Per email: 8,000 in, 770 out = $0.0400 + $0.0116 = about $0.052
Monthly: about $1,031
Malik measured before changing anything:
claim_type field to Call 2 matched Call 1's accuracy on a 200-email holdout set.After:
Call A extract fields + claim_type 1,900 in / 320 out (larger)
= $0.0095 + $0.0048 = $0.0143
Call B draft acknowledgment 1,200 in / 200 out (smaller)
= $0.0006 + $0.0003 = $0.0009
Call C summary, on demand (6%) avg $0.0010 per email
Per email: about $0.0162
Monthly: about $324
Holdout field accuracy: unchanged at 98%.
Why this is stronger. Each change was measured before it shipped. The model for extraction was chosen on total cost, including adjuster time, not on token price. The removed calls had no measurable value. Tokens were cut from content the task didn't use, while the schema and validation stayed intact. Fewer sequential calls should also cut response time, which Malik confirms in the pilot.
Setup. Pick a workflow you run or could run with two or three AI calls. In a spreadsheet, list each call with estimated input and output tokens. (A rough guide: 1,000 English words is often around 1,300 tokens. Your tool may show exact counts.)
Task. Pick illustrative per-million-token rates for a larger and smaller model. Calculate cost per run and per month at a volume you choose. Then, for each call, write one sentence on what it contributes and what would break without it. Propose one change and estimate its saving.
What a good result looks like. Your spreadsheet shows the formula for each cost cell. Each call has a stated purpose. Your proposed change includes what you'd measure to confirm quality holds before shipping it.