Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
25 min read
At Level 4 you decide how much of a workflow a model should generate freely and how much should be fixed in advance. This module covers two results that pull in different directions. One shows that constraining generation with templates makes automated pipelines far more reliable. The other shows that letting a model explore and evaluate several paths can solve problems a single pass can't, at a steep cost. Knowing both lets you choose the least freedom that does the job.
The reliability gap. Alidu, Ciavotta, and De Paoli (2025) studied how to turn a plain-language description of a data pipeline into a working, scheduled pipeline. They compared four approaches across 260 experiments, 13 models, and five case studies:
The authors name the hybrid as the best of the approaches in which a model writes the code, and report it was more than twice as efficient as direct prompting per successful pipeline. This is a preprint and covers one kind of task, so treat the exact figures with care. The direction is the lesson: each step that took freedom away from the model raised reliability.
The trade. Templates only produce what someone anticipated. A request outside the templates fails or has to go to a person. The hybrid gives up some reliability to handle more variety. Which one you want depends on how varied your requests are and what a failure costs.
Why constraint wins. A model asked to write a whole pipeline must get the task logic right and also every detail of structure, naming, and syntax. Each is a chance to fail. In the constrained designs, the model does what it's good at: reading a messy request and producing a structured description of what is wanted. Fixed, tested code does what must be exact. The model's output is small, has a defined shape, and can be validated before anything runs.
The pattern in general.
The pattern applies well beyond data pipelines: report generation, document assembly, ticket routing, configuration.
Branching search. Yao and colleagues (2023) went the other way for a different class of problem. In Tree of Thoughts, the model proposes several possible next steps, evaluates each, continues from the promising ones, and backtracks from dead ends. On Game of 24, a puzzle that requires combining four numbers to reach 24, GPT-4 with chain-of-thought prompting solved 4% of cases. With Tree of Thoughts it solved 74%.
What that result does and doesn't mean. The puzzle has features most business tasks lack: many possible paths, early choices that can't be repaired later, and partial states that can be judged as promising or hopeless. Where a task has those features, such as planning under hard constraints, exploration can pay. It costs many times the calls of a single pass, and it needs a reliable way to evaluate a partial result. If the evaluation is weak, the search confidently explores the wrong branch.
A practical middle. You rarely need full search. A cheaper version: generate three candidate plans, score each against written criteria in a separate prompt, and carry the best one forward. That gives some of the benefit of exploring alternatives at three or four calls instead of dozens.
Choosing. Start from the most constrained design that could work.
Add freedom only when a test on representative cases shows the constrained design fails and the freer one passes, and report the cost per successful task alongside the pass rate.
Rafael Dominguez leads analytics for a regional grocery chain with 62 stores. Store managers ask for recurring reports in plain language: "Every Monday, send me last week's shrink by department, flagged if it's over 4%." His team wants requests turned into scheduled reports without an analyst hand-building each one.
Before:
You are a senior data engineer. Write a complete, production-ready
scheduled pipeline that fulfills this request from a store manager.
Include extraction, transformation, the report, and scheduling.
Request: [manager's message]
The model must invent table names, write query logic, choose a schedule format, and structure the whole job, all correctly, in one pass. When it fails, the failure could be anywhere. Nothing is validated before code exists.
After:
Turn a store manager's report request into a report specification.
Do not write code. Fixed templates build the report from your
specification.
Return a JSON object with exactly these fields:
- metric: one of ["shrink_pct", "sales", "units", "waste_lbs"]
- group_by: one of ["department", "category", "store"]
- period: one of ["prior_week", "prior_month", "month_to_date"]
- stores: a list of store numbers, or "requester_store"
- threshold: an object {"field", "operator", "value"}, or null
- schedule: {"day": Mon to Sun, "hour": 0 to 23}
- unclear: a list of anything in the request you could not map
to the fields above. Use [] if nothing is unclear.
Use only the listed values. If the request asks for something
the fields can't express, put it in "unclear" and set the
affected field to null. Do not guess.
Request: [manager's message]
The record is checked against the schema, and store numbers are checked against the requester's permissions. A valid record goes to tested templates that build the query, the report, and the schedule. A record with anything in "unclear" or a null field goes to an analyst. Rafael's team runs 120 past requests through both designs and reports, for each, the share that produced a correct report without analyst rework and the cost per correct report.
Why this is stronger. The model's job shrank to the part that needs language understanding. Everything that must be exact is fixed code that was tested once. An invalid or incomplete record is caught before anything is built, and the "unclear" field gives the model an honest way to say a request doesn't fit. The comparison is on real requests, with cost counted.
Setup. Pick a recurring request your team handles in a standard way: a report, a status update, a routing decision. Write down the stages that never change and the three to six details that vary by request.
Task. Write a slot-filling prompt that returns those details as a record with allowed values, an "unclear" list, and no code. Collect ten real or realistic requests, including two that don't fit. Run them, and check each record by hand.
What a good result looks like. Most records are valid and correct. The two requests that don't fit land in "unclear" instead of being forced into a field. You can name which validation check would catch each wrong record, and which part of the work you would hand to fixed code.