Before You Build an AI Agent, Score the Workflow

Start with whether the workflow needs an agent at all. Model choice comes later.

An agent is software that can interpret a goal, choose among actions, use tools, inspect results, and continue until it stops. Attach a chatbot to a process and you still lack that loop.

Every extra decision adds flexibility. It also adds another place for cost, latency, or error.

Anthropic’s guidance on effective agents recommends starting with the simplest solution that can work. Add complexity only when it produces a demonstrable improvement. OpenAI’s practical guide points toward agents when work involves complex decisions, brittle rule sets, or a lot of unstructured information.

Apply those principles to the backlog with a scorecard.

1. Is there a bounded outcome?

A good candidate has a finish line that two people can recognize in roughly the same way.

“Help with customer service” has no finish line. “Resolve eligible order-status requests, document the outcome, and route exceptions to the right queue” does. The second version names the input, the expected action, the system of record, and the escalation path.

Write the outcome as a verb plus an observable business result. Then list what must be true before the work is complete. If the team cannot agree on completion, the agent will automate the ambiguity.

Score one point if the workflow has a specific outcome, a known owner, and an explicit stopping condition.

2. Does the work require judgment across changing inputs?

Traditional automation is excellent when inputs are structured and the rules are stable. A form submission can trigger a notification. A status change can create a task. A fixed calculation can run without a language model.

An agent becomes more plausible when the work includes emails, documents, notes, or conversations that vary in form. It also becomes more plausible when the next action depends on context, or when you cannot fully specify the sequence in advance.

That is where model-driven classification, extraction, planning, and tool selection can add value.

Isolate the decisions that need flexible interpretation. Leave deterministic steps as ordinary software. A reliable design may use an agent for one judgment in the middle of an otherwise conventional workflow.

Score one point if the workflow contains recurring contextual decisions that would be hard to maintain as fixed rules.

3. Can the agent inspect real evidence as it works?

An agent that only generates plausible text proposes a workflow. The work stays undone.

Useful agents need access to ground truth: the current CRM record, the API result, the ticket state, an approved knowledge source, or another system’s validation response.

Anthropic emphasizes environmental feedback because it lets an agent assess progress instead of assuming that an action succeeded.

For each step, ask what evidence proves that the step worked. “The model said it updated the account” is not evidence. A successful tool response, followed by a read-back of the changed field, is.

This requirement often exposes integration work that a demo hides. That is healthy. Better to learn early that a critical system has no safe interface than after a polished prototype creates false confidence.

Score one point if important actions return verifiable results and the workflow can read those results back.

4. Are there safe checkpoints and exception paths?

Design autonomy around consequences, not novelty.

A low-risk agent may draft an internal summary and let a person decide what to do. A medium-risk agent may update reversible fields but require approval before contacting a customer. A high-risk action may stay outside the agent entirely: moving money, changing access, making a regulated decision, deleting records.

Define three things before implementation:

1. actions the agent may take without approval
2. actions that require a human checkpoint
3. conditions that stop the run and create an exception

Also define limits: maximum steps, maximum spend, allowed tools, permitted data, and timeouts. A clear refusal or escalation is a valid outcome. Silent improvisation is not.

NIST’s AI Risk Management Framework treats governance, mapping, measurement, and management as connected work. Put controls in the workflow design. Leave them out of a policy document added after launch.

Score one point if permissions, approval gates, limits, and escalation ownership are explicit.

5. Can you measure performance against today’s process?

A successful demo leaves the business case unmade.

Before building, capture a small baseline from the current workflow. Useful measures may include cycle time, queue age, rework, exception rate, completion rate, cost per completed item, or time people spend on specific steps. Choose measures that reflect the outcome, not the model.

Then assemble a representative evaluation set. Include normal cases, edge cases, incomplete inputs, conflicting instructions, and situations that should escalate. Record the expected result for each.

This set becomes the basis for comparing a prompt, workflow, or agent against the current process.

Measurement also protects against unnecessary complexity. If a simpler prompt-and-review flow performs as well as an autonomous loop, the simpler system is likely easier to understand and operate. Add complexity only when the evaluation shows that it earns its place.

Score one point if the team has a baseline, a representative test set, and an agreed definition of acceptable performance.

Reading the score

• 0–1 points: The workflow is not ready. Clarify the process or improve the underlying data first.
• 2–3 points: Test a narrow assistive workflow. Keep a person in the loop. Avoid broad autonomy.
• 4 points: A bounded pilot may be justified, with explicit controls and evaluation.
• 5 points: The workflow is a strong candidate for an agent pilot. Production value remains unproven.

The score is a decision aid. Discovery still has to happen. A five-point workflow can still fail because of poor data, inaccessible systems, unacceptable risk, or economics that do not work. A two-point workflow may become viable after the team standardizes one missing step.

The useful product of this exercise is the list of assumptions that must be tested. That list gives a pilot a clear shape: one workflow, a bounded toolset, known checkpoints, representative cases, and measures that can support a go, revise, or stop decision.

Choose the workflow before you choose the framework.