Not Everything Needs an LLM—But an LLM Can Help Build Everything

Two claims about large language models sound contradictory until you separate runtime from build time.

At runtime, not every customer service step needs an LLM. Status lookups, SLA timers, refund eligibility tables, queue routing, and ordered status transitions are often better as ordinary software. Same inputs, same path, same audit trail.

At build time, an LLM can help create almost any of those pieces: scripts, integrations, dashboards, test cases, runbooks, and the glue between systems. Teams ship ordinary automation faster. Leave live decisions in software where a model adds nothing.

That split is the practical design move for service organizations. Put models where language, ambiguity, and multi-step judgment show up. Keep predictable work in code and workflows. Use models while you construct the stack that runs without them.

The expensive habit: LLM-by-default

Many programs ask where they can put AI. Ask instead which steps need flexible interpretation, and which only need correct execution.

Anthropic’s guidance on effective agents emphasizes starting with the simplest solution that works. Add complexity only when it improves results. OpenAI’s practical guide points toward agents when work involves complex decisions, brittle rule sets, or substantial unstructured input. It points away from them when a fixed procedure already handles the case.

A password-reset flow with clear identity checks is rarely a good first home for open-ended autonomy. A multi-product billing dispute with incomplete notes often is.

An LLM on a deterministic step can still look impressive in a demo. In production it may add latency, non-determinism, token cost, and evaluation burden for work a rules engine or API already finishes. The failure is putting AI on a job software already owned.

Two places models earn their keep

1. Runtime judgment

Use a model when the work depends on language or context that is hard to freeze as permanent rules:

• classifying free-text tickets and chats into intent and queue
• extracting entities from emails, attachments, and case notes
• summarizing a long thread for a human
• drafting replies inside an approved knowledge base and tone guide
• deciding whether a case is ready for an automated action or needs escalation
• coordinating multi-step resolution when the next tool call depends on intermediate evidence

Even here, keep the model inside a bounded workflow. Allowed tools. System-of-record read-backs. Step limits. Explicit handoffs.

A draft is not a send. A proposed refund is not a posted refund.

NIST’s AI Risk Management Framework is a reminder that mapping, measurement, and management belong in the operating design. Keep them out of a policy binder added after launch.

2. Build-time construction

Separately, models are strong at making the non-model parts:

• generating and refining integration code against CRM, ITSM, and order APIs
• scaffolding dashboards for queue health, containment, rework, and escalation reasons
• turning SOPs into executable workflow definitions and test fixtures
• writing evaluation sets from historical tickets, with privacy controls
• producing operator runbooks, change notes, and on-call checklists
• proposing schema mappings and data-quality checks for knowledge sources

In this mode the LLM multiplies engineers and operators. What ships can be ordinary software: deterministic routing, typed APIs, scheduled jobs, and BI that skips a model on every page view.

You get generative speed without paying generative cost and variance on every customer contact.

Prefer deterministic execution. Use generative systems to build, and to judge.

A customer service reference shape

Gartner has projected that agentic systems will autonomously resolve a large share of common customer service issues over the coming years. More recent service-leader survey work has emphasized expanded human responsibilities as AI changes contact mix. Headcount cuts are the wrong frame.

Whatever mix a team targets, the architecture still needs automated, assisted, and human-owned paths.

Intake and triage

1. Ingest the contact into a ticket or conversation object.
2. Normalize structured fields with ordinary validation.
3. Classify and extract with a model only where free text is the main signal.
4. Route with deterministic rules once intent, product, language, and priority are written to fields the queue engine understands.

The model’s job ends when the case is well-labeled. The router’s job is boring and correct.

Three resolution lanes

Lane A. Deterministic self-serve and automation.
Order status, shipment tracking, in-policy reschedules, FAQ answers from approved content, password flows with existing identity proofing. No open-ended generation on the happy path. If you generate at all, constrain it to presentation over known API facts.

Lane B. Bounded AI workflow.
Read the account, retrieve policy passages, draft a response, propose a next action, and stop at a gate. Examples: goodwill credit within a hard cap, subscription change inside published rules, warranty eligibility when documents are complete. Tool calls must return verifiable results. Read them back before claiming success.

Lane C. Human ownership with AI assistance.
Fraud suspicion, regulated complaints, multi-party escalations, emotional distress, legal risk, or anything outside the authority envelope. The model prepares a brief and timeline. The person decides and executes privileged actions.

Handoff and evidence

Most AI service programs fail at the transfer, not the greeting.

A good handoff fires when confidence, policy, sentiment, or authority rules say stop. It passes the full thread, extracted facts, tools already tried, and the reason for escalation. It never forces the customer to restart. It lands in a queue staffed for that exception type.

Treat handoff quality as a primary metric beside containment.

For every automated or semi-automated action, define proof: API responses and field-level read-backs, knowledge chunk IDs, policy version, who approved a gated step, and timestamps. “The model said the address was updated” is not evidence. A successful update call plus a later read of the shipping address is.

What to build with LLMs so runtime stays lean

Teams often underuse models in the workshop and overuse them on the floor. A practical backlog:

1. Workflow inventory from real tickets and notes, not only the SOP PDF.
2. Interface map of safe read/write operations on CRM, order, billing, identity, and knowledge systems.
3. Authority matrix: observe, draft, execute-with-approval, execute-autonomously.
4. Automation first. Implement Lane A as tests-backed software and platform workflows.
5. Thin intelligence. Classification, extraction, drafting, or tool use only where Lane B needs them.
6. Operator surfaces. Volume by intent, automation rate, rework, escalation reasons, cost per completed contact, and quality samples you already collect.
7. Evaluation harness. Golden transcripts, refusals, tool failures, and regression packs before widening autonomy.
8. Change control. Prompt, tool, and policy changes versioned like application releases.

LLMs can accelerate inventory, authority design, automation code, dashboards, and eval sets. Production execution still prefers code and controlled workflows.

Why the split helps

Deterministic steps fail in familiar ways and can be unit tested. Isolating models shrinks the surface that needs probabilistic evaluation.

Tokens, retries, and long contexts are real costs. Spend them on ambiguous judgment and on build acceleration. Leave status codes to the order API that already returned them.

Fixed workflows and logged tool results produce cleaner trails than free-form agent narratives. High-impact actions stay behind explicit gates. That makes NIST-style risk management operational.

Integrations, dashboards, fixtures, and runbooks dominate calendar time. Using models to construct those artifacts shortens time-to-pilot. Keep live contacts off a generative path unless the step needs one.

As routine contacts move into Lanes A and B, people spend more time on exceptions, relationship-heavy cases, and supervising automated paths. Roles change, and often broaden, as AI shifts volume.

Grow autonomy in order: draft, then gated execute, then narrow auto-execute. Each widening of scope is an evidence review.

Implementation checklist

1. Name one bounded workflow outcome. Not “improve support.”
2. Split steps into deterministic, judgment, and human-only.
3. Confirm system-of-record actions can be verified with read-backs.
4. Write the authority envelope and escalation owners.
5. Build Lane A without a model wherever rules already exist.
6. Add Lane B only for steps that fail as pure rules.
7. Instrument handoff quality, rework, and exception reasons. Not only automation rate.
8. Use LLMs to generate software, tests, and dashboards for the above. Run those artifacts under normal engineering control.
9. Sample automated and escalated contacts on a fixed cadence before raising autonomy.

If a step is stable, structured, and high-volume, make it software. If it is messy, linguistic, or multi-system with incomplete information, consider a bounded model-driven workflow. If it can create lasting customer or regulatory harm, keep a human in authority until evidence says otherwise.

While you build that stack, let models help create the code, dashboards, evaluations, and documentation. Production can stay as simple as the work allows.