Any team can call an LLM API. Very few can run one in production.

Why common implementations fail

Teams adopt AI as a mandate rather than a fit. The result is features where the model adds variance without adding value: a support router that a keyword map handled at 99% accuracy, now at 92% and $0.01 per message; a "smart" form validator that a regex did deterministically. Worse, the probabilistic layer becomes load-bearing, so removing it later means rebuilding the workflow.

The decision framework we apply

We score a candidate use case on four axes: ambiguity (is the input unstructured language or judgment?), tolerance (can the workflow absorb an occasional wrong answer, or does one error cause real harm?), economics (does automating this save more than it costs at expected volume?), and defensibility (is there a deterministic fallback when the model is unavailable?). A use case that scores low on ambiguity belongs in code. One that scores low on tolerance needs a human in the loop, not full automation.

Trade-offs we make explicit

Even good AI use cases trade determinism for flexibility, and latency and cost for capability. We make those trade-offs visible up front: a summarization feature might be worth 3 seconds and a fraction of a cent; a real-time input validator almost never is. Where a hybrid works, a cheap deterministic pass that only escalates ambiguous cases to a model, we design for it, because it is usually the cheapest and most reliable answer.

The takeaway

The engineering maturity here is restraint. We will happily talk you out of a model when a plain function is the better product decision, and push hard toward one when the return is genuinely there.

What we instrument

Every model call emits a structured trace: prompt template version, resolved input, retrieved context, model and parameters, token counts, latency, cost, and the raw output. Traces carry a correlation ID through the whole workflow, so a single user action is reconstructable end to end. This is the AI equivalent of structured logs with request IDs, non-negotiable infrastructure, added on day one, not bolted on after an incident.

Evaluation as a first-class signal

We maintain a versioned evaluation set, representative inputs paired with a scoring rubric. It runs in CI so no prompt or model change ships without a measured quality delta, and it runs against sampled production traffic so live quality is a dashboard, not a guess. When a provider silently updates a model, the eval score moves and an alert fires before users notice.

Governance controls

Prompts and policies are versioned artifacts in source control with review, not strings edited in a console. Outputs that drive consequential actions are logged with their inputs for audit. Where regulation or risk demands it, we add human sign-off and immutable audit trails. Personally identifiable data is redacted or tokenized before it leaves your boundary.

The takeaway

If you cannot see it and cannot explain it, you cannot operate it. We build the microscope and the paper trail before we build the feature, so "is it getting worse, and why?" always has an answer.

Confidence-gated autonomy

We design workflows around explicit confidence signals, the model's own calibrated certainty, agreement between multiple passes, retrieval match quality, or business-rule checks. High-confidence, low-stakes outcomes execute automatically. Low-confidence or high-stakes outcomes escalate to a person with the context pre-assembled. Over time, the autonomous band widens as evaluation proves it is safe to.

Reviewer experience is the product

The difference between a loop that scales and one that collapses is the reviewer's tooling. We surface the model's reasoning, the sources it used, and a one-click accept/override, and we capture every override as labeled training and evaluation data. A good loop gets cheaper over time because the system learns from the corrections.

Failure and fallback

Every step assumes the model can be unavailable, slow, or wrong. Work queues so a provider outage delays rather than drops. Timeouts fall back to a deterministic path or a graceful "needs review" state. Nothing silently disappears; every item ends in a known, auditable state.

The takeaway

Automation should remove toil, not accountability. The best human-in-the-loop systems spend human attention like a scarce budget, on exactly the decisions that need it, and nowhere else.

Right-size the model per task

Not every call deserves the flagship model. Classification, extraction, and routing often run beautifully on a small, fast, cheap model; the premium model is reserved for the genuinely hard generation step. Routing tasks to the smallest model that passes evaluation is frequently a 5-20× cost reduction with no measurable quality loss.

Cache aggressively, including semantically

Identical requests should never hit the model twice; a plain response cache handles those. Beyond that, semantic caching returns a stored answer for requests that are close enough in meaning, and retrieval results are cached independently of generation. For high-repetition workloads this quietly removes a large fraction of spend.

Bound context, retries, and loops

Context is billed by the token, so we trim ruthlessly: retrieve the few most relevant chunks, not the whole document. Retries use capped exponential backoff, not infinite loops. Agentic workflows get hard step and budget ceilings so a reasoning loop can never spend without limit. Streaming means users stop generation early, and stop paying, when they have what they need.

Meter by tenant, alert on anomalies

Cost is attributed per feature and per customer, so you can see unit economics and catch a runaway before it compounds. Budgets and alerts turn "why is the bill 4× this month?" into a notification you saw on day one.

The takeaway

LLM cost is an architecture decision, not an accounting surprise. Model choice, context discipline, caching, and hard ceilings are what keep a successful feature from becoming an expensive one.