Why Your Agents Pass the Demo and Fail in Production

The evaluation discipline most enterprises skipped

Every enterprise AI program now has the same photograph in its deck: an agent that books the meeting, reconciles the invoice, or resolves the support ticket, flawlessly, on stage. The demo is real. The problem is that the demo is also a sample size of one. The moment that agent meets the long tail of real inputs, real users, and real edge cases, the question changes from "can it do this?" to "how often does it do this correctly, and how would we know if it stopped?"

That question is an evaluation question, and most organizations have not built the discipline to answer it. The data is striking. In LangChain's 2026 State of Agent Engineering survey of more than 1,300 practitioners, 57 percent of organizations report having agents in production, yet quality remains the single biggest barrier to scaling them. Nearly 89 percent have implemented observability, but only 52 percent have adopted evaluations. Read those two numbers together and the gap is obvious: almost everyone is watching their agents, far fewer are systematically judging whether the agents are right.

Observability tells you what happened. Evaluation tells you whether what happened was good. You can have perfect traces of an agent confidently doing the wrong thing. For enterprise leaders, that gap is the difference between a pilot that impresses a steering committee and a system that can carry a regulated, customer-facing workload without becoming a liability.

This article is about closing that gap. It covers why evaluating agents is a genuinely different problem from benchmarking a model or testing traditional software, what you actually measure, how the evaluation lifecycle works in production, and how the native tooling from AWS, Google Cloud, and Microsoft now compares to the third-party ecosystem. The thesis is simple: evaluation is the engineering discipline that separates an agentic demo from an agentic deployment, and it is buildable.

Why agent evaluation is a different problem

It is tempting to treat agent evaluation as a familiar problem in new clothing: run a test set, check the score, ship. That instinct fails for agents, for three structural reasons.

First, agents are non-deterministic in a way that compounds. A single language model call carries some randomness. An agent that plans, calls tools, retrieves context, and hands off to sub-agents may chain ten or twenty such calls in a single run. Small variation in step three cascades into materially different decisions by step seven. The same agent, given the same input, can take a different path on each run. That means a single test pass tells you almost nothing. You need many samples to get a stable read on behavior, which is a different statistical posture than asserting that a function returns the expected value.

Second, errors compound rather than stay contained. In traditional software, a bug usually produces a localized, reproducible failure. In an agent, a weak plan or a single wrong tool call early in the trajectory corrupts everything downstream. By the time you see a bad final answer, the root cause may be five steps back. Evaluating only the final output tells you that something broke, not where or why.

Third, and most consequentially, the final answer is only the last step of a longer process. An agent can call every tool correctly and still fail the user's actual goal. It can also reach a correct-looking answer through a wasteful, unsafe, or non-reproducible path that will not hold up at scale. This is why mature agent evaluation works across three altitudes rather than one:

End-to-end evaluation asks the business question: did the agent accomplish what the user actually wanted? This is task completion, and for most enterprise use cases it is the clearest single signal of whether the agent worked.

Trajectory-level evaluation asks whether the path was sound and efficient: did the agent plan well, avoid redundant or irrelevant actions, and reach the outcome without unsafe intermediate steps? Trajectory analysis catches duplicate tool calls, wasted tokens, and dangerous reasoning patterns that output-only checks miss entirely.

Component-level evaluation asks which specific part broke: the retriever, a particular tool, a sub-agent, a prompt. This is what makes a failing eval actionable rather than merely alarming.

The practical rule is to evaluate at all three altitudes, because each answers a question the others cannot. End-to-end tells you whether you have a problem. Trajectory tells you whether the path is trustworthy. Component tells you what to fix.

Figure 1: The three altitudes of agent evaluation. Each scope answers a question the others cannot.

What you actually measure

Across those altitudes, the metrics that matter for agents cluster into a small number of families. Getting these named and instrumented is most of the battle.

Tool calling. Did the agent select the right tool, with the right arguments, in the right order? Much of this is checkable deterministically: exact tool names, required parameters, and expected execution sequences do not need a model to judge them. Where the tool layer is the most common failure point, this is the highest-leverage thing to measure.

Planning. Did the agent decompose the task sensibly before acting? Poor plans are the upstream cause of most compounding failures.

Task completion. Did the full run accomplish the user's goal? The definition of "done" is task-specific, which is exactly why this typically requires judgment rather than string matching.

Reasoning quality. Was the agent's reasoning coherent, grounded, and free of hallucination? An answer can be fluent and still wrong; a tool call can succeed and still be the wrong move.

Once an agent runs in production, three more enter the picture and they are the ones executives feel directly: safety (does the agent stay within policy and avoid harmful or non-compliant actions), latency (response time, now a top-tier barrier as agents move into customer-facing work), and cost (token and compute economics per successful task).

The second half of measurement is choosing the right judge. The principle is to use deterministic checks for anything exact and a model-based judge only for the subjective. Tool correctness, schema validity, and required-field presence are deterministic; reasoning quality, helpfulness, and task completion are not. For the subjective cases, the LLM-as-a-judge pattern gives a model a rubric, the task, and the agent's output or trajectory, and asks for a score and a critique. It runs in three modes: pointwise scoring (rate one response on a scale), pairwise comparison (pick the better of two, which is more reliable because relative judgment is easier than absolute), and reference-grounded critique (compare against a known-good answer). A newer pattern, Agent-as-a-Judge, uses a full agentic system to critique another agent's entire trajectory rather than just its output, and aligns more closely with human expert judgment on process-heavy tasks. It is worth watching, though for most enterprise programs a well-designed LLM judge plus deterministic checks is the working combination today.

When the correct answer is exact, a deterministic check is faster, cheaper, and more reliable than any model. Tool trajectory is the classic case:

For the subjective dimensions, an LLM judge takes a rubric, the task, and the agent's output, and returns a structured score and critique. Note the explicit 0 to 5 scale, which aligns more closely with human raters than a 10-point scale:

A word of caution that belongs in every evaluation design: model-based judges have their own failure modes. Scores drift between runs and between judges, absolute scales are noisier than relative ones, and a judge can be wrong with confidence. Treat the judge as a component to be validated against human review, not as ground truth.

The evaluation lifecycle: offline, online, optimize

Evaluation is not a pre-launch gate you pass once. In production it is a continuous loop with three stages, and the major platforms have converged on exactly this shape.

Figure 2: The evaluation lifecycle. Offline gating catches regressions before users do, online monitoring catches what offline could not anticipate, and the optimize loop turns both into improvement.

‍Offline. Before anything ships, you run the agent against curated datasets of representative inputs with known-good expectations, often called golden datasets, ideally drawn from real engagement data rather than synthetic toy cases. These evals run in continuous integration, gating deployment so that a regression in tool selection or task completion blocks the release rather than reaching users. This is where evaluation behaves most like software testing. Wired into continuous integration, a failing eval blocks the merge:

Online. Once live, you sample real production traffic and score it continuously, because the inputs your agent meets in the wild will always exceed what any offline set anticipated. Online evaluation is how you catch the silent drift that happens as models update, tools change, and user behavior shifts, the degradation that never shows up in a demo and rarely shows up in a dashboard until a customer complains. The native services express this as a two-step client flow, generate then score, that runs continuously against sampled traffic:

The exact import paths and metric names differ across AWS, Google Cloud, and Microsoft, and the SDKs evolve, so treat the snippet above as illustrative of the shape rather than a copy-paste reference.

Optimize. The newest and most valuable stage closes the loop: turning evaluation signal into validated improvements. Instead of a human reading traces and guessing at fixes, the system surfaces ranked, evidence-backed changes to prompts and tool definitions, then validates them against held-out test cases or controlled experiments before promoting anything. This is where evaluation stops being a report card and becomes a mechanism for the agent to measurably get better over time.

Offline catches regressions before users do. Online catches what offline could not anticipate. Optimize turns both into improvement. An enterprise practice needs all three.

The platform landscape: native versus third-party

A genuinely useful evaluation strategy is vendor-neutral, because most enterprises run on more than one cloud and because the right tool depends on the job rather than the logo. Here is the current state of play. Note that two of the three hyperscaler platforms were renamed in 2026, which matters if your internal documentation still uses the old names.

Figure 3: Native evaluation capabilities across the lifecycle. Native optimization loops are most built-out on AWS and Microsoft as of mid-2026.

‍Amazon Web Services. Bedrock AgentCore Evaluations reached general availability in March 2026. It offers two modes that map directly onto the lifecycle above: online evaluation that continuously samples and scores live production traces, and on-demand evaluation for regression testing in CI/CD and interactive development. It ships with 13 built-in evaluators spanning response quality, safety, task completion, and tool usage, and supports Ground Truth in the form of reference answers, behavioral assertions for session-level goals, and expected tool-execution sequences. For domain-specific needs, teams configure custom evaluators using their own prompts and model, or implement code-based logic in Python or JavaScript. As of April 2026, AWS extended this with an optimization layer: recommendations that analyze production traces to propose improved prompts and tool descriptions, validated through batch evaluations and A/B tests before any change ships, with human approval required.

Google Cloud. In April 2026, Vertex AI was rebranded as the Gemini Enterprise Agent Platform. Its Gen AI evaluation service evaluates models, agents, and applications against your own criteria, and now supports partner models including Anthropic and Llama. Its Agent Evaluation capability scores task success and safety across multi-turn conversations, continuously scores agents against live traffic through online monitors, and lets teams debug reasoning paths through a unified trace viewer. The lifecycle shape, offline scoring plus online monitoring plus trace-level debugging, is the same.

Microsoft. Azure AI Foundry is now Microsoft Foundry, and its observability story is organized around four verbs that should sound familiar by now: Trace, Evaluate, Monitor, and Optimize. Its agent evaluators deliberately operate at two altitudes, scoring both the final outcome of the workflow and the quality of each intermediate step. Evaluations are generally available with continuous production monitoring piped into Azure Monitor, and at Build 2026 Microsoft extended tracing and evaluation to any agent framework through a single OpenTelemetry pipeline, so a team using one framework for orchestration and another for retrieval does not have to choose between the stack it wants and the observability it needs.

The third-party ecosystem. Native tooling is increasingly capable, but specialized platforms still earn their place, and the common enterprise pattern is to pair a lightweight framework for CI gating with a platform for annotation, regression tracking, and dashboards. The landscape sorts cleanly by job:

LangSmith is the natural fit for teams standardized on LangChain or LangGraph, with strong trajectory evaluation, annotation queues, and both offline and online modes. Braintrust positions itself as a full-lifecycle platform connecting datasets, scoring, production monitoring, and CI-based release gates. Arize Phoenix offers OpenTelemetry-native tracing and evaluation as open source, a low-friction starting point. Galileo focuses on production guardrails, low-latency scoring, and compliance-oriented evaluation. For code-first teams, DeepEval brings broad open-source metric coverage and pytest-style integration so evals run as part of the deployment pipeline, while Ragas remains the reference point for retrieval-specific RAG evaluation. Langfuse serves teams for whom self-hosting and data residency are non-negotiable, and Promptfoo is the common choice for red teaming and security testing.

The decision between native and third-party is not ideological. Native platforms minimize integration surface and keep evaluation next to where the agent runs and is observed. Third-party tools win when you need depth in a specific dimension, framework-native ergonomics, multi-cloud consistency, or capabilities the native platform has not yet matched. Most mature programs use both.

Figure 4: A tooling decision ladder. Default to native, add third-party for specific depth, and build custom only for what nothing else covers.

‍An illustrative evaluation practice

The following describes a representative way a vendor-neutral engineering team can sequence these capabilities on an enterprise engagement. It is an illustrative composite drawn from common practice across the cloud platforms and the third-party ecosystem, not a description of a specific client deployment, and it is offered as a starting framework to adapt rather than a prescriptive standard.

The sequence begins with defining what good means for the specific use case, in business terms, before any tooling is chosen. For a customer-service agent that means agreeing on what counts as a resolved ticket, an acceptable tone, and a policy boundary that must never be crossed. These definitions become the evaluation criteria.

Next comes golden-dataset construction, building a set of representative inputs with expected outcomes, drawn wherever possible from real historical interactions rather than invented examples, so the eval set reflects the actual long tail the agent will face. Subject-matter experts review and label a curated subset to anchor the criteria in human judgment.

With criteria and data in place, the team defaults to the native evaluation capability of whichever cloud the workload runs on, AWS, Google Cloud, or Microsoft, for the offline gating and online monitoring backbone, because keeping evaluation adjacent to the runtime reduces integration cost and operational surface. Third-party tools are introduced selectively where they add real depth: a code-first framework for rich CI-stage checks, a RAG-specific tool where retrieval quality is the dominant risk, a red-teaming tool for adversarial and safety testing.

Evaluation then runs at all three altitudes. Deterministic checks validate tool selection and arguments. Model-based judges, themselves validated against the expert-labeled subset, score task completion and reasoning. Trajectory evaluation watches for wasteful or unsafe paths. These checks are wired into CI gating so that a regression blocks the release.

Once live, production sampling scores a percentage of real traffic continuously, with negative-signal traces and high-stakes use cases prioritized for human review, since reviewing every trace is impossible at production volume. Findings feed an optimization loop: surfaced issues become candidate prompt and tool-definition changes, validated against the held-out golden set before promotion.

The throughline is that evaluation is treated as an engineering system with its own datasets, gates, and feedback loops, not as a one-time quality check. Teams adopting this approach should validate each stage against their own regulatory, data-residency, and domain requirements before relying on it in production.

From evaluation to governance and ROI

For a CIO or CDO, the reason this matters is not technical elegance. It is that evaluation is what makes agentic AI accountable rather than merely functional, and accountability is the precondition for scaling.

Tie evaluation criteria to business key performance indicators rather than abstract accuracy, and the eval suite becomes a continuous, quantified answer to the question every executive eventually asks: is this working, and how do we know? Regression gates turn "the agent still behaves correctly" into a guarantee that is enforced on every release rather than hoped for. Online monitoring turns silent degradation into an alert. And the optimization loop turns the whole apparatus into the kind of evidence-backed improvement story, and the ROI dashboard, that a finance leader will ask to see before approving the next expansion.

There is a strategic point underneath the operational one. The enterprises that scale agents past the pilot stage will not be the ones with the most impressive demos. They will be the ones that built the discipline to prove, continuously, that their agents behave correctly across the messy reality of production. That discipline is evaluation. The demo gets you the meeting. Evaluation is what lets you keep the system in production once the meeting is over.