The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

When you deploy an agent to production, one of the hardest questions is whether it’s actually working? The bar isn’t whether it runs. It’s whether it does what the business needs, at scale. Many teams will tell you they can’t really see this. They spot-check outputs, watch latency, and collect user feedback when they can get it. None of that tells you whether the agent finished the user’s task, stayed on topic across a long session, leaked personally identifiable information (PII), or quietly skipped half the steps it was supposed to follow. Figure 1: Teams usually watch surface signals, but business outcomes are decided by the agent behaviors below the waterline Agent evaluation solves this problem. What makes this challenging is that a good score on one metric can hide a failure on another. A support agent can resolve every ticket and still expose personal information in its replies. In this case, the agent is failing although it has a high completion rate. To get evaluation right, measure the behaviors that matter for each agent, at the right level of detail. This post shows how to do this on Amazon Bedrock AgentCore. We use the AgentCore built-in evaluators together with the DeepEval and AutoEval evaluators directly from AgentCore Evaluations. Start with business outcomes, instead of metric names The most common mistake is to start with the metric list and pick the names that sound right. “Helpfulness sounds good, so we use it.” The result runs and produces numbers, but it gives almost no useful insight. The metric fits its label, not what success means for the agent. Instead, start with three steps. Define success in business terms. For a customer support agent, that might be first-contact resolution. For a research agent, it might be producing a complete, well-sourced answer grounded entirely in retrieved documents. For a content moderation assistant, it means flagging everything that should be flagged and only what should be flagged. Break the outcome into observable behaviors. A customer support agent that achieves first-contact resolution will typically complete the user’s stated goal, stay coherent across turns when the issue requires clarification, keep responses within compliance boundaries, and correctly invoke backend tools like ticket creation or order lookup. Map each behavior to an evaluator at the right level. Trace-level metrics cover per-turn or per-tool-call behaviors. Session-level metrics cover outcomes that only make sense when you look at the whole conversation. Figure 2: Translating a business outcome into observable behaviors and then into specific evaluators at the trace or session level This three-step process is the mental model that makes evaluation useful. The rest of this post is about what AgentCore gives you to execute it. The evaluator catalog AgentCore has a broad catalog of evaluators from three sources: Built-in evaluators are authored and quality-tested by AWS. You reference them by their ID ( Builtin.GoalSuccessRate , Builtin.Helpfulness , and 13 more) and the service handles everything else. They are benchmarked for reliability and cover the common dimensions, so they are the straightforward place to start. Third-party managed evaluators come from DeepEval and AutoEval , two established open-source evaluation libraries. When you use them in managed mode, AgentCore hosts the code, picks the judge model, and runs inference on its own capacity, exactly like built-in evaluators. If your team already uses one of these libraries in development, you can run the same metrics in production without a separate evaluation service. Custom evaluators take a built-in or third-party evaluator as a base, so you can swap in your own model and modify the prompt or rubric. Figure 3: The three evaluator tiers trade AWS quality assurance for increasing control over the model and account that run inference After creation, a custom derived evaluator is used exactly like other ones. The evaluator landscape AgentCore ships with a broad catalog of evaluators, so you rarely start from a blank page. Before you write a custom judge, check whether one already measures the behavior you care about. The following figure shows the whole landscape. Figure 4: The full evaluator landscape on AgentCore across built-in, DeepEval, AutoEval, and custom sources Built-in AWS evaluators fall into five groups: Response quality covers whether the answer is accurate, grounded, and well-formed: Correctness , Faithfulness , Helpfulness , ResponseRelevance , Conciseness , Coherence , InstructionFollowing , and Refusal . Task and goal completion has GoalSuccessRate , which looks at the whole session and asks whether the user’s goal was met. Tool use has ToolSelectionAccuracy and ToolParameterAccuracy , which check whether the agent picked the right tool and passed it the right arguments. Skills has SkillSelectionAccuracy and SkillInstructionFollowing , for agents that load skills: was the right skill invoked, and were its steps followed? Safety has Harmfulness and Stereotyping . Next to these are three ground-truth trajectory checks: TrajectoryExactOrderMatch , TrajectoryInOrderMatch , and TrajectoryAnyOrderMatch . They compare the agent’s actual tool path against the one you expected. Reach for them when how the agent got there matters as much as what it produced. DeepEval provides managed open-source metrics in four groups: Task and goal completion: TaskCompletion , GoalAccuracy , and ConversationCompleteness . Conversational quality: TurnRelevancy , KnowledgeRetention , and Summarization . Tool use: ToolUse . Safety: Bias , Toxicity , and PIILeakage . These metrics are especially useful for multi-turn agents. ConversationCompleteness catches an agent that answers the latest message but quietly drops earlier requests. KnowledgeRetention catches an agent that forgets something the user said three turns ago. AutoEval offers more specialized signals: Security flags malicious output, such as attempts to inject instructions or reveal system prompts. Possible catches over-cautious agents that wrongly declare a valid task impossible. Humor matters for consumer-facing agents where personality is part of the product. Custom evaluators . When no evaluator in the catalog fits, you can write a custom LLM-as-a-judge. Every evaluator’s exact definition, scoring scale, and input requirements are in the prompt template library in the AgentCore documentation . That includes the judge prompt it uses, the trace or session level it runs at, and whether it needs a reference answer or an expected trajectory. You can call the ListEvaluators API to get the full catalog and your own custom evaluators in one response. Multi-dimension evaluators Each evaluator so far returns one score per trace or session. That works when you measure one behavior. But step 2 of the outcome mapping rarely gives one behavior per evaluator. A support reply must resolve the issue, follow refund and data policy, and stay professional. One judge that weighs all three returns one number, and you cannot tell which criterion drove it. A multi-dimension evaluator is a custom LLM-as-a-judge with a list of dimensions instead of one scale. Each dimension has its own name, instruction, and numeric or categorical scale. The judge reads the trace once and scores every dimension in the same call. AgentCore returns one result per dimension. Figure 5: A multi-dimension evaluator scores several dimensions from a single judge call and returns one result per dimension with no composite score Use a multi-dimension evaluator when your success criteria can fail independently, such as an answer that’s accurate but non-compliant, or resolved but rude. It also needs every criterion to be judgeable from the same evidence, so that one judge call replaces several near-identical evaluators re-reading the same trace. When you score the criteria separately, you can gate on each one. A release that improves tone but regresses policy