Use decision models with Amazon Bedrock AgentCore Evaluations

Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust. Amazon Bedrock AgentCore Evaluations , a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call your choice of external system, model, or API, including a decision model. Decision models suit questions with a fixed set of answers that an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. After you validate a question on your own traffic, a decision model can be sufficient for that decision or score, and you can extend the same approach to specific harmfulness or stereotyping categories in your policy. Because both backends can answer several questions about the same state in a single request, you can use them as a screening layer rather than a replacement, running a pass or fail signal across your metrics to select the turns that warrant deeper assessment by the built-in evaluators, whose explanations help you investigate what went wrong. This post shows how to connect two decision model backends to AgentCore Evaluations through code-based evaluators. The first is Jev , a hosted decision model from TypeSafe. The second is Strands Decider 2B , an open-source decision model you run yourself. By the end of this post, you will know how to convert the AgentCore evaluator input into a decision model request, and the model’s probabilities back into an AgentCore evaluation result. First, though, when should you reach for a custom LLM-as-a-judge evaluator, and when does a decision model fit better? The next section breaks down what a decision model is and the trade-offs involved. Decision models A decision model is a small, specialized model trained to select from a fixed set of options and return calibrated confidence scores rather than generating free-form text. A general-purpose large language model (LLM) evaluator produces a score with a rationale. A decision model instead produces a probability distribution over predefined answers. That gives you a direct confidence measure you can threshold, aggregate, and act on programmatically. This makes decision models well-suited for evaluation tasks where the judgment can be stated as a choice among a fixed set of options: “Is this response grounded?” or “Did the agent complete the user’s goal?” Decision models are most useful when you want a well-calibrated confidence score alongside the label, not just a binary judgment. They are also appropriate when you want to enforce a specific rubric consistently without relying on a general-purpose LLM to interpret it. Jev, the hosted decision model from TypeSafe, and Strands Decider , an open-source, 2-billion-parameter decision model from the Strands Agents team, both support three types of questions. A noul question poses a yes/no question and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies, returning the most probable option along with the full distribution and a confidence. A score question asks which of 2 to 10 ordered levels applies, returning a probability-weighted score, the distribution over levels, and a confidence. Many evaluation rubrics fit one of these types. “Is every fact in this response supported by the tool results?” is a yes/no question, and “How helpful was this response, on a four-level rubric?” is a score. Decision models versus LLM-as-a-judge The built-in evaluators in AgentCore today use a general-purpose LLM to assess the agent response against a rubric and explain its judgment. This is valuable when the evaluation requires connecting evidence, identifying missing information, or working through a response in depth in tasks where the explanation is as useful as the score. Decision models are not a replacement for LLM evaluators. They’re an addition that the code-based evaluator framework makes possible. Use them for narrow, well-defined checks alongside the built-in evaluators’ broader assessment. Solution overview To show how these evaluator types work together, this post builds an end-to-end example of an HR Assistant agent evaluated by deterministic code-based checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by Jev or Strands Decider 2B. The agent is built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. It answers employee questions using five tools that return fixed mock data such as paid time off (PTO) balances, PTO requests, HR policies, benefits, and pay stubs. We use fixed data in this code sample to make it straightforward to verify evaluator behavior from LLM-as-a-judge and decision models. When you additionally have ground truth, you can confirm whether each evaluator catches the right failures. This code sample helps you set up the framework for testing decision models compared to LLM-as-a-judge for your own use cases. Figure 1: End-to-end architecture of the HR Assistant evaluated by code-based checks, built-in evaluators, and decision-model evaluators backed by Jev or Strands Decider Prerequisites To deploy the sample, you need the following: An AWS account in a Region where AgentCore Evaluations is available. The sample defaults to US West (Oregon). Amazon CloudWatch Transaction Search enabled in the account. Access to a foundation model in Amazon Bedrock . Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker. Jev requires a TypeSafe API key. Visit typesafe.ai for access details. Strands Decider requires a host running the Strands Decider server. You can run it locally for testing, or on Amazon Elastic Compute Cloud (Amazon EC2) inside your virtual private cloud (VPC) so the Lambda function can reach it. Seeing it in action You can find the full code sample in the AgentCore samples GitHub repository . Two Python scripts handle everything: ../utils/deploy.py deploys the HR Assistant to AgentCore runtime. evaluate.py deploys evaluators, invokes the agent, and runs the evaluation modes. Running evaluate.py without flags executes five steps: Creates the Lambda execution role ( AgentCoreLambdaEvaluatorRole ). Packages and deploys two deterministic Lambda evaluators: hr-response-length (a trace-level evaluator that checks whether the agent’s response stays within length bounds) and hr-fact-checker (a session-level evaluator that regex-validates HR facts against the known mock data). Registers both as AgentCore code-based evaluators through the control plane. Runs on-demand evaluation with EvaluationClient and OnDemandEvaluationDatasetRunner across five labeled scenarios, combining the code-based evaluators with Builtin.Correctness , Builtin.Helpfulness , and Builtin.ResponseRelevance . Creates an online evaluation configuration that continuously scores live sessions at 100 percent sampling. At this point you have a working evaluation pipeline with deterministic checks and LLM-as-a-judge scoring. Decision-model evaluators are opt-in. You can add them with a flag,