UAEF: A framework-agnostic system for closed-loop evaluation of AI agents

Teams building AI agents on AWS use a range of frameworks, including LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore. Each framework records agent execution in its own way, so the structure of messages, tool calls, and handoffs between agents differs from one to the next. Evaluation tooling tends to follow the same boundaries: teams build metrics, scoring logic, and result storage against a single framework’s trace format, and they must rebuild them when they switch frameworks or run agents on more than one. This coupling makes a basic development question harder to answer: did a change to the prompt, model, or tools improve the agent? Answering it requires consistent metrics across versions of an agent and a shared record of past results to compare against. Agent quality also spans several dimensions that a single accuracy score does not capture: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination, and reasoning. In this post, we introduce the Universal Agent Evaluation Framework (UAEF). Customers across 12 organizations in the telecom, healthcare, and financial services industries used UAEF to evaluate their AI agents with one consistent set of metrics and compare agent versions before release. UAEF is a framework-agnostic evaluation system: it uses adapters to convert traces from each framework into a common format, then scores those traces with 36 built-in metrics across 7 dimensions. It stores the results in Amazon DynamoDB and Amazon Simple Storage Service (Amazon S3) and generates reports and insights that inform the next iteration of the agent. This post describes how UAEF works, provides a playbook for choosing metrics and preparing datasets, and points to runnable notebooks for each supported framework. UAEF: One evaluation layer for agent frameworks UAEF is a framework-agnostic evaluation system for AI agents. It takes the agent a team already has and scores it the same way across supported frameworks and runtimes. UAEF takes two inputs: the agent under test and a dataset to run it on. The dataset provides the inputs for each test case, and it can also include ground-truth answers, the expected output for each input. Ground truth is optional, and only metrics that compare the agent’s response against a reference use it. To evaluate tool calling, the ground truth must also include the expected tool calls for each test case: which tools the agent should call, in what order, and with what arguments. Everything after that happens inside the UAEF. Figure 1 shows the path a single evaluation takes, from the inputs to the reports and insights. First, an adapter reads the agent’s native execution, its messages, tool calls, and agent-to-agent handoffs, and normalizes it into one canonical trace, so that the steps that follow are identical no matter which framework produced the agent. Next, UAEF screens the trace for personally identifiable information (PII), such as email addresses, phone numbers, and credit card numbers, before data is stored. The evaluation engine then scores the trace on a selection from 36 built-in metrics across 7 quality dimensions. When you persist a run, UAEF records it as an experiment in Amazon DynamoDB and Amazon S3: Amazon DynamoDB holds the per-experiment metadata used for listing and comparison, and Amazon S3 holds the full results. With runs stored, a team can compare an agent version against a baseline to detect regressions and generate reports and insights that explain what changed and why. Those reports close the loop: the team evaluates the agent, learns what regressed or improved, changes the prompt, model, or tools, and evaluates again. Figure 1: UAEF closed-loop evaluation data flow The rest of this section follows that flow in order: the adapters that make UAEF framework-agnostic, the evaluation engine and its metrics, PII detection, experiment storage and comparison, and the reports that close the loop. Canonical trace and adapters Adapters are what make UAEF framework-agnostic. An adapter reads the record of an agent’s run, including its messages, tool calls, and agent-to-agent handoffs. It converts that record into one canonical representation called the AgentTrace , which is the only thing the rest of UAEF scores. Because of that conversion, UAEF works with the agent, whatever SDK it was built with (such as LangGraph, LangChain, or Strands Agents). It can run locally, behind an HTTP endpoint, or on a managed runtime such as Amazon Bedrock AgentCore, with traces in Langfuse or not captured at all. UAEF provides an adapter for each of these, and they produce the same AgentTrace , so metrics, scores, and comparisons stay consistent across every one of them. A trace reaches UAEF two ways. If one already exists, from a past run, a log, or an observability store, it is handed to UAEF and the matching adapter normalizes it. If not, UAEF invokes the agent for you: given a runtime Amazon Resource Name (ARN) or an HTTP endpoint, it calls the agent on each dataset input, collects the response, and adapts it in one step. Either way, the output is the same AgentTrace . Because every metric, mode, and report operates on the AgentTrace rather than on a framework native format, the scoring that follows is identical no matter where the trace came from. That separation, normalization first and scoring second, is what lets a team compare an agent built on one framework against an agent built on another on a single scale. Evaluation engine The evaluation engine scores a canonical trace across 7 quality dimensions, each capturing a distinct aspect of agent behavior that a single accuracy number would hide. Tool calling measures whether the agent selected the right tools, called them in a sensible order, and passed valid arguments. Response quality measures whether the final answer is relevant, complete, accurate, and grounded in the provided context rather than hallucinated. Responsible AI measures safety, bias, toxicity, and resistance to prompt injection. Performance measures operational cost in latency, token use, and throughput. Multi-turn measures how well the agent sustains a conversation, including context retention, coherence, and goal completion across turns. Multi-agent measures coordination in systems of cooperating agents, such as task delegation and the division of work. Reasoning measures the internal logic of the agent’s chain of thought, including consistency and freedom from fallacies. Each dimension is computed by one or more of 36 built-in metrics, and every metric returns a normalized score in the interval [0, 1], where 1 is best, so dimensions and the overall result share one scale. Metrics come in two families. Deterministic metrics compute a score directly from the trace and the ground truth and reproduce exactly on repeated runs. Large language model (LLM)-as-a-judge metrics score qualities that resist a closed-form definition, such as response relevance or safety, by prompting a judge model on Amazon Bedrock. You can also add your own custom metrics or draw on the RAGAS and DeepEval integrations. Because judge metrics feed an agent’s own output back into a prompt, that content is treated as untrusted input: an agent that can influence its judge can influence its own scores. The rubric is sent separately from the content, and the content is escaped so it cannot break out of its delimiter and be read as instructions. Each judge response is parsed strictly and must cite a verbatim span of the content it scored. A score whose evidence is not found is rejected rather than recorded. Judge scores are evaluation signal, not a safety decision, and should not gate a security control. For runtime enforcement on the agent itself, apply Amazon Bedrock Guardrails in its own invocation path. The full model is in SECURITY.md. Experiment management and comparison UAEF records each evaluation as an experiment, so results from different agen