AWS introduces an open-source tool to compare AI agents across frameworks

AWS introduced the Universal Agent Evaluation Framework, or UAEF, on November 11, giving developers an open-source system for comparing AI agents across supported frameworks. It converts different execution records into a common format, then applies consistent scoring so teams can track whether changes to prompts, models or tools improve an agent.

The starting problem is fragmentation. Frameworks record messages, tool calls and handoffs between agents differently. AWS says teams often build their scoring and storage around one format, then must rebuild that work when they switch frameworks or use several. That makes it harder to tell whether an agent improved or simply went through a different evaluation pipeline.

First, give every run the same record

UAEF uses adapters to translate each execution record, called a trace, into a shared representation named AgentTrace. The scoring engine reads that representation rather than a framework’s native format. Support includes LangGraph, LangChain and Strands Agents, as well as agents running on Amazon Bedrock AgentCore.

Teams can supply an existing trace from a past run or log. Without one, UAEF can call an agent through an HTTP endpoint or an AgentCore runtime identifier, collect its response and convert the result. Both routes produce the same record for scoring.

AWS says customers across 12 organizations in telecom, healthcare and financial services have used UAEF to evaluate agents and compare versions before release.

Then score behavior—not just the final answer

The seven dimensions cover tool calling, response quality, responsible AI, performance, multi-turn conversations, multi-agent coordination and reasoning. Checks range from tool selection and argument validity to conversational context and token use. Each metric returns a score between zero and one, with one representing the best result.

• Deterministic metrics calculate scores directly from traces and reference answers, producing identical results when repeated on the same inputs.
• AI-judge metrics ask a model on Amazon Bedrock to assess qualities such as relevance or safety. Developers can also add custom metrics or use RAGAS and DeepEval integrations.

Those judges introduce their own attack surface: the agent’s output becomes input to another model. UAEF separates the grading instructions from that content and escapes the content to prevent it from being read as instructions. Judge responses must cite an exact passage from the material being scored; scores with evidence that cannot be found are rejected.

AWS warns that judge scores are evaluation signals, not safety decisions, and should not gate a security control. Runtime protections belong in the agent’s own execution path.

Turn scores into a development history

When a team saves an evaluation, UAEF stores experiment metadata in DynamoDB and full results in S3. Runs can record the framework, model, tools, system prompt, parameters and a Git commit, tying scores to a specific design. UAEF screens traces for personal information before storage and supports redaction of stored copies.

A comparison engine checks shared metrics against a baseline and labels changes as improvements, regressions or unchanged. Its default threshold is 5%, but teams can adjust it. Regressions are grouped by dimension, exposing cases where a prompt change helps response quality but hurts tool use.

Statistical reports show score distributions, confidence intervals, outliers and failed test cases without using an LLM. Separately, developers can ask a Bedrock model to analyze a report and suggest causes or next steps. Those suggestions are hypotheses for the next revision, not measured explanations of why a score changed.

The Apache 2.0 release includes a Python library, deployable AWS service, web interface and runnable notebooks. Code and interface use the same evaluation engine. AWS’s getting-started guide points to the awslabs/uaef repository, but advises teams to begin with metrics suited to their task rather than running every check and spending unnecessary Bedrock invocations.