AWS AgentCore Evaluations Uses OpenTelemetry to Score Agents Across Frameworks

Teams using different agent SDKs can send their systems through a single evaluation layer rather than maintain a separate scoring pipeline for each framework. Amazon Bedrock AgentCore Evaluations is designed to score agents whose telemetry follows supported OpenTelemetry or OpenInference conventions.

The supported list includes Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK and the Claude Agent SDK. The important distinction is architectural: AWS says the service reads standardized execution telemetry, rather than requiring each agent to use a framework-specific evaluation integration.

The scoring system starts with an execution record

OpenTelemetry is a vendor-neutral system for emitting traces, metrics and logs. A trace is a collection of spans, with each span recording one step of work. On AgentCore Runtime, the AWS Distro for OpenTelemetry routes spans and event records to Amazon CloudWatch.

When an evaluation runs, AgentCore Evaluations fetches that CloudWatch data and rebuilds a session. It uses an invoke-agent span for the user request and final response, inference spans for model calls, and execute-tool spans for tool names, inputs and results.

After reconstruction, the same evaluator set can assess every supported framework. AWS lists GoalSuccessRate, Correctness, Helpfulness and custom LLM-as-a-judge evaluations; the service does not need retrieval, reranking, guardrail or memory spans to run those evaluators.

Generic support is open-ended, not automatic

The system determines how to interpret telemetry from the instrumentation scope name stamped on spans. Named framework integrations are supported directly, while telemetry from scopes beginning with opentelemetry.instrumentation.* or openinference.instrumentation.* goes through generic processing paths for the respective conventions.

That makes the coverage broader than the named SDKs, but sets a firm boundary for custom tracing. A scope such as mycompany.agent.tracing is not picked up even if its spans otherwise follow the conventions; AWS uses the recognized prefix as the signal that the instrumentation deliberately conforms to its documented schema.

Trace collection can still block response-quality scores

Evaluation also requires spans to carry a session.id that matches the runtimeSessionId, allowing the service to group turns into a session. AgentCore Runtime injects that attribute automatically through ADOT, according to AWS.

Message content is the more consequential operational constraint. With unified observability, spans and content reside in the same per-agent log group. Older configurations can split spans into a shared log group and store content as correlated records elsewhere; selecting only the span source leaves content empty and causes response-quality evaluators to return an error.

The result is a portability promise with a practical test: an agent can change frameworks without changing the scoring service only if its instrumentation, session grouping and message collection remain compatible. For teams already centralizing agent traces in CloudWatch, that shifts evaluation work toward telemetry hygiene rather than SDK-specific plumbing.