AWS Makes CloudWatch Omni Available for Testing and Debugging AI Agents

AWS has made Amazon CloudWatch Omni generally available, giving teams a single service to trace, evaluate and experiment with generative AI applications and agents. Its central bet is that latency and error rates are not enough when a small prompt or configuration change can alter an agent’s answers without producing a conventional system error.

A monitoring problem that begins after the request succeeds

Traditional monitoring can show whether a request completed and how long it took. It does not by itself establish whether an agent selected the right tool, retrieved useful material, followed a sound route, or produced a helpful answer. AWS positions Omni as a response to that gap, pairing observability with evaluations and experiments for agent systems.

The service records an agent’s execution as a structured trace. Teams can inspect model calls, tool invocations and other steps, then drill into inputs, outputs, token use and latency. That record is meant to make an unexpected answer or path inspectable rather than a mystery hidden behind a final response.

From a local run to a shared investigation

AWS is offering Omni through an extension for VS Code and Kiro, plus a standalone web experience outside the AWS Management Console. A developer can use it locally without an AWS account, then optionally connect to AWS to retain telemetry, share traces and monitor production systems. The IDE extension is free to use.

For operations, AWS says users can reach the web application through a dedicated organization URL and enterprise identity management without AWS Management Console access. Existing CloudWatch telemetry can appear in Omni without reconfiguration, according to AWS, while administrators can organize access through IAM Identity Center and team Spaces.

Testing changes before they become regressions

The evaluation layer is the other half of the launch. Omni includes 17 built-in evaluators, including measures for coherence, helpfulness, faithfulness and routing correctness. Users can select traces for scoring, turn production traffic into test datasets, and run the same dataset against agent variants to compare scores, latency and token use.

What teams can compare inside Omni

• Prompt versions, with the option to track configurations and roll back an underperforming version.
• Two execution traces side by side, to locate behavioral differences after a prompt or configuration change.
• Agent variants against a shared dataset, using evaluation scores alongside latency and token usage.

AWS says Omni supports LangChain, LangGraph, CrewAI, the OpenAI SDK, Strands and Vercel AI SDK in Python and TypeScript. Its instrumentation uses OpenInference and AWS Distro for OpenTelemetry, which AWS says can cover agents running on AWS services or other clouds. It also integrates third-party evaluators including AutoEval and DeepEval.

The unresolved judgment is what to score

Omni can make more of an agent’s behavior measurable, but the launch does not remove the underlying judgment about quality. Built-in or third-party scores can help flag a regression; teams still need to choose which dimensions matter for their application and whether a score is strong enough to approve a change. AWS has supplied the testing workflow. The next move is deciding how much authority to give its evaluations.