CloudOpsBench finds reasoning gaps behind most AI failures in cloud tasks

AI agents can run successful cloud commands, check their work and still leave the job unfinished. In a November 2 failure analysis, the CloudOpsBench team attributed about 80% of failed tasks to reasoning gaps, not missing service knowledge. Its new benchmark examines 45 AWS operations tasks, including work where a wrong action can destroy data.

CloudOpsBench v1.0 tests DevOps: the work of maintaining, securing and changing cloud infrastructure. Rather than asking agents to answer questions about services, it gives them operational assignments. These include repairing failures, moving services, rotating cryptographic keys and shutting down projects. The authors distinguish this work from typical software-engineering tasks because it is safety-critical and difficult to grade with a conventional test suite.

The work agents were assigned

• Project decommissioning: 13 tasks involving shutting down projects and removing their resources.
• Security and access: 12 tasks focused on securing access to cloud resources.
• Migration and deployment: seven tasks covering moving and deploying services.
• Configuration and monitoring: six tasks; repair and recovery: another seven.

The team says the tasks produce identical results in real AWS and a high-fidelity emulator, a software recreation of the cloud environment. That allows agents to be evaluated without a real cloud account. The claimed match is an important part of the benchmark’s design: it offers a way to test operational decisions safely while retaining the behavior of the services those decisions affect.

The failure analysis separates three abilities: knowing how services behave, reasoning about the assignment and taking the right actions. The authors found no failures caused by misunderstanding a service in these tests. Instead, agents often surveyed only part of the environment, treated relevant resources as outside the assignment or stopped early. The team argues that more documentation or better retrieval would therefore likely not have changed these results.

A queue-deletion task shows the distinction. AWS has queues in SQS, a queue service, but also in MediaConvert, a video-processing service. In the illustrated Claude Fable 5.1 attempt, the agent found two project queues in SQS, checked 11 other regions and deleted both queues. Its commands succeeded. A subsequent check found no matching SQS queues, and the agent reported that none remained in the account. But it had never listed MediaConvert queues, where a project queue still existed.

The benchmark’s verifier failed the attempt because the requested cleanup was incomplete. The authors say every tested model knew MediaConvert could contain queues, yet no attempt by any model listed them. The missing step was not learning that another service existed; it was applying that knowledge when deciding where to look.

About 80%Reasoning gaps

The authors attributed about 80% of failed tasks to reasoning gaps, including incomplete searches and mistaken judgments about task scope.

About 20%Wrong actions

Wrong actions accounted for about 20% of failed tasks, including actions that broke something or left work unfinished.

Other mistakes were destructive rather than incomplete. During a bucket-migration task, agents encountered an in-progress multipart upload—a large file still arriving in pieces. They treated it as leftover data and aborted it. According to the analysis, that destroyed the data, and the upload could not be recovered. This example falls into the wrong-action category: the agent found the resource but chose an operation with an irreversible consequence.

The project’s leaderboard lists Claude Fable 5.1 at maximum reasoning effort, running through Claude Code, first with a 60.6% resolution rate. GPT-6 Astra at maximum effort, using Codex, follows at 58.9%. The table identifies both the model and its agent software, and also reports runtime, token use and cost. Those entries describe particular agent setups, not just model names.

The published figures do not fully agree. The failure-analysis introduction says Fable completed 29 of 45 tasks and Astra completed 26. Its chart instead shows pass rates of 60% and 53%, respectively; the leaderboard gives the different rates above. These should not be combined into one precise completion tally. The narrower conclusion holds across the published presentations: even the strongest tested agents left a substantial share of the benchmark unfinished.

CloudOpsBench is available on GitHub under the Apache 2.0 license, according to The Decoder. Extensions for Google Cloud, Azure and Oracle Cloud Infrastructure are planned. The current v1.0 findings concern the 45 AWS tasks, not those future environments.