Snowflake says tailored tools raised Claude Code’s troubleshooting pass rate to 69% in internal test

Giving an AI agent access to operational data did not make it a reliable troubleshooter in Snowflake’s internal test. In findings published October 19, the company said specialized software interfaces raised Claude Code’s pass rate from 7.8% to 69%. The improvement came from helping the agent find relevant data and perform common diagnostic checks.

From finding logs to checking health

The experiments focused on Observe by Snowflake and questions an on-call engineer might ask: why a service is slow, whether it is degrading, or which systems it depends on. Answering requires interpreting telemetry—the logs, metrics and traces that record how software behaves—not simply retrieving a matching document.

The baseline already allowed raw queries and keyword searches. But the agent still had to work out what names meant, locate the right datasets and write queries that could explain a problem. Snowflake organized its additions into two layers of application programming interfaces, or APIs:

• Grounding APIs connect a question’s terms to specific entities and their data. A search by meaning can map “payments” to a service called “checkout.” Indexes then identify which datasets and metrics contain that service’s records. Adding this layer raised the pass rate to 26.2%.
• Domain APIs handle common diagnostic operations, such as assessing service health and mapping dependencies, without requiring the agent to construct those calculations itself. Their responses also preserve the dataset, metric and time window needed to investigate a finding further.

In Snowflake’s illustrated example, raw-query access helped the agent identify slow checkout. With both API layers, it followed the investigation to an orders database and used a targeted query to find an exhausted connection pool. The specialized tools guided the investigation; they did not eliminate raw queries.

What counted as a correct answer

The evaluation covered 56 curated questions, ranging from locating a service’s data to investigating failures. Each had a reference answer and grading rubric. Snowflake ran each prompt three times, producing 168 trials per configuration; the published pass rates measure the share of those runs that passed.

According to the benchmark methodology, Claude Code used claude-opus-4-7 at high reasoning effort through Snowflake’s MCP server, which connects the agent to tools. A separate AI judge, Cortex gpt-5.4, scored answers against the references on a five-point scale. Scores of four or five—substantially correct answers—counted as passes.

That makes the headline result a company-run measurement on a defined question set, with AI-based grading, rather than a general accuracy rate for production troubleshooting. Snowflake also uses this evaluation in its automated software checks: API changes must meet a quality threshold before engineers can merge them.

The missing operation mattered most

Snowflake also removed capabilities one at a time to see which mattered most. Removing domain operations caused the largest drop, returning performance to the grounding-only level. The agent could still find entities and relevant data, but had to reconstruct service health and dependencies from raw records.

Finding the right data remained important. Removing the connections between entities and datasets lowered success to 51.2% and increased token use by about 30%. All nine dataset-search prompts fell below the passing threshold. Removing entity resolution instead lowered success to 54.8%. These effects overlap and cannot be added together.

Preserving context between investigation steps had a smaller measured effect: removing it lowered success to 65.9%. Snowflake has not yet measured its value across an engineer’s follow-up questions, where that context may matter more. That remains an open test beyond the benchmark’s individual investigation prompts.