Snowflake publishes coding benchmark showing cost gaps among closely matched agents

Snowflake published SnowPR on October 27, a coding-agent benchmark drawn from 65 real pull requests on its private code. In the company’s tests, four systems using opus-5 posted nearly identical pass rates. Its own internal REM mode cost the least, making the software around the model a clearer differentiator on price than accuracy.

Each system tried every task three times. Across the four opus-5 setups, 84.1% to 85.1% of trials passed—a spread Snowflake says 65 tasks cannot reliably distinguish. REM mode solved 62 tasks at least once. That is a three-attempt measure, not a claim that it completed 62 tasks on the first try.

Real changes, with the answers hidden

SnowPR spans 11 product areas. Sixty-two tasks come from Snowflake’s main shared code repository, and three from a separate Go developer-tools repository. The main repository contains 63,000 non-test source files and 17 million lines of non-test code, mostly Java and C++. This is depth within one company, rather than breadth across many organizations.

Agents receive the code as it stood before the change, plus a description of the work. They cannot see the original patch or its tests. After an agent finishes, Snowflake inserts those hidden tests and runs them through Bazel, its build system. Every graded test must pass.

Checks before a task enters the benchmark

• The original fix must pass, and submitting no change must fail.
• Each task needs at least one test of new behavior and 10 regression cases, which check that existing behavior still works.
• People verify every task. Eleven automatic checks and an adversarial review look for flaws; an AI model helps select candidates but does not assign the final score.

One tool changes the bill

The comparison tests both models and harnesses—the software that supplies an agent’s tools and manages its work. Cortex Code’s standard code mode provides separate file, shell and code-search tools. REM mode exposes operations as Python functions inside one interactive programming tool, letting the model chain several operations in a single step.

Against Cortex Code’s code mode, REM mode cost 23% less and used 14% fewer new tokens, but took longer. New tokens count input not served from cache plus generated output. REM mode remains internal only and is not available to customers.

Claude Code cost $6.51 per trial with opus-5, while Cursor cost $6.93. With the cheaper haiku-5.5 model, REM mode passed 70.8% of trials at $0.24 each, versus Claude Code’s 64.1% at $0.26. Still, every opus-5 setup passed 10 to 21 percentage points more trials than every haiku-5.5 setup.

The task itself can move the score

Repository complexity exposed weak spots. Averaged across seven agents, tasks crossing Java and C++ passed 55% of trials, versus 80% for Java alone. Only two tasks crossed that language boundary, so the result rests on a small slice of the benchmark. Changes touching three or more source files passed 70%, versus 89% for smaller changes.

Instructions also changed outcomes. Holding the hidden tests and correct fix constant, 10 agents passed 83% of trials with one engineer’s task description and 57% with another’s. That makes the wording part of what SnowPR measures, alongside the agent’s ability to navigate and edit code.

Grading is deliberately bounded. For one task, asking Bazel which tests a change affected returned 23,946 of 24,028 control-plane test targets. Running that many tests for every attempt was impractical, Snowflake says. SnowPR therefore grades against tests shipped with each pull request, rather than treating a passing attempt as verification across the entire codebase.