A percentage on a coding-agent leaderboard looks like a clean comparison. On SWE-bench Pro’s public board, it is not currently a verifiable historical record. Our audit of all 25 rows visible on September 15 found that none can be tied through public materials to the complete chain needed to identify the exact evaluation behind it: a dated run, immutable task set and container revision, pinned harness and access policy, and a row-specific run artifact. That leaves every visible score inconclusive for strict comparison after concerns about answer leakage and task quality—not proven false.
That distinction is essential. An inconclusive score may reflect a legitimate run. Missing public receipts do not show that a model retrieved an answer, and they do not establish that a displayed resolve rate is numerically wrong. They do mean a reader cannot independently determine which version of the benchmark, environment, and rules produced a particular score, or reliably compare a result with later fixes and revisions.
What a score receipt needs
SWE-bench Pro asks an agent to produce a code patch for a real software task. Its Resolve Rate measures the share of tasks whose patches pass the new tests for the requested change while preserving pre-existing functionality. That makes the evaluation environment part of the result: changing the tasks, tests, containers, agent harness, tool access, or cost and turn limits can change what a score represents.
For a strict row-by-row comparison, the audit looked for five public receipts
- A run date, so a result can be placed before or after a known change.
- An immutable dataset or task manifest, including the evaluated subset.
- An immutable container or image revision, rather than only a mutable loading instruction or image tag.
- A pinned harness, configuration, and policy for internet and tool access.
- A downloadable artifact tied unambiguously to that one leaderboard row.
Why provenance became urgent
The central concern is not abstract reproducibility. In April, a public GitHub issue described future Git history remaining reachable in SWE-bench Pro’s public Docker images. The issue author said an agent could inspect future commits, branches, or tags to obtain or substantially derive a target solution. It included examples from public trajectories and reported successful exploitation across all scanned public images. A proposed cleanup pull request remained open at the audit cutoff.
This is an answer-key problem. A normal coding agent can use the working code, task description, and tests to reason toward a patch. If the environment also exposes the future commit that contains the patch, repository history can become an oracle. The public record does not allow a reader to connect any current leaderboard row to a dated environment that confirms whether that route was blocked.
A benchmark can drift without every score moving together
It would be a mistake to subtract one guessed “leakage bonus” from every result. A separate SWE-bench Pro Verified analysis reported paired protected evaluations in which GLM-5.2 fell 21.48 percentage points, while DeepSeek-V4-Pro moved by less than one point. That model-dependent difference suggests the effect of a vulnerability depends on model behavior and evaluation setup. It does not supply a correction factor for Scale’s 25 rows.
There is also evidence of ordinary benchmark drift. Scale’s older seven-row board says it used 730 problems; the current public dataset is described as 731 instances. The repository records removed outdated or unintended tests and changes to evaluation scripts, but does not identify which leaderboard rows were recomputed. A historical board can therefore contain scores measured over different denominators or tests without a public crosswalk explaining the difference.
The leaderboard gives partial clues, but not enough to close that gap. Five current rows are marked as using mini-swe-agent, while the page describes the other results through the SWE-Agent scaffold. None pins a harness commit, configuration, internet policy, or allowed-tool policy. Older trajectory links do not solve the problem either: all seven rows on the historical board point to one generic directory rather than a row-specific artifact.
Leakage is not the only reason percentages may not line up
OpenAI’s July audit raised a separate issue: task validity. It estimated that roughly 30% of the 731 public tasks were broken, based on both agent-assisted and human review paths. The company cited overly strict tests, underspecified or misleading prompts, and low-coverage tests, then retracted its earlier recommendation to adopt SWE-bench Pro. SWE-bench Pro Verified separately revised 102 instances. These audits use different methods and definitions, so their counts should not be treated as interchangeable.
They do, however, sharpen the same practical requirement. A result needs an explicit task manifest and a regrading crosswalk before it can be compared with a revised benchmark. Neither task-quality publication supplies a complete row-by-row regrade mapping for Scale’s historical leaderboard. A high score might be a valid result under its original conditions, a result affected by task selection, or something else entirely; the visible record cannot distinguish those cases.
What would make the board usable for hard comparisons
The remedy is more specific than publishing another ranking. Each row would need a dated, downloadable receipt: a task manifest; container digest or revision; harness commit and configuration; a stated internet and tool policy; limits such as cost and turns; and a row-specific trajectory or result artifact. Results should also identify whether they were run before or after changes addressing Git-history exposure, tests, or task definitions.
Reader comments
Newest comments first. Replies stay oldest first.