Superpower Daily

Superpower Daily

Your reading space

Sign in
LlamaIndex Begins Stainless Exit, Seeking to Preserve SDK Inter...Apple Releases macOS 27 With a Siri That Can Read Your ScreenAudit Finds No SWE-bench Pro Leaderboard Score Has a Complete P...Salesforce Connects Gemini Enterprise to Its Data Without Custo...Emergence Finds AI Agents Invent Opaque Shared Dialects in Coop...Meta Launches Meta One Subscriptions for More AI Across Its AppsLlamaIndex Begins Stainless Exit, Seeking to Preserve SDK Inter...Apple Releases macOS 27 With a Siri That Can Read Your ScreenAudit Finds No SWE-bench Pro Leaderboard Score Has a Complete P...Salesforce Connects Gemini Enterprise to Its Data Without Custo...Emergence Finds AI Agents Invent Opaque Shared Dialects in Coop...Meta Launches Meta One Subscriptions for More AI Across Its Apps
Swipe

Research investigation R0915 / claim audit

After the Answer-Key Leak: Which SWE-bench Pro Scores Still Identify a Verifiable Dataset Version?

The public record establishes leakage and task-quality concerns, but the decisive procurement question is score-level provenance. In the bounded public sample, no visible SWE-bench Pro row preserves the complete metadata chain needed for strict historical or pre/post-remediation comparison.

Current public editionv1Sep 15, 2026
Read the original analysis
Verified observations
30

6 measured fields

Supported claims
9

9 material findings

Cited sources
7

7 primary or authoritative

Research score
85

Automated topic and evidence score

Interactive figureAfter the Answer-Key Leak: Which SWE-bench Pro...
CSV JSON
Data status0 verified records across 1 period

Snapshot only. There is not enough history to claim a trend yet.

Verified observationHover or focus any mark for exact valuesLast updated Sep 15, 2026

Version ledger

Frozen public editions

Each edition preserves the records, method, sources, and downloads available at publication time.

  1. v1 / latestSep 15, 202630 records / 7 sources

    Initial public snapshot with 30 records and 7 cited sources.

Coverage note

This is a frozen public-record audit at the September 15, 2026 cutoff, not a rerun or a claim about unpublished vendor records. The sample is limited to 25 visible public leaderboard rows.

Dataset ID
spd:after-the-answer-key-leak-which-swe-bench-pro-scores-still-identify-a-verifiable-dataset-version-0063ec17
Stable URL
/research/after-the-answer-key-leak-which-swe-bench-pro-scores-still-identify-a-verifiable-dataset-version-0063ec17
Version
v1
Coverage
2026-09-15
Records
30
Fields
7
Updated

Read the data

The records behind the figure

CSV JSON
After the Answer-Key Leak: Which SWE-bench Pro Scores Still Identify a Verifiable Dataset Version? data records
EntityMetricValueUnitObservedSourceTransform
Visible leaderboard rowsrows_explicitly_marked_pre_or_post_remediation0 of 25rows2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_publicCounted rows explicitly labeled as predating or following leakage/task-quality remediation.
Visible leaderboard rowsrows_with_immutable_dataset_and_container_revision0 of 25rows2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_publicCounted rows naming both an immutable dataset/task revision and immutable container digest or revision.
Visible leaderboard rowsrows_with_immutable_harness_and_tool_policy0 of 25; five rows identify mini-swe-agent only at family levelrows2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_publicCounted rows pinning harness revision/configuration and tool or internet policy.
Visible leaderboard rowsrows_with_public_run_date0 of 25rows2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_publicCounted visible rows containing a row-level run date.
Visible leaderboard rowsrows_with_unambiguous_row_specific_downloadable_run_artifact0 of 25rows2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_publicCounted rows linked to a uniquely identifiable downloadable prediction or trajectory artifact.
claude-4-5-haikuvisible_resolve_rate39.45%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
claude-4-5-Sonnetvisible_resolve_rate43.60%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
claude-4-Sonnetvisible_resolve_rate42.70%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
claude-opus-4-5-20251101visible_resolve_rate45.89%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
claude-opus-4-6 (thinking)visible_resolve_rate51.90%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
codestral-2405visible_resolve_rate1.51%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
deepseek-v3p2visible_resolve_rate15.56%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gemini-3.1-pro (thinking)visible_resolve_rate46.10%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gemini-3-flashvisible_resolve_rate34.63%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gemini-3-pro-previewvisible_resolve_rate43.30%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gemma-3-27b-itvisible_resolve_rate11.38%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
glm-4.6visible_resolve_rate9.67%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gpt-5.2visible_resolve_rate29.94%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gpt-5-2025-08-07 (High)visible_resolve_rate41.78%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gpt-5.2-codexvisible_resolve_rate41.04%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gpt-5.4 (xHigh)visible_resolve_rate59.10%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
gpt-oss-120bvisible_resolve_rate16.20%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
kimi-k2-instructvisible_resolve_rate27.67%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
llama3-1-405b-instructvisible_resolve_rate11.18%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
llama4-maverick-17b-instructvisible_resolve_rate5.24%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
minimax-2.1visible_resolve_rate36.81%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
Muse Sparkvisible_resolve_rate55.00%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
Muse Spark 1.1visible_resolve_rate61.50%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
qwen3-235b-a22bvisible_resolve_rate21.41%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—
qwen3-coder-480b-a35bvisible_resolve_rate38.70%; strict reproducibility status: inconclusivepercent2026-09-15https://labs.scale.com/leaderboard/swe_bench_pro_public—

Measurement technique

How to read this report

  1. 01Evidence matrix plan: treat the 25 rows visible on Scale’s public leaderboard at the September 15, 2026 cutoff as the bounded sample.
  2. 02For each score, test for five public receipts: row-level run date; immutable dataset/task manifest; immutable container or image revision; pinned harness, configuration, and tool or internet policy; and a row-specific downloadable run artifact.
  3. 03Record remediation status separately: whether a row is explicitly marked pre- or post-remediation for git-history leakage or task-quality changes.
  4. 04Compare visible leaderboard disclosures with Scale’s repository, historical leaderboard, dataset commit history, issue and pull-request records, OpenAI’s task-quality audit, and SWE-Bench Pro Verified.
  5. 05Classify a score as strictly reproducible only if the complete public receipt chain is present; otherwise classify it as inconclusive rather than invalid.
Next report / 01AI Model Economics Index All research reports