{"title":"After the Answer-Key Leak: Which SWE-bench Pro Scores Still Identify a Verifiable Dataset Version?","description":"The public record establishes leakage and task-quality concerns, but the decisive procurement question is score-level provenance. In the bounded public sample, no visible SWE-bench Pro row preserves the complete metadata chain needed for strict historical or pre/post-remediation comparison.","dataset_id":"spd:after-the-answer-key-leak-which-swe-bench-pro-scores-still-identify-a-verifiable-dataset-version-0063ec17","canonical_url":"https://superpowerdaily.com/research/after-the-answer-key-leak-which-swe-bench-pro-scores-still-identify-a-verifiable-dataset-version-0063ec17","version_url":"https://superpowerdaily.com/research/after-the-answer-key-leak-which-swe-bench-pro-scores-still-identify-a-verifiable-dataset-version-0063ec17","version":"v1","snapshot_hash":"48687605deee4ea388f1759b7ef789801554b9ff0d53e81d619781e4fa11fce6","date_created":"2026-09-15T15:46:14.038Z","date_modified":"2026-09-15T15:46:14.038Z","license":{"name":"Superpower Daily data reuse terms","url":"https://superpowerdaily.com/terms"},"license_url":"https://superpowerdaily.com/terms","temporal_coverage":"2026-09-15","coverage_note":"This is a frozen public-record audit at the September 15, 2026 cutoff, not a rerun or a claim about unpublished vendor records. The sample is limited to 25 visible public leaderboard rows.","measurement_technique":["Evidence matrix plan: treat the 25 rows visible on Scale’s public leaderboard at the September 15, 2026 cutoff as the bounded sample.","For each score, test for five public receipts: row-level run date; immutable dataset/task manifest; immutable container or image revision; pinned harness, configuration, and tool or internet policy; and a row-specific downloadable run artifact.","Record remediation status separately: whether a row is explicitly marked pre- or post-remediation for git-history leakage or task-quality changes.","Compare visible leaderboard disclosures with Scale’s repository, historical leaderboard, dataset commit history, issue and pull-request records, OpenAI’s task-quality audit, and SWE-Bench Pro Verified.","Classify a score as strictly reproducible only if the complete public receipt chain is present; otherwise classify it as inconclusive rather than invalid."],"methodology":["Evidence matrix plan: treat the 25 rows visible on Scale’s public leaderboard at the September 15, 2026 cutoff as the bounded sample.","For each score, test for five public receipts: row-level run date; immutable dataset/task manifest; immutable container or image revision; pinned harness, configuration, and tool or internet policy; and a row-specific downloadable run artifact.","Record remediation status separately: whether a row is explicitly marked pre- or post-remediation for git-history leakage or task-quality changes.","Compare visible leaderboard disclosures with Scale’s repository, historical leaderboard, dataset commit history, issue and pull-request records, OpenAI’s task-quality audit, and SWE-Bench Pro Verified.","Classify a score as strictly reproducible only if the complete public receipt chain is present; otherwise classify it as inconclusive rather than invalid."],"metrics":[{"label":"Verified observations","value":"30","detail":"6 measured fields"},{"label":"Supported claims","value":"9","detail":"9 material findings"},{"label":"Cited sources","value":"7","detail":"7 primary or authoritative"},{"label":"Research score","value":"85","detail":"Automated topic and evidence score"}],"columns":[{"key":"entity","label":"Entity"},{"key":"metric","label":"Metric"},{"key":"value","label":"Value"},{"key":"unit","label":"Unit"},{"key":"observed","label":"Observed"},{"key":"source","label":"Source"},{"key":"transform","label":"Transform"}],"data":[{"unit":"rows","value":"0 of 25","entity":"Visible leaderboard rows","metric":"rows_explicitly_marked_pre_or_post_remediation","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":"Counted rows explicitly labeled as predating or following leakage/task-quality remediation."},{"unit":"rows","value":"0 of 25","entity":"Visible leaderboard rows","metric":"rows_with_immutable_dataset_and_container_revision","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":"Counted rows naming both an immutable dataset/task revision and immutable container digest or revision."},{"unit":"rows","value":"0 of 25; five rows identify mini-swe-agent only at family level","entity":"Visible leaderboard rows","metric":"rows_with_immutable_harness_and_tool_policy","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":"Counted rows pinning harness revision/configuration and tool or internet policy."},{"unit":"rows","value":"0 of 25","entity":"Visible leaderboard rows","metric":"rows_with_public_run_date","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":"Counted visible rows containing a row-level run date."},{"unit":"rows","value":"0 of 25","entity":"Visible leaderboard rows","metric":"rows_with_unambiguous_row_specific_downloadable_run_artifact","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":"Counted rows linked to a uniquely identifiable downloadable prediction or trajectory artifact."},{"unit":"percent","value":"39.45%; strict reproducibility status: inconclusive","entity":"claude-4-5-haiku","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"43.60%; strict reproducibility status: inconclusive","entity":"claude-4-5-Sonnet","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"42.70%; strict reproducibility status: inconclusive","entity":"claude-4-Sonnet","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"45.89%; strict reproducibility status: inconclusive","entity":"claude-opus-4-5-20251101","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"51.90%; strict reproducibility status: inconclusive","entity":"claude-opus-4-6 (thinking)","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"1.51%; strict reproducibility status: inconclusive","entity":"codestral-2405","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"15.56%; strict reproducibility status: inconclusive","entity":"deepseek-v3p2","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"46.10%; strict reproducibility status: inconclusive","entity":"gemini-3.1-pro (thinking)","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"34.63%; strict reproducibility status: inconclusive","entity":"gemini-3-flash","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"43.30%; strict reproducibility status: inconclusive","entity":"gemini-3-pro-preview","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"11.38%; strict reproducibility status: inconclusive","entity":"gemma-3-27b-it","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"9.67%; strict reproducibility status: inconclusive","entity":"glm-4.6","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"29.94%; strict reproducibility status: inconclusive","entity":"gpt-5.2","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"41.78%; strict reproducibility status: inconclusive","entity":"gpt-5-2025-08-07 (High)","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"41.04%; strict reproducibility status: inconclusive","entity":"gpt-5.2-codex","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"59.10%; strict reproducibility status: inconclusive","entity":"gpt-5.4 (xHigh)","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"16.20%; strict reproducibility status: inconclusive","entity":"gpt-oss-120b","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"27.67%; strict reproducibility status: inconclusive","entity":"kimi-k2-instruct","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"11.18%; strict reproducibility status: inconclusive","entity":"llama3-1-405b-instruct","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"5.24%; strict reproducibility status: inconclusive","entity":"llama4-maverick-17b-instruct","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"36.81%; strict reproducibility status: inconclusive","entity":"minimax-2.1","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"55.00%; strict reproducibility status: inconclusive","entity":"Muse Spark","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"61.50%; strict reproducibility status: inconclusive","entity":"Muse Spark 1.1","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"21.41%; strict reproducibility status: inconclusive","entity":"qwen3-235b-a22b","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null},{"unit":"percent","value":"38.70%; strict reproducibility status: inconclusive","entity":"qwen3-coder-480b-a35b","metric":"visible_resolve_rate","source":"https://labs.scale.com/leaderboard/swe_bench_pro_public","observed":"2026-09-15","transform":null}],"sources":[{"url":"https://arxiv.org/pdf/2609.08149","name":"arXiv","title":"arxiv.org","records":0},{"url":"https://github.com/scaleapi/SWE-bench_Pro-os","name":"Scale AI","title":"GitHub - scaleapi/SWE-bench_Pro-os: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?","records":0},{"url":"https://github.com/scaleapi/SWE-bench_Pro-os/issues/93","name":"GitHub / Scale AI repository","title":"Git Reward Hacking in SWEBench Pro OSS · Issue #93 · scaleapi/SWE-bench_Pro-os","records":0},{"url":"https://github.com/scaleapi/SWE-bench_Pro-os/pull/94","name":"GitHub / Scale AI repository","title":"Remove Future Git History from Dockerfiles by ConnorBAdams · Pull Request #94 · scaleapi/SWE-bench_Pro-os","records":0},{"url":"https://scaleapi.github.io/SWE-bench_Pro-os","name":"Scale AI","title":"scaleapi.github.io","records":0},{"url":"https://labs.scale.com/leaderboard/swe_bench_pro_public","name":"Scale AI","title":"SWE-Bench Pro (Public Dataset)","records":30},{"url":"https://openai.com/index/separating-signal-from-noise-coding-evaluations","name":"OpenAI","title":"Separating signal from noise in coding evaluations","records":0}],"provenance":{"publisher":"Superpower Daily","source_count":7,"source_urls":["https://arxiv.org/pdf/2609.08149","https://github.com/scaleapi/SWE-bench_Pro-os","https://github.com/scaleapi/SWE-bench_Pro-os/issues/93","https://github.com/scaleapi/SWE-bench_Pro-os/pull/94","https://scaleapi.github.io/SWE-bench_Pro-os","https://labs.scale.com/leaderboard/swe_bench_pro_public","https://openai.com/index/separating-signal-from-noise-coding-evaluations"],"methodology":["Evidence matrix plan: treat the 25 rows visible on Scale’s public leaderboard at the September 15, 2026 cutoff as the bounded sample.","For each score, test for five public receipts: row-level run date; immutable dataset/task manifest; immutable container or image revision; pinned harness, configuration, and tool or internet policy; and a row-specific downloadable run artifact.","Record remediation status separately: whether a row is explicitly marked pre- or post-remediation for git-history leakage or task-quality changes.","Compare visible leaderboard disclosures with Scale’s repository, historical leaderboard, dataset commit history, issue and pull-request records, OpenAI’s task-quality audit, and SWE-Bench Pro Verified.","Classify a score as strictly reproducible only if the complete public receipt chain is present; otherwise classify it as inconclusive rather than invalid."],"snapshot_hash":"48687605deee4ea388f1759b7ef789801554b9ff0d53e81d619781e4fa11fce6"},"distributions":{"csv":"https://superpowerdaily.com/api/research/after-the-answer-key-leak-which-swe-bench-pro-scores-still-identify-a-verifiable-dataset-version-0063ec17?format=csv","json":"https://superpowerdaily.com/api/research/after-the-answer-key-leak-which-swe-bench-pro-scores-still-identify-a-verifiable-dataset-version-0063ec17?format=json"}}