Research investigation R1108 / claim audit
Qwen’s reported math gain supports edit discovery—not an editor ranking
The original experiment supports effective edit discovery within its reported run. The sampled follow-ups provide distinct evidence for artifact verification, behavioral replication and known-edit reconstruction, but do not establish a matched ranking of editors or independent scripted-search superiority.
This synthesis concerns the supplied public Qwen3.5-0.8B comparisons, not all weight-editing systems. Supplied snapshots were retrieved at 21:31:47.673Z, after the requested November 8 cutoff; mutable pages are not immutable cutoff captures. Recovering those records does not constitute a new collection or experiment.
This synthesis concerns the supplied public Qwen3.5-0.8B comparisons, not all weight-editing systems. Supplied snapshots were retrieved at 21:31:47.673Z, after the requested November 8 cutoff; mutable pages are not immutable cutoff captures. Recovering those records does not constitute a new collection or experiment.
- Dataset ID
- spd:qwen-s-weight-editing-gains-which-public-comparisons-test-discovery-rather-than-reconstruction-b59754c3
- Stable URL
- /research/qwen-s-weight-editing-gains-which-public-comparisons-test-discovery-rather-than-reconstruction-b59754c3
- Version
- Live
- Coverage
- Live collection
- Records
- 0
- Fields
- 7
- Updated
| Entity | Metric | Value | Unit | Observed | Source | Transform |
|---|
Measurement technique
How to read this report
- 01Use the supplied bounded documentary audit: five supplied documents plus the Experiment 8 README and reproduction script, with no experimental reruns. Preserve the requested evidence cutoff of 2026-11-08T21:31:04.936Z.
- 02Build an evidence_matrix with rows for original edit discovery, artifact verification, behavioral replication, known-edit reconstruction, decoding alternatives and blinded discovery attempts. Columns should identify source, inherited choices, task split, sampling, budget, measured outcome, matching status and supported inference.
- 03Original discovery row: the original post and Experiment 8 documentation report practice-based edit selection followed by unseen MATH-500 evaluation, with accuracy rising from 7.9% to 48.1%. Classify checkpoint comparison as matched under unchanged decoding; restrict the discovery inference to the reported run.
- 04Artifact-verification row: statsleelab’s inspected Experiment 8 script examines released edits and recomputes stored evaluation counts. Classify it as artifact verification, not a fresh search or behavioral evaluation.
- 05Behavioral-replication row: the follow-up manuscript reports that hash-matched released tensors raised accuracy from 10.2% to 64.8% and reduced truncation from 90.2% to 15.2% on 64 paired practice problems, with four samples each. Treat this as manuscript-reported behavioral replication, not held-out benchmark replication.
- 06Reconstruction row: the manuscript reports 60.9% accuracy and recovery of 93% of Astra’s gain, but the script inherited layers 15/19, scale 6 and 64 columns per layer. Classify the practice-set comparison as matched for behavior but inconclusive for independent setting discovery or equivalent performance.
- 07Decoding row: loop interruption reportedly reached 47.7% on the same paired practice evaluation. Classify this as a matched generation-policy alternative, not a weight edit; only three decoding alternatives were tested.
- 08Blinded-search row: four Opus/Sonnet runs improved development scores using self-chosen problems and stopped after 18–23 of 25 requests. Classify editor ranking as inconclusive because harnesses differed, best-development selection was unadjusted and shared-set evaluation remained undone.
Sources
Evidence
5 publishers supporting 0 records. Expand a publisher to inspect its cited pages.