{"title":"Qwen’s reported math gain supports edit discovery—not an editor ranking","description":"The original experiment supports effective edit discovery within its reported run. The sampled follow-ups provide distinct evidence for artifact verification, behavioral replication and known-edit reconstruction, but do not establish a matched ranking of editors or independent scripted-search superiority.","dataset_id":"spd:qwen-s-weight-editing-gains-which-public-comparisons-test-discovery-rather-than-reconstruction-b59754c3","canonical_url":"https://superpowerdaily.com/research/qwen-s-weight-editing-gains-which-public-comparisons-test-discovery-rather-than-reconstruction-b59754c3","version_url":"https://superpowerdaily.com/research/qwen-s-weight-editing-gains-which-public-comparisons-test-discovery-rather-than-reconstruction-b59754c3","version":"Live","snapshot_hash":null,"date_created":null,"date_modified":"2026-11-08T21:38:42.108Z","license":{"name":"Superpower Daily data reuse terms","url":"https://superpowerdaily.com/terms"},"license_url":"https://superpowerdaily.com/terms","coverage_note":"This synthesis concerns the supplied public Qwen3.5-0.8B comparisons, not all weight-editing systems. Supplied snapshots were retrieved at 21:31:47.673Z, after the requested November 8 cutoff; mutable pages are not immutable cutoff captures. Recovering those records does not constitute a new collection or experiment.","measurement_technique":["Use the supplied bounded documentary audit: five supplied documents plus the Experiment 8 README and reproduction script, with no experimental reruns. Preserve the requested evidence cutoff of 2026-11-08T21:31:04.936Z.","Build an evidence_matrix with rows for original edit discovery, artifact verification, behavioral replication, known-edit reconstruction, decoding alternatives and blinded discovery attempts. Columns should identify source, inherited choices, task split, sampling, budget, measured outcome, matching status and supported inference.","Original discovery row: the original post and Experiment 8 documentation report practice-based edit selection followed by unseen MATH-500 evaluation, with accuracy rising from 7.9% to 48.1%. Classify checkpoint comparison as matched under unchanged decoding; restrict the discovery inference to the reported run.","Artifact-verification row: statsleelab’s inspected Experiment 8 script examines released edits and recomputes stored evaluation counts. Classify it as artifact verification, not a fresh search or behavioral evaluation.","Behavioral-replication row: the follow-up manuscript reports that hash-matched released tensors raised accuracy from 10.2% to 64.8% and reduced truncation from 90.2% to 15.2% on 64 paired practice problems, with four samples each. Treat this as manuscript-reported behavioral replication, not held-out benchmark replication.","Reconstruction row: the manuscript reports 60.9% accuracy and recovery of 93% of Astra’s gain, but the script inherited layers 15/19, scale 6 and 64 columns per layer. Classify the practice-set comparison as matched for behavior but inconclusive for independent setting discovery or equivalent performance.","Decoding row: loop interruption reportedly reached 47.7% on the same paired practice evaluation. Classify this as a matched generation-policy alternative, not a weight edit; only three decoding alternatives were tested.","Blinded-search row: four Opus/Sonnet runs improved development scores using self-chosen problems and stopped after 18–23 of 25 requests. Classify editor ranking as inconclusive because harnesses differed, best-development selection was unadjusted and shared-set evaluation remained undone."],"methodology":["Use the supplied bounded documentary audit: five supplied documents plus the Experiment 8 README and reproduction script, with no experimental reruns. Preserve the requested evidence cutoff of 2026-11-08T21:31:04.936Z.","Build an evidence_matrix with rows for original edit discovery, artifact verification, behavioral replication, known-edit reconstruction, decoding alternatives and blinded discovery attempts. Columns should identify source, inherited choices, task split, sampling, budget, measured outcome, matching status and supported inference.","Original discovery row: the original post and Experiment 8 documentation report practice-based edit selection followed by unseen MATH-500 evaluation, with accuracy rising from 7.9% to 48.1%. Classify checkpoint comparison as matched under unchanged decoding; restrict the discovery inference to the reported run.","Artifact-verification row: statsleelab’s inspected Experiment 8 script examines released edits and recomputes stored evaluation counts. Classify it as artifact verification, not a fresh search or behavioral evaluation.","Behavioral-replication row: the follow-up manuscript reports that hash-matched released tensors raised accuracy from 10.2% to 64.8% and reduced truncation from 90.2% to 15.2% on 64 paired practice problems, with four samples each. Treat this as manuscript-reported behavioral replication, not held-out benchmark replication.","Reconstruction row: the manuscript reports 60.9% accuracy and recovery of 93% of Astra’s gain, but the script inherited layers 15/19, scale 6 and 64 columns per layer. Classify the practice-set comparison as matched for behavior but inconclusive for independent setting discovery or equivalent performance.","Decoding row: loop interruption reportedly reached 47.7% on the same paired practice evaluation. Classify this as a matched generation-policy alternative, not a weight edit; only three decoding alternatives were tested.","Blinded-search row: four Opus/Sonnet runs improved development scores using self-chosen problems and stopped after 18–23 of 25 requests. Classify editor ranking as inconclusive because harnesses differed, best-development selection was unadjusted and shared-set evaluation remained undone."],"metrics":[{"label":"Verified observations","value":"0","detail":"0 measured fields"},{"label":"Supported claims","value":"9","detail":"9 material findings"},{"label":"Cited sources","value":"6","detail":"6 primary or authoritative"},{"label":"Research score","value":"78","detail":"Automated topic and evidence score"}],"columns":[{"key":"entity","label":"Entity"},{"key":"metric","label":"Metric"},{"key":"value","label":"Value"},{"key":"unit","label":"Unit"},{"key":"observed","label":"Observed"},{"key":"source","label":"Source"},{"key":"transform","label":"Transform"}],"data":[],"sources":[{"name":"Author draft hosted on aiXiv","title":"aixiv-us.oss-us-west-1.aliyuncs.com","url":"https://aixiv-us.oss-us-west-1.aliyuncs.com/aixiv/prod/35771bb1-1fe5-428d-ab9d-1668fd5c106e_wsp_loopfix_author_draft.pdf","records":0},{"name":"statsleelab","title":"GitHub - statsleelab/weight_space_programming","url":"https://github.com/statsleelab/weight_space_programming","records":0},{"name":"Qwen","title":"Qwen/Qwen3.5-0.8B · Hugging Face","url":"https://huggingface.co/Qwen/Qwen3.5-0.8B","records":0},{"name":"RikiyaT/lm_understand-logs","title":"raw.githubusercontent.com","url":"https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md","records":0},{"name":"statsleelab","title":"raw.githubusercontent.com","url":"https://raw.githubusercontent.com/statsleelab/weight_space_programming/main/experiments/exp8_math/reproduce_exp8.py","records":0},{"name":"Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab","title":"Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights","url":"https://rikiyat.github.io/blog/weight-space-programming","records":0}],"provenance":{"publisher":"Superpower Daily","source_count":6,"source_urls":["https://aixiv-us.oss-us-west-1.aliyuncs.com/aixiv/prod/35771bb1-1fe5-428d-ab9d-1668fd5c106e_wsp_loopfix_author_draft.pdf","https://github.com/statsleelab/weight_space_programming","https://huggingface.co/Qwen/Qwen3.5-0.8B","https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md","https://raw.githubusercontent.com/statsleelab/weight_space_programming/main/experiments/exp8_math/reproduce_exp8.py","https://rikiyat.github.io/blog/weight-space-programming"],"methodology":["Use the supplied bounded documentary audit: five supplied documents plus the Experiment 8 README and reproduction script, with no experimental reruns. Preserve the requested evidence cutoff of 2026-11-08T21:31:04.936Z.","Build an evidence_matrix with rows for original edit discovery, artifact verification, behavioral replication, known-edit reconstruction, decoding alternatives and blinded discovery attempts. Columns should identify source, inherited choices, task split, sampling, budget, measured outcome, matching status and supported inference.","Original discovery row: the original post and Experiment 8 documentation report practice-based edit selection followed by unseen MATH-500 evaluation, with accuracy rising from 7.9% to 48.1%. Classify checkpoint comparison as matched under unchanged decoding; restrict the discovery inference to the reported run.","Artifact-verification row: statsleelab’s inspected Experiment 8 script examines released edits and recomputes stored evaluation counts. Classify it as artifact verification, not a fresh search or behavioral evaluation.","Behavioral-replication row: the follow-up manuscript reports that hash-matched released tensors raised accuracy from 10.2% to 64.8% and reduced truncation from 90.2% to 15.2% on 64 paired practice problems, with four samples each. Treat this as manuscript-reported behavioral replication, not held-out benchmark replication.","Reconstruction row: the manuscript reports 60.9% accuracy and recovery of 93% of Astra’s gain, but the script inherited layers 15/19, scale 6 and 64 columns per layer. Classify the practice-set comparison as matched for behavior but inconclusive for independent setting discovery or equivalent performance.","Decoding row: loop interruption reportedly reached 47.7% on the same paired practice evaluation. Classify this as a matched generation-policy alternative, not a weight edit; only three decoding alternatives were tested.","Blinded-search row: four Opus/Sonnet runs improved development scores using self-chosen problems and stopped after 18–23 of 25 requests. Classify editor ranking as inconclusive because harnesses differed, best-development selection was unadjusted and shared-set evaluation remained undone."],"snapshot_hash":null},"distributions":{"csv":"https://superpowerdaily.com/api/research/qwen-s-weight-editing-gains-which-public-comparisons-test-discovery-rather-than-reconstruction-b59754c3?format=csv","json":"https://superpowerdaily.com/api/research/qwen-s-weight-editing-gains-which-public-comparisons-test-discovery-rather-than-reconstruction-b59754c3?format=json"}}