Qwen’s reported math gain supports AI edit discovery—not an editor ranking

An AI appears to have discovered an effective repair for Qwen3.5-0.8B—not merely copied one. But the subsequent comparisons do not establish which AI is the better editor. Superpower Daily’s documentary audit finds support for discovery within the original reported run, while separating that result from checks and follow-up tests that answer narrower questions.

We examined the original research post, Experiment 8 documentation, statsleelab repository and reproduction script, follow-up manuscript, and Qwen model card. The question was whether each comparison tested finding an edit, rebuilding one, changing generation settings or improving practice scores. The requested evidence cutoff was November 8, 2026, at 21:31:04 UTC. This was a bounded document review, with no experimental reruns. All experimental outcomes below remain attributed to their authors; our finding concerns what their methods support, not independent confirmation of their results.

The original run did search for a repair

The catalyst was the November 2 research post Sparks of Weight-Space Programming, by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. In Experiment 8, their Astra editor received access to Qwen’s weights—the numerical values governing its behavior—and practice problems. It could inspect internal activity and submit direct weight changes, but could not use gradient descent, the procedure conventional training uses to calculate updates. Its task was to improve an existing model, not reconstruct a supplied solution.

The crucial separation was between practice and evaluation. According to the Experiment 8 documentation, Astra chose its edits using training-split problems and never saw the benchmarks. The submitted model was evaluated afterward, with the same generation settings as the unedited model. MATH-500 used four samples per problem. That design supports a discovery claim within this run: the editor selected a repair before seeing the test results, and the edited checkpoint improved under unchanged conditions.

The final repair targeted repetition, changing 64 columns in each of two internal weight matrices, in layers 15 and 19. It touched 131,072 weights. Finding it was not a single calculation: the strongest run used 50 model requests, 158 conversational turns and 80 Python runs. One request could batch as many as 512 generations or inspect 1,024 texts. The separate 10-, 25- and 50-request runs produced different edits; those budgets are not standardized units of computation.

The reproduction label covers different work

The statsleelab repository describes a bit-exact reproduction, but its inspected Experiment 8 script does something narrower than repeat Astra’s investigation. It examines released tensor differences and recomputes benchmark totals from stored score files. It does not launch a fresh editor search or generate fresh Qwen answers. That is artifact verification: useful checking of the released objects and arithmetic, not an independent demonstration that another editor can discover the repair. This classification applies to that script, not every demonstration in the repository.

A separate follow-up manuscript, Replicating Astra’s Weight Edit in Qwen3.5-0.8B, goes further by reporting fresh behavioral tests. Its authors rebuilt the released edit and matched both edited tensor hashes, digital checks of their contents. On 64 practice problems, with four samples each, accuracy rose from 10.2% to 64.8%, while truncation fell from 90.2% to 15.2%. Conditions used paired problems and sampling seeds. This supports behavioral replication on that practice sample—not replication of the original held-out benchmark result.

A reconstruction inherited consequential choices

The follow-up also reports a script reaching 60.9% accuracy and recovering 93% of Astra’s observed gain. But the script inherited layers 15 and 19, an edit-strength setting of six, and 64 columns per layer. It estimated its own internal directions and selected its own columns using 128 additional practice problems, separate from the evaluation sample. This is a meaningful test of rebuilding a known approach, not independent evidence that the script would have discovered those settings.

Those inherited choices mattered. An automatic layer-selection rule reached 40.2% accuracy, still borrowing Astra’s strength setting and update form. Nor does 93% establish equal performance: the manuscript’s estimated script-minus-Astra difference had a 95% interval from −9.6 to +1.8 percentage points. That leaves room for a close result or an appreciably worse one.

The same paired evaluation also tested three decoding changes—changes to how answers are generated, without altering weights. The best detected a repetition loop, interrupted it and closed the thinking block, reaching 47.7% accuracy. The released weight edit scored higher on this sample, but three alternatives do not amount to an exhaustive search for optimized decoding. The 47.7% practice score also cannot be placed beside the original 48.1% MATH-500 result as though they measured the same task.

Qwen’s model card adds a baseline qualification. It recommends temperature 1.0 for text thinking and 81,920 tokens for complex benchmarks, versus the experiment’s 0.6 and 32,768. Those recommendations do not demonstrate equivalent accuracy. They do limit the conclusion: the original comparison isolates a checkpoint change under fixed settings, not superiority over an optimized generation policy. Reduced truncation can raise answer scores without establishing added mathematical knowledge; accuracy and truncation alone cannot isolate the mechanism.

Blinded searches have not produced a common finish line

The follow-up’s four blinded searches address discovery more directly. Opus 5.5 and Sonnet 5.5 each ran with two prompts, without seeing Astra’s edit or the follow-up results. All improved their development scores, but stopped at a spending limit after 18–23 of 25 allowed requests. Their environment also differed from Astra’s, including memory and output caps and reasoning settings. Each editor chose its own test problems, so the reported gains do not provide a common basis for ranking them.

The displayed editor gains were each run’s best observed development result, without adjustment for that selection. Shared-set evaluation of stored candidates remained undone, as did the follow-up’s MATH-500 and GSM8K-test comparisons. Its new supporting experimental records were unreleased. Three outstanding pieces therefore deserve attention next:

• Held-out results for the follow-up methods, separating gains on its paired practice sample from performance on the original benchmarks.
• A shared evaluation of the blinded editors’ stored candidates, rather than comparisons between separately chosen development problems.
• The follow-up’s supporting records, allowing closer scrutiny of its reconstruction, decoding alternatives and searches beyond the manuscript’s reported outcomes.