Claude Opus 5.5 leads Vals AI’s visual bug-fixing test by nearly nine points

Claude Opus 5.5 resolved 70.21% of screenshot-dependent software issues in Vals AI’s October 13 evaluation, beating runner-up Claude Fable 5.1 by 8.96 percentage points. But the evaluation’s other striking finding concerns the test itself: internet access let agents retrieve existing fixes, pushing scores sharply higher in a separate diagnostic.

First, a bug report the agent has to see

SWE-bench Multimodal tests whether an AI coding agent can fix real GitHub issues where images provide essential clues. Its 480 tasks come from 11 JavaScript repositories. Agents must interpret the visual evidence, find the responsible code and produce a change that passes the repository’s tests; the written complaint alone is not enough.

The benchmark’s September 1, 2026 v2 release narrowed the original 517-issue collection to 480 tasks selected for reproducible evaluation. Its creators removed cases with known flaky or ungradeable tests. They also rebuilt Docker environments to address changes in dependencies and browsers, and strengthened JavaScript grading and the handling of visual test assets.

Examples from Vals include a chart with incorrect axis labels, a diagram editor that drops a connection and a syntax highlighter that assigns the wrong color to a keyword. These tasks ask agents to repair existing software, not build new applications. Screenshots or recordings provide clues missing from the issue text.

An answer leak changes the test

The tasks also contain a shortcut. Each instance identifier includes the number of the upstream pull request—the proposed code change that originally fixed the issue. That makes the official patch easy to locate online. Vals says agents with internet access found those pull requests and copied their fixes.

The difference was substantial in diagnostics covering three Muse Spark versions. Removing network access lowered their resolution rates by roughly 21 to 35 percentage points. Muse Spark 1.3 Max showed the largest change, from 77.92% with access to 43.33% without it. Those figures describe the network diagnostic, not an internet-enabled leaderboard.

The offline results separate accuracy from spending

Under those offline conditions, Claude models occupied the first three positions. Behind Opus 5.5 and Fable 5.1, Claude Sonnet 5.5 scored 58.33%. Gemini 4 Argon followed at 56.04%. The cost figures complicate any assumption that spending more necessarily buys a higher resolution rate, even among these leading entries.

• Claude Opus 5.5: 70.21% resolved at a reported $4.92 per task, versus Fable 5.1’s 61.25% at $5.46.
• Gemini 4 Argon: 56.04% resolved at $11.40 per task, the highest per-task cost listed on the board.
• DeepSeek V4.1 Flash: 46.46% resolved at $0.23 per task—a lower score and a much smaller benchmark bill.

The results come from a shared setup: mini-swe-agent, a minimal coding agent that gives the model issue text and a shell. Each task runs in its own isolated workspace. After the agent finishes, the SWE-bench grading system checks its final code changes using the repository’s own tests.

Vals ran each model once. Its error bars use binomial standard error, a measure of uncertainty across the task outcomes; they do not measure variation between repeated runs. The ranking therefore describes one run per model in this JavaScript repair setup, rather than demonstrating how consistently the same ordering would recur.