Loading page…
Holding models fixed exposed large cost differences—and a performance shortfall for OpenCode. Repeated runs also showed why small benchmark leads can be hard to trust.
Listen to this story
A Shandong University study submitted October 3 finds that coding-agent harnesses can produce similar benchmark scores while changing API costs substantially. On the 447-task comparison, Claude Code and mini-SWE-agent were within five percentage points, a margin the researchers treated as statistically equivalent—not proof of identical performance. OpenCode scored lower, though token limits and a tighter time limit complicate that comparison. For teams choosing agent infrastructure, the results suggest measuring cost and success on the exact model–harness pairing rather than assuming a more elaborate or leaner wrapper will perform better.
The study covered five models on 45 difficult SWE-bench Verified tasks; only two Qwen models were also tested on the 447-task pool.
On the hard-task set, changing harnesses altered pass/fail outcomes at a median rate of 13%, similar to ordinary rerun variation.
With GLM-5.3-Flash, Claude Code cost nearly twice as much as mini-SWE-agent and more than three times as much as OpenCode; OpenCode also made fewer calls.
The software wrapped around an AI coding model can make a task cost three times as much without delivering a consistent accuracy gain. A Shandong University study, detailed by VentureBeat on October 23, found similar results for Claude Code and a minimalist alternative, while a third system, OpenCode, fell behind on the larger test set.
The software surrounding a coding model is known as a harness: it supplies instructions and tools, and manages conversation history, so the model can work in a code repository. Researchers Yangze Liu and Zhongyi Han compared Claude Code, OpenCode and mini-SWE-agent while keeping the models fixed. Their paper, What Does a Harness Buy? Tokens, Mostly, was submitted on October 3.
The contrast was substantial. mini-SWE-agent consists of about 100 lines of Python, offers one bash tool for running commands and does not compress conversation history. The researchers tested five models on 45 difficult SWE-bench Verified software-repair tasks; only two Qwen models also ran on the larger pool of 447 tasks. Claude Opus 5 was included as a reference only within Claude Code.
On the 447-task pool, Claude Code and mini-SWE-agent finished less than two percentage points apart on both models. The researchers found them statistically equivalent within a five-point margin. That is a bounded result, not proof that the systems perform identically or that smaller differences cannot exist.
The hard-task results explain why that distinction matters. Switching harnesses changed whether a task passed or failed at a median rate of 13%. Simply rerunning the same configuration produced the same median rate. A harness also did not reliably keep winning the same tasks from one run to the next.
OpenCode was the important exception: it trailed by up to nine percentage points on the larger pool. On Qwen3.8-27B, roughly half the gap was associated with attempts cut short at a 32,000-token output limit, often before a patch was written. Claude Code encountered the same cap but automatically asked the model to continue.
The comparison also had an uncontrolled difference: OpenCode imposed a much tighter time limit on locally run models, according to The Decoder’s account of the study. Its poorer score should not be flattened into a claim that every harness was equivalent.
The cost mechanism starts before the agent has done much work. System instructions and tool definitions are sent again at each step, so their initial size is multiplied by the number of model calls. Tool output and the rate at which context grew differed much less between harnesses, the researchers found.
With GLM-5.3-Flash, a Claude Code trial cost nearly twice as much as mini-SWE-agent and more than three times as much as OpenCode. But OpenCode also stopped after fewer calls, contributing to its lower bill. Cheap attempts and successful attempts were not interchangeable measures.
Nor was the smallest harness always cheapest. On DeepSeek-V4-Flash, mini-SWE-agent used 141 model calls per trial against Claude Code’s 104. It ended up costing slightly more: about 15.2 cents versus 14.6 cents. The model’s behavior could outweigh the savings from a shorter preamble.
The researchers ultimately ran the evaluation offline after early web-enabled tests showed that 312 of 439 DeepSeek and Qwen trials had found the original human-written fix on GitHub. They excluded those runs and disabled network access, reasoning that the results reflected retrieval of an existing answer, not an unaided repair.
Sample size imposed another limit. At the variation observed, 45 tasks detected even a 13-point gap only about half the time. The 447-task comparison could resolve differences of roughly five points—still larger than many advertised harness improvements. The authors recommend publishing an identical-setup rerun alongside comparisons to expose ordinary variation.
Cost-cutting settings also depended on the exact model-and-harness pairing:
Loading discussion...
Join the conversation
Explain what would make the extra cost worthwhile.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.