Leading coding agents finish fewer than 8% of assignments in SWE-CoWork study
The benchmark makes agents gather requirements from simulated colleagues. Giving two models the full instructions improved individual test results—not complete-task success.
SWE-CoWork tests coding agents across 101 real repositories by requiring them to gather scattered, sometimes conflicting requirements from simulated colleagues, implement changes, and verify delivery. The study finds that better information alone did not raise full-assignment completion: some agents improved on individual tests when given complete requirements but still failed to finish assignments. For teams evaluating coding agents, the results point to follow-up, implementation, and verification as distinct bottlenecks—not just code generation or prompt quality.
01
The benchmark’s simulated engineering organizations have 26 colleagues with different responsibilities and limited knowledge; they are not observations of agents working in real teams.
02
Complete requirements improved fail-to-pass test results by about 16 percentage points for Seed Evolving and GPT-6 Astra, but did not increase full-task success.
03
Claude-Opus-5.5 directed 93% of its first work-card questions to accountable owners but acquired only 64% of colleagues’ requirements.
Passing software tests and finishing an assignment were sharply different achievements in SWE-CoWork, a benchmark introduced in a November 15 research paper. Across 11 models, the two leaders passed about 63% of tests for required changes but completed fewer than 8% of assignments. The challenge included finding out what colleagues needed—not just writing code.
Researchers led by Xinping Lei built the study around a distinction: an agent must establish the requirements, carry out the changes and verify delivery. Their results locate failures at several points along that chain, rather than identifying missing instructions as the sole obstacle.
A brief handoff, not a complete specification
SWE-CoWork contains 101 development assignments across 101 real code repositories, covering 11 programming languages and 12 application domains. Each assignment places an agent inside a simulated engineering organization with 26 colleagues. Those colleagues have different responsibilities, limited knowledge and controlled collaboration behavior.
The agent starts with a brief handoff. Requirements are scattered among colleagues and workplace records, and some evidence conflicts. It must recover and reconcile that information, implement the changes, then face private executable tests. The repositories are real; the organizational setting is simulated, not an observation of agents employed in actual teams.
The accompanying dataset includes task instructions, execution budgets, starting code revisions and task-package paths. Its visible assignments illustrate how much work sits beyond producing a patch:
An ONNX maintenance assignment spans C++ and Python, requires compatibility and safety checks, and calls for baseline and post-change verification. It explicitly forbids presenting verification as complete by disabling tests.
A PostCSS assignment tells the agent to check renamed interfaces against recent records, contact the appropriate colleagues and check messages before finishing. Its final report must distinguish completed work, verification and unfinished items.
Knowing the requirements did not finish the work
One experiment separated information gathering from execution. Providing complete requirements upfront improved fail-to-pass results by approximately 16 percentage points for Seed Evolving and GPT-6 Astra. Yet complete-task success stayed unchanged. Better instructions helped those models satisfy more individual checks without increasing their rate of fully delivered assignments.
Finding the right person was also different from collecting everything needed. Across runs, Claude-Opus-5.5 directed a mean of 93% of its first questions about queried work cards to their accountable owners. But it obtained only 64% of the requirements held by colleagues. The first figure measures question routing; the second measures information acquired.
The researchers’ analysis of agents’ work traces found deferred questions that never received follow-up. It also found incomplete implementations after the relevant requirements had already been acquired. Together, these findings distinguish losing track of an unanswered question from failing to execute a known requirement—two different breaks in the same delivery process.
Verification exposed another gap: 31% of requirements agents reported as completed failed their tests. The study’s practical distinction is between an agent declaring work done and the delivered changes passing the checks. In SWE-CoWork, a completion report was not a reliable substitute for that final evidence.
Sources
huggingface.coTokenwave-AI/swe-cowork · Datasets at Hugging Face
arxiv.orgSWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
Reader comments
Newest comments first. Replies stay oldest first.