GPT-6 Astra leads coding test but finishes only 26.4% of tasks, The Decoder reports

GPT-6 Astra led ten coding models but fully satisfied just 26.4% of tasks, according to The Decoder’s November 8 coverage of Co-Design Bench. The study’s strongest model passed 69.9% of requirements per task on average under a shared setup—a sharp gap between completing individual features and delivering an application revision that meets the whole request.

The coverage also details differences between task types and working environments. Astra fully solved 41% of multimodal tasks, compared with just under 18% of repository tasks. In a separate experiment, changing the software around models could reverse their ranking.

Real requests, deliberately difficult cases

The Co-Design Bench paper, submitted to arXiv on October 26, 2026, contains 273 tasks drawn from 26,314 real-user coding sessions. According to The Decoder, researchers favored requests users had rephrased or changes they had rolled back. The collection therefore emphasizes troublesome cases, rather than representing all everyday coding work.

Multimodal Editing covers localized changes, sometimes using images, audio, video or documents. Repository Editing requires coordinated revisions across a codebase. One example involves extracting an amount from a receipt and changing every total that depends on it—not merely displaying the number.

Passing features is not finishing the job

The researchers grade revisions against 2,331 hidden, expert-written acceptance criteria—the conditions a successful change must meet. An AI judge uses source code and browser evidence; a separate language-model verifier challenges positive verdicts. Visual and interaction claims are also audited.

69.9%Average requirements passed per task

Partial credit captures individual requirements met.

26.4%Tasks fully satisfied

Complete success requires meeting every criterion.

All ten models completed fewer criteria on repository edits combining three or four change types than on those combining one or two. The categories include layout, design, interaction and content.

The software around the model changes the score

The main comparison used a shared harness: software through which agents read files, write changes and run code. With different harnesses, Qwen3.8-27B scored 75% or 56% on larger overhauls. The authors recommend reporting model and working environment together.

The benchmark gives each task one fixed request. It does not test follow-up questions or mid-task user corrections, and its AI judges can make mistakes. Its completion rate is not a measurement of how often people actually repaired AI-generated code.