Our Analysis Finds VCB 1–100 Scores Don’t Prove Coding-Agent Regression

The September Vibe Code Bench 1–100 leaderboard put every listed model below 30%. That result does not show coding agents got worse. Our analysis finds that the new score measures a materially different job: extending a working application through dependent changes while preserving earlier behavior, rather than building an application from a specification.

The low figures are striking. Claude Opus 5 leads the published VCB 1–100 test snapshot at 28.53%, followed by Claude Fable 5.1 at 28.00% and GPT-6 Astra at 27.64%. Beside the earlier Vibe Code Bench result, where the leading model passed 61.8% of workflows, those columns can look like a collapse. They are not a valid before-and-after comparison.

The earlier score answered a different opening question

The earlier benchmark asked an agent to build a complete web application from a natural-language specification. It covered 100 specifications and 964 browser workflows. Each application was generated independently and then tested through user-facing workflows.

VCB 1–100 begins after that opening job is complete. It starts with a fully passing VCB v1.1 application, then gives an agent up to ten ordered product requests. The same codebase, application state, conversation and tool history persist through the sequence, so later requests can depend on earlier features, permissions, data models and user journeys.

A new rule made preservation part of the result

The change is not just a longer prompt. Earlier Vibe Code Bench credited a workflow when at least 90% of its substeps passed, then averaged workflow accuracy. VCB 1–100 credits an iteration only if all its workflows pass, and it stops a scenario after the first incomplete or incorrect iteration.

That makes regression prevention central. Each iteration combines checks for the requested new behavior with selected tests for earlier behavior against the same submitted app. An agent can add the requested feature and still receive no credit if it breaks an earlier flow. The newer dataset includes 11,299 workflows; 3,747 are regression checks, although they do not retest every prior flow.

The operating conditions moved too

The available generation time also changed. The earlier benchmark allowed five hours for each independently generated application. VCB 1–100 gives a task one ten-hour generation budget shared across all of its iterations. That shared allowance covers understanding an existing codebase, making successive changes and maintaining the app through the sequence.

Reset behavior differs as well. Earlier workflows used fresh browser sessions and isolated workflow data. The newer task retains the evolving code and application state alongside the agent’s conversation and tool history. Those conditions may surface an important kind of software-maintenance failure, but they are not equivalent to the independent build setting.

The apparent overlap cannot settle the question

Most leaderboard rows represent different model generations. Gemini 3.1 Pro Preview is an apparent label overlap, showing 32.03% on the earlier board and 6.69% on VCB 1–100. But the task, test content, scoring, budget and state rules changed together. The pair therefore cannot isolate a model-only effect, and the shared label does not establish an identical provider snapshot, routing path or backend configuration.

A clean comparison is further limited by missing public linkage between the earlier OpenHands scaffold and the VCB 1–100 harness. A separate GitHub issue records that VCB 1–100 grading responses could carry a null evaluator version despite a real version at the service endpoint. That is an audit-trail limitation, not evidence that the reported scores are wrong.

The next move is a matched crosswalk

This analysis does not establish that VCB 1–100 is harder in every operational sense, nor does it make the new results uninformative. The scores show that completing long chains of dependent product changes without breaking selected prior behavior remains difficult under this protocol. They should not be treated as a general measure of coding ability or direct proof of deterioration from the earlier benchmark.

What would resolve the issue is narrower than a new leaderboard: matched reruns using the same provider snapshot and documented harness under both protocols, or a public version-and-harness crosswalk. Until then, VCB 1–100 is best read as a distinct test of sustained software change, not as a replacement score for the earlier build-from-scratch benchmark.