Cognition releases tougher coding benchmark, with top model scoring 45.5%

Cognition released FrontierCode 2.0 on October 27 to test a harder part of AI coding: making changes that fit a repository when users have not spelled everything out. Claude Opus 5.5 led the company’s results at 45.5%. The benchmark grades correctness alongside design choices and requirements inferred from the surrounding code.

That percentage needs careful reading. FrontierCode scores are weighted totals across grading requirements, not simply the share of tasks completed. A solution that fails a blocking requirement receives zero. The leading score therefore does not mean AI coders fail roughly half of all engineering jobs; it describes performance under Cognition’s particular tasks and grading rules.

Cognition contrasts its approach with benchmarks that give agents extensive specifications. Its example from Terminal-Bench 4.0 spells out supported inputs, excluded features, correctness checks and a speed target. Cognition calls that useful for isolating model intelligence, but wants FrontierCode to capture something else: whether an agent can turn an informal request into an acceptable contribution.

That requires more than making the requested feature work. FrontierCode evaluates whether changes reuse existing abstractions, follow repository conventions and include tests that actually catch bugs. Its expanded emphasis on thoroughness targets fixes that appear to answer the prompt but leave important functional problems behind. Cognition gives two examples:

• Fixing related bugs: a prompt identifies a defect in one code path, but a thorough agent finds and repairs the same defect in a neighboring path.
• Matching existing support: a request mentions one input type, but a thorough agent notices that similar code accepts others and supports those too.

The grading shift is substantial. Blocking checks for thoroughness rose from 168 in FrontierCode 1.1 to 501 in version 2.0. In results pooled across four models, thoroughness had the highest blocking-check failure rate at every reasoning setting—even above checks for core behavior.

The lower scores accompany changes to both the work and the judging. Version 2.0 adds broader task coverage, including security patches and performance improvements, with stricter correctness and design requirements. Compared with version 1.1, reference solutions modify 67% more files and 89% more lines on average, in codebases that are 25% larger.

Cognition says more than 20 open-source developers built evaluations from repositories they maintain, spending over 40 hours per task. Their objective was to judge whether an agent’s output could actually be accepted into the codebase, rather than merely satisfy the literal request.

The reference answers also received additional scrutiny. Cognition says human and automated audits uncovered subtle edge cases in solutions taken from upstream repositories. In 80% of version 2.0 tasks, the reference solution differs from the original pull request, up from 60% in version 1.0.

Yet asking agents to infer more does not mean rewarding every additional change. Cognition says higher reasoning settings can reduce scores: models tend to write more code and touch more files, including changes users may not prefer. Passing more implementation checks can be offset by failures on rules that limit the work’s scope.

Cognition loosened its scope rules after finding that some unexpected solutions were reasonable but still failed. Averaged across effort levels, the scope-check failure rate fell from 9.0% in version 1.1 to 3.8% in version 2.0. The company acknowledges the competing interpretation: going beyond a prompt can sometimes improve the code.

That tension remains in the new results. Across four pooled models, increasing reasoning effort from low to maximum reduced failures on thoroughness checks from 26.0% to 15.3%. But scope-check failures rose from 3.8% to 4.9%. More effort helped agents catch missing work while making them somewhat more likely to do unwanted work.