Study finds only 16% of 522 AI benchmarks separately score organizational demands

Strong AI benchmark scores are often read as evidence that models can do people’s jobs. A new study challenges that leap: across 522 prominent benchmarks, fewer than a quarter claimed to represent organizational demands, and only 16% separately measured one. The authors argue that solving a task and navigating a workplace are distinct capabilities.

The paper, Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking, was submitted to arXiv on November 2. Authors Victor Barres, Aden Barton, Nicolai J. Foss and Henrik M. Nielsen propose a framework for describing those demands. Mercor published their explanation of the findings on November 5.

What the audit counted

The sample combined three sources: benchmarks in the Artificial Analysis Intelligence Index and Epoch AI’s Benchmarking Hub; selected oral, spotlight and award papers at major machine-learning conferences; and benchmarks reported by at least two of five labs. Those labs were OpenAI, Anthropic, Google, xAI and Meta. Each benchmark counted once. The industry-index selections were tagged as of October 8, 2026.

The authors distinguished three levels of coverage. A benchmark paper could mention an organizational dimension, claim to build it into the evaluation, or measure it with a separate result. The six dimensions concerned interdependent activities, knowledge held by different people, differing authority, time constraints, conflicting interests and changing conditions. The headline percentages concern the latter two levels—not simply whether a paper discussed workplace complexity.

That distinction limits what the finding establishes. This is an audit of benchmark coverage, not a test showing that models fail at organizational work. The authors also caution that missing these dimensions does not make a benchmark easy or strip it of economically useful information.

A correct bug fix can still be blocked

The authors illustrate the gap with a software engineer fixing a customer-reported bug. The code change may be small, but support holds the logs, billing owns the affected code, and the product manager disputes whether the behavior is a bug. Meanwhile, an account manager has promised the customer a fix that week. This is an illustration, not a measured workflow.

Technical correctness alone does not settle whether a change is authorized. Yet when benchmarks represent authority, the authors find it is almost always a written policy in the prompt. Their workplace example instead involves overlapping authority and approval requirements that change as the engineer discovers which team’s code is affected.

From task conditions to workplace outcomes

The proposed taxonomy separates organizational difficulty into four layers. Rather than treating a job as a bundle of technical assignments, it traces how the conditions around an assignment create demands that must be managed:

• Structure of work: the conditions an actor faces, such as information spread across teams or mixed authority.
• Coordination requirements: what those conditions require, such as obtaining logs or agreeing on timing.
• Governance mechanisms: how an organization handles those needs, through arrangements such as ticket queues and release calendars.
• Performance outcomes: whether the work succeeds, including required approvals and delivery by the promised date.

The audit mainly tagged the first layer; labels for the other three were exploratory. Even among benchmarks claiming a first-layer dimension, papers explicitly described a governance mechanism or a grader check only about a third of the time. Where mechanisms appeared, they were usually simple, such as written rules supplied in a prompt.

The authors describe their taxonomy as preliminary and propose it as a shared vocabulary for future evaluations. Their argument is not that technical progress lacks value. It is that gains on well-specified tasks may translate into limited productivity gains unless agents also improve at handling the dependencies and decisions surrounding those tasks.