Mercor publishes AI benchmark map showing narrow coverage of U.S. work

Roughly 90% of American working hours involve tasks with little or no coverage in Mercor’s selected AI benchmarks, the company estimates. Its new interactive Benchmark Map connects 29 tests to U.S. occupations, exposing how narrowly those evaluations reach into everyday work. Mercor calls the finding provisional: its initial list is incomplete and may understate coverage.

A covered task is not a covered job

The map uses O*NET, the Department of Labor’s classification of occupations and their tasks. It spans 923 occupations, grouped into 22 job families, and 18,797 task statements. Each occupation appears as a tile shaded by the share of its tasks represented in the benchmark index. Users can explore individual job families rather than treating the economy as one score.

Mercor counts a task as covered when at least five benchmark prompts map to it. Under that rule, 208 occupations have at least one covered task; six job families have no qualifying occupations. That is a threshold for representation in a test set—not a finding that an entire occupation has been evaluated.

The mapping itself uses a language model to assign benchmark prompts to matching O*NET tasks. One prompt can match several tasks. Mercor says it checked this automated tagging against human experts’ labels on its own APEX benchmarks, a subset of the index.

Where the tests concentrate

The gaps are uneven. Mercor finds that tasks classified as exposed to AI automation are about six times more likely to appear in its index than unexposed tasks. Even after accounting for that exposure, tasks in occupations with larger total wage bills are more likely to be represented. The wage-bill measure combines employment and pay, rather than ranking jobs by individual salary alone.

Its legal-work comparison makes the distinction concrete. The selected benchmarks heavily represent AI-exposed tasks performed by lawyers, but barely reach tasks performed by similarly exposed legal secretaries or court clerks. Mercor describes smaller, middle-income knowledge-work occupations as an undercovered part of the economy.

Within occupations, the index also favors tasks that domain experts rate as more important. Mercor estimates that the highest-rated tasks are about six to ten times more likely to be represented than the lowest-rated ones, after accounting for occupation and AI exposure. The company argues that prioritizing important, exposed work is reasonable, but leaves substantial blind spots without more diverse tests.

The selection rules shape the result

This is not an inventory of every AI evaluation. Mercor’s methodology prioritizes tests of agents doing work, rather than answering questions or advising someone how to proceed. Its selection rules also favor economic realism, breadth across occupations and tests that leading models have not already mastered:

• Public access: included benchmarks have at least 25 publicly available tasks. Mercor also includes APEX tests whose held-out prompts it can access.
• Breadth: a cap on benchmarks per job family limits the dominance of software engineering and research tests.
• Realistic difficulty: tasks should resemble enterprise work while remaining challenging for leading models.

Those choices have costs. The job-family cap reduces measured coverage in heavily tested fields such as software engineering. Excluding tests that models have mastered can undercount coverage of easier tasks. Mercor also acknowledges that economic realism is subjective. To translate task coverage into working hours, it uses another study’s language-model-derived estimates of time spent on each task—not direct time tracking.

Uneven coverage also complicates turning benchmark results into a job-level score. A simple average gives extra weight to frequently tested tasks. Mercor tried weighting tasks equally across four APEX occupations: investment banking, law, accounting and software engineering. Weighted and unweighted results were generally similar, it says. That exercise addresses how represented tasks count; the map separately shows how much work still lacks representation.