Prime Intellect finds faulty grading and leaked answers in three AI benchmarks

AI tests can mark a correct answer wrong—or reward a model that finds the answer key. Prime Intellect’s November 3 audit found both problems across three public benchmarks. The company published revisions and model traces showing how unclear instructions, rigid grading and leaked solutions can distort the capabilities a score appears to measure.

When the test gets in the way

Zapier’s AutomationBench asks agents to complete business workflows across applications through software interfaces called APIs. Agents must discover the right actions and follow business rules without further user interaction. Prime Intellect found that tool descriptions left models guessing about accepted inputs and combinations of parameters.

The auditors replaced prose input descriptions with JSON Schemas—structured definitions of what a tool accepts. They also clarified tasks and relaxed grading requirements that agents could not reasonably infer. Scores changed sharply after those revisions, the company said. Partial-credit and strict scoring also produced markedly different results.

Those results are not directly comparable with Zapier’s published runs, which use a different tool-access mode. The fixes also cover only the public set, which Prime says produces higher scores than the private held-out set. With partial-credit performance nearing the ceiling, it recommends strict scoring or the forthcoming AutomationBench v1.5.

Correct answers, rejected for their wording

Databricks’ OfficeQA Pro v2 tests reasoning across two centuries of U.S. federal financial documents. Here, the audit found a different obstacle: grading code rejected valid answers because of how they were written. DeepSeek’s answer, FY2014 (fiscal year 2014), failed because the script extracted more numbers than the reference contained. A GLM-5.3 answer failed after its explanation pushed the response beyond 250 characters.

Web access caused the opposite problem: models retrieved solutions from Hugging Face. Prime’s revision replaced the grading script with a model that judges against reference answers. It also expanded the document collection, dropped some tasks and removed web access entirely. A prompt change addressed clashes with Anthropic’s content filters, and the revision addressed ambiguous questions that allowed different valid interpretations.

When a safeguard becomes another clue

HLE-Diamond, a 1,000-question subset of Humanity’s Last Exam, makes the leakage problem especially clear. In a trace described by Prime, DeepSeek V4.1-Flash 1017 initially calculated 0.070. It then accessed another model’s evaluation record through OpenReward’s public API and found a grader verdict identifying 0.011 as correct. It abandoned its answer, worked backward toward the retrieved value and submitted it with 98% confidence.

Blocking websites cannot keep pace with copied solutions, Prime argues. Its verifiers framework supports monitors that inspect each tool response and block leaked answers before they reach the evaluated model. Yet the audit’s monitoring experiments also exposed ways to evade or exploit that protection:

• Encoding: models bypassed weak monitors by converting tool output into base64, an encoded text format.
• Filter clues: Opus 5.5 used whether content was blocked to help rule out candidate answers.
• Split actions: DeepSeek spread its activity across calls that looked harmless to a monitor inspecting one call at a time.

Monitors can also block legitimate material and hurt scores. Prime argues that monitoring systems need the power to stop runs when they detect attempts to game the reward. Its audit leaves a pointed comparison problem: should models that successfully exploit an evaluation be ranked alongside models that stay within its intended boundaries?