Artificial Analysis shifts 60% of its AI index to private tests or held-out answers

Most of Artificial Analysis’ revised AI capability index now rests on private test sets or held-out answers. Released on October 27, 2026, Intelligence Index v5 puts 60% of its score behind that boundary, while adding coding and scientific research tasks that require models to carry out work rather than simply answer questions.

In Artificial Analysis’ results, Claude Opus 5.5 leads with 58 points, just ahead of GPT-6 Astra at 57 and GPT-6.1 Sol at 55. All three scores use maximum effort settings. But the revised index has no model that wins every evaluation.

Private tasks, stricter finish lines

The new coding component, Cognition’s FrontierCode 2.0, contains 132 tasks in real open-source repositories. It grades whether a repository maintainer would merge an agent’s change. Failing any blocking check earns zero. The tasks remain private to avoid contamination—exposure of test material before evaluation.

Terminal-Bench-Science adds 70 scientific research tasks performed through a terminal, a text-based computer interface. Experts curate and review the tasks, and every test must pass for a task to count. That makes the finish line a completed workflow, not partial success on its checks.

Three existing evaluations also change

• HLE-Diamond replaces Humanity’s Last Exam with a refined 1,000-question version, evenly divided between reasoning and knowledge.
• AA-Omniscience v1.1 removes public questions and questions where performance had saturated, adds new ones, and revises ambiguous or erroneous material.
• CritPt v2 contributes a private, 40-question hard subset of research-level physics problems. Models tackle it with a code-execution tool in an offline Python environment.

The composite gives equal weight to four areas: knowledge work, coding, science and general capability. Each contributes 25%, though individual evaluations within those areas carry different weights.

The overall winner does not sweep

Opus leads three of the nine evaluations, including FrontierCode 2.0 at 46%. Astra leads Terminal-Bench-Science at 63%, compared with Opus at 59%. Astra also tops the hard physics subset at 72% using high effort; Opus scores 61% at maximum effort.

The open-weights comparison needs a distinction. Artificial Analysis lists Kimi K3.1 Pro as the strongest open-weights model at 46 points, but its weights have not yet been released. The publisher’s stated gap between the overall leader and that category grows from three points in v4.3 to 12 in v5.

Among models whose weights are actually released, DeepSeek V4.1 Flash 1017 and GLM-5.3 tie at 32, both at maximum effort. They sit 26 points below the overall leader and below Claude Haiku 5.5, which scores 36 at maximum effort.

More effort changes the bill

Even within one model, the score comes with a spending choice. Opus rises from 35 points at low effort to 58 at maximum effort. Its reported cost per Intelligence Index task rises roughly tenfold, from $0.98 to $10.04.

The revised tasks also demand more generated text. Every measured model produces more output tokens—units of generated text—per task than in v4.3. The median rises from 35,000 to 59,000. Four agentic evaluations account for 55% of the index’s weight but a median of about 90% of output tokens.

Alongside v5, Artificial Analysis introduced Progress Index v1, placing language models released since November 2022 on one continuous capability scale. The publisher assigns the two indexes different jobs: use the Intelligence Index to compare models for current work, and the Progress Index to compare capability across generations, including models released years apart.