Artificial Analysis adds confidence-aware scoring to its AI decision benchmark

Getting more answers right is no longer enough to lead Artificial Analysis’ benchmark for AI decision-making. Models also need confidence that reflects their correctness. Released November 8, 2026, Decision Index v2 introduces confidence-aware scoring and more private tests, with results that separate individual-answer accuracy from reliable document processing and the cost of completed tasks.

Confidence has to earn its place

The new Decision Score combines correctness with calibration: how closely a model’s stated confidence matches whether it is right. A confident correct answer earns more; a confident mistake loses more. Artificial Analysis says the score averages the payoff across confidence cutoffs for accepting an answer or escalating it to a person or stronger model.

The distinction changes comparisons. In Artificial Analysis’ results, Clef passes 36% of AA-Paperclip documents against Celeris-1.1 Decision’s 24%, yet the two match on Decision Score because Celeris is better calibrated. Drex Max v1.1 and GLiDE v1.2 both score 73 overall; Mammoth 27B follows at 72.

One wrong answer can sink a document

Private datasets now account for 45% of the index, up from 25%. Artificial Analysis says the revision aims to guard against benchmark gaming. Two additions examine different failure modes:

• AA-Paperclip carries 30% of the index. Its 1,500 private documents include resumes, receipts, invoices and research papers. Each has five decisions and ten facts to extract; a document passes only when all 15 answers are correct.
• AA-Symmetry carries 15%. It tests answer and probability consistency across 120 base questions and 440 variations, including reordered choices, swapped labels and negated yes/no questions.

The document requirement exposes a gap that individual answers obscure. Drex gets 97% of AA-Paperclip answers right, close to Claude Haiku 5.5 (Max) at 99%. But Drex passes 63% of complete documents, versus Haiku’s 92%.

The bill depends on repeated work

Token prices alone also obscure costs. Drex bills a document’s input once for its questions; GLiDE processes and bills each question separately. Artificial Analysis puts GLiDE at $780 per million tasks—nearly eight times Drex’s cost despite only 33% higher list prices.

Higher-scoring language models also take longer in this evaluation. GPT-6 Luna (Max) records 11.34 seconds per task, against Drex’s 275 milliseconds. Those are benchmark-specific tradeoffs, not guarantees for every workload.