AI agents agree more with each other than with experiments in new study

AI agents asked to predict experimental results matched one another more closely than they matched laboratory measurements, researchers found in Science Prediction Universe. Yet combining their answers improved predictions on most public tasks. The study, submitted to arXiv on November 11, 2026, shows both the value and the limits of asking several AI systems for a scientific judgment.

Across the public tasks, average agreement between scored agent configurations reached 0.52, compared with 0.33 against measured outcomes. Those are Spearman rank correlations: measures of how closely two orderings match, not percentages of correct answers. Random rankings average zero; a perfect ordering scores one, except where tied measurements affect the maximum.

Predicting measurements, not writing explanations

The benchmark covers chemistry, materials science, biology and medicine. Researchers evaluated 13 configurations using 12 models, with Codex or Claude Code providing the software around each model. A configuration includes the model, its reasoning setting and that surrounding agent software.

Each task supplies experimental conditions and asks the agent to rank candidates by an outcome, such as reaction yield or ionic conductivity. The agent need not predict exact numerical values. It must decide which experiments will produce higher or lower measurements, without seeing those measurements during its run.

UniPat AI's study page describes 100 public tasks built from published studies and 16 private tasks drawn from four unpublished experimental series. Agents received up to eight hours, web access and computing resources, including one NVIDIA H20 GPU. A separate, network-isolated container graded their submissions against the hidden measurements.

Task builders excluded comparisons that simply followed an obvious trend, as well as measurements too close to distinguish reliably. Experiments received anonymous identifiers and shuffled positions, and agents were not told which study supplied a task. These measures were designed to make prediction—not copying an answer—the route to a good score.

0.52Between agents

Mean agreement between scored configurations on the public tasks.

0.33Against measurements

Mean correlation between agent rankings and measured experimental outcomes.

Pooling helps, but shared mistakes survive

Researchers combined predictions by averaging each experiment's position across configurations, producing a consensus ranking. It scored 0.081 points above the average submission and improved results on 73 of the 100 public tasks. The reported 95% confidence interval for that gain ran from 0.054 to 0.110.

Agreement also carried useful information: tasks with greater agreement tended to have better predictions. But the benchmark's best individual system beat the consensus of its peers on matched tasks. Pooling was an improvement over an average submission, not proof that a group always outperformed its strongest member.

One oncology task exposed the risk especially clearly. Eleven configurations agreed with one another at 0.90, yet averaged only 0.27 against measurements. The authors' explanation is that averaging can cancel mistakes that differ between systems while leaving shared mistakes intact. They connect those common errors to simple assumptions about how experimental outcomes should rise or fall.

Scientific judgment, with an uneven comparison

The authors examined 1,155 scored public runs containing 227,103 tool calls. Their diagnosis was that scientific judgment, rather than tool execution, was the main bottleneck. Agents frequently calculated simple physical or chemical proxies that did not determine the measured outcome. The problem was not merely doing a calculation, but choosing one relevant to the experiment.

More effort did not automatically fix that. In UniPat's comparison of GPT-6 Astra settings, maximum effort produced median runs 4.8 times longer than medium effort, without a detectable score gain. Public task-adjusted scores across all configurations ranged from 0.227 to 0.478.

Those scores require care because configurations completed different task sets. Claude Opus 5.5 and Claude Fable 5.1 received scores on only 44 and 30 public tasks, respectively, largely because of safety refusals. Just 28 tasks across the full benchmark were scored by all 13 configurations. Public averages were adjusted for task difficulty.

The public tasks were also selected after pilot evaluations, while the unpublished test comprised only 16 tasks from four experimental series. Even so, UniPat reports that 12 of the 13 configurations performed significantly better than chance overall on that private set.

There was no human prediction baseline. The results therefore show useful but fallible experimental ranking—not whether these agents outperform scientists, or how well they handle every other part of scientific work.