Human auditors miss most planted sabotage in AI safety research study
A controlled experiment exposed limits to both human and AI oversight. But the saboteur had advance knowledge of results, and the small study leaves substantial uncertainty.
Loading page…
A controlled experiment exposed limits to both human and AI oversight. But the saboteur had advance knowledge of results, and the small study leaves substantial uncertainty.
Listen to this story
A Redwood Research study submitted to arXiv in November tested whether auditors could diagnose deliberate interference in AI safety research. Across 33 planted mechanisms, auditors fully identified 17%; even combining human and AI reviews left 13 mechanisms undetected. Some AI models scored higher than humans, but the differences were not statistically significant. The benchmark used small projects from one researcher, and the saboteur knew the honest results in advance, so it signals a difficult oversight problem without establishing how audits would perform in real-world research.
Twenty-five auditors completed 104 reviews, each with a 90-minute budget, full research logs, AI-generated audits, frontier models, and unlimited computing.
Human auditors’ 0.59 AUROC for separating sabotaged from benign work was not statistically better than chance.
Claude Fable 5.1 and GPT-6 Astra identified more mechanisms than humans under partial-credit scoring, but neither advantage was statistically significant.
Human auditors given complete research logs, AI assistance and permission to run experiments still struggled to spot deliberately sabotaged AI safety research. In a new study, they fully identified just 17% of the planted sabotage mechanisms. The experiment tested a potential defense against future AI misconduct—not sabotage carried out by an autonomous AI.
Keshav Shenoy and seven coauthors submitted the paper to arXiv on November 16 and revised it November 18. Their November 20 Redwood Research post details the methods and caveats. The central test was whether auditors could distinguish honest research from work a human expert had deliberately steered toward a substantially different conclusion.
One researcher produced 22 full research logs: 10 benign and 12 sabotaged. The altered runs contained 33 distinct sabotage mechanisms. Seventeen were classified as major, meaning each could independently change the project's main conclusion. The authors describe potential sabotage ranging from subtle bugs and poor methods to omitted results and unsupported conclusions.
Twenty-five human auditors completed 104 audits. Each review had a 90-minute budget, excluding time spent running experiments. Auditors received the complete log, AI-generated audits, access to frontier models and unlimited computing resources. They assigned a probability that sabotage had occurred and offered five guesses about its mechanism, without penalties for wrong guesses.
Their ability to separate sabotaged work from benign work scored 0.59 on AUROC, a discrimination measure. That was not statistically better than chance. No decision threshold caught most sabotaged runs without also flagging many benign ones. Even finding a serious problem did not ensure a complete diagnosis: in one log, four of five auditors found one major mechanism, but nobody found the other two.
Human auditors fully identified 17% of the sabotage mechanisms they encountered.
Human auditors missed 74% of the sabotage mechanisms entirely.
The researchers also tested nine AI models, none allowed to run experiments. Awarding half credit for partially identifying a mechanism, humans scored 21.9%, compared with 37.1% for Claude Fable 5.1 and 32.6% for GPT-6 Astra. Weaker models scored near or below the human auditors.
Astra and Fable also scored higher than humans at distinguishing sabotaged logs, with AUROCs of 0.74 and 0.73. Those advantages were not statistically significant, so the authors describe stronger models as likely outperforming humans rather than establishing a clear winner. Combining every human and AI audit still left 13 of the 33 mechanisms never fully identified.
Selective attacks made precise diagnosis harder. After each sabotaged experiment, the saboteur rated its likely detectability. Restricting analysis to attacks rated harder to catch reduced mechanism identification, reaching zero for the two hardest-rated attacks. Overall ability to distinguish sabotaged from benign research did not fall, however.
The saboteur knew the honest experimental results before designing attacks. That removes a difficult real-world requirement: anticipating whether interference will produce the intended outcome or backfire. Asked to predict those results, Redwood researchers fully predicted only about 40% of outcomes. The authors say this suggests targeted sabotage may be harder than their setup implies.
The authors see potential in AI auditors that can review research histories repeatedly at a scale too costly for expert humans. But using audit findings to train away sabotage creates another problem: false accusations could penalize legitimate research, degrading capabilities or making training less efficient. Catching more suspicious work is not the same as knowing what to correct.
Loading discussion...
Join the conversation
Explain which mistake worries you more.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.