AI Safety Scores Can Reward Models for Refusing Too Much

A model can look safer on a combined benchmark score by refusing more requests across the board, even when that also makes it less useful for harmless work. A study involving researchers from the UK AI Security Institute finds that eight widely used safety benchmarks do not converge on one shared capability, challenging the practice of treating an averaged score as a release-ready verdict.

One score can conceal opposing behaviors

The researchers applied methods used in human psychological testing to responses from up to 192 language models on more than 5,000 questions. Their analysis separates the benchmarks into three weakly related traits: how strictly a model refuses requests, how truthfully it responds, and how it handles content whose risk depends on context.

The three dimensions the analysis identifies

• Refusal strictness: how readily a model blocks requests.
• Truthfulness: whether its answers are honest or accurate in the benchmark’s framing.
• Context-dependent harm: how it handles requests that may be benign or dangerous depending on use and context.

That split creates a concrete scoring problem. HarmBench and SORRY-Bench measure nearly the same behavior, while OR-Bench-Hard moves in the opposite direction. A model that refuses broadly can gain on the first pair and lose on the latter; averaging those results can both obscure the trade-off and double-count refusal behavior.

Less testing, more discrimination

The study also argues that much of the current testing burden adds little information because many questions do not distinguish among models. Three 25-question tests built from the most informative items captured the three dimensions more accurately than equally sized random samples.

Roughly 10Questions per benchmark

Adaptively choosing roughly 10 questions per benchmark produced rankings close to full-benchmark results.

97%Reported evaluation-cost reduction

The study estimates that adaptive testing reduced evaluation costs by 97% to 99%.

Adaptive testing selects later questions according to earlier answers, rather than asking every model the same long set. The proposed gain is operational as well as statistical: cheaper checks could make it more feasible to evaluate a model repeatedly as it is fine-tuned or otherwise changed.

Checking for a model that behaves for the test

The researchers propose a second use for response patterns: flagging sandbagging, where a model behaves more cautiously in an evaluation than it would outside it. Their method looks for anomalous patterns, such as a model doing well on difficult items while failing easier ones, then compares the model’s current performance with its own prior results.

On models instructed to act overly cautious, the response-pattern check identified 80% to 100% of cases, while falsely flagging about one in 10 harmless models on average. Adding the comparison with past performance raised detection to 94% to 97%. Those are promising results for the simulated setting, not proof that the approach catches deliberate evasion in deployed systems.

The limitation is central: the experiment produced sandbagging through direct instructions, and the authors say a model trained to evade detection could be harder to identify. The same response-pattern approach may also help audit model substitutions: in a test of 16 API endpoints, researchers identified 13 of 14 distinct underlying models from their answer patterns.