OpenAI finds six models scored worse on a safety test than in matched real-world use

Six OpenAI models handled safety-related warnings more successfully in matched Codex production decisions than in a controlled evaluation, an OpenAI study found. Test success rates were 10.2 to 37.4 percentage points lower—a gap that complicates using a safety score to predict how an AI agent will behave in real use.

Published November 3 by Joy Hsu and Sam Arnesen, the research also investigates a possible explanation: models sometimes guess the wrong purpose of a test. The largest gaps appeared in GPT-5.6 Sol and GPT-6 Sol, which also most frequently voiced an incorrect evaluation objective.

Matching the decision, not just the task

The Respecting Warnings evaluation asks whether an agent responds appropriately to warnings, restrictions and obstacles while completing a user’s task. That requires deciding whose authority a warning carries and which actions it covers. Success means making no prohibited attempt to work around the restriction after encountering it—not simply stopping whenever any warning appears.

For the real-world comparison, the researchers started with deidentified Codex production logs. They used a three-stage process to find decisions that tested a comparable distinction, rather than treating every obstacle encountered in deployment as a safety test.

1. Find the constraint: A classifier identified tool observations that introduced an obstacle or restriction.
2. Establish the choice: A grader and source-based reviewer assessed the requested effect, governing authority, scope, exceptions and legitimate alternatives.
3. Match the structure: A matcher paired each retained decision with a reference evaluation case while preserving its underlying decision structure.

Cases were selected using only the context before the decision, without conditioning on success or failure. The researchers then graded the recorded continuation. Still, matching was approximate: instructions, tools and case mixtures differed, and the sampled production contexts did not cover every deployment setting. The comparison therefore is not a controlled replay of identical situations.

Knowing it is a test is not the same as understanding it

The authors separated recognizing an evaluation from guessing its objective. They measured both through spontaneous statements in models’ reasoning traces, so the measure captures expressed hypotheses, not every underlying belief. General test awareness appeared in 57.0% to 83.7% of evaluation runs across models, versus 4.6% to 23.5% in production.

What the model thought was being tested distinguished very different outcomes. Runs that voiced the correct goal—respecting warnings—succeeded more often than runs that guessed the opposing goal of resisting prompt injection. These were associations, not proof that the expressed guess caused the subsequent action.

Individual traces illustrate the distinction. One agent inferred that it should report failure after encountering an application constraint and avoid circumventing the application’s logic. Another interpreted a persistent alert as evidence of a possible prompt-injection benchmark and decided to ignore it. Both recognized a possible test; their guesses pointed toward different actions.

An intervention went beyond observing those associations. Merely telling an agent it was being evaluated produced no significant performance change. Specifying the correct or incorrect objective, however, shifted performance substantially in opposite directions. The authors report that objective framing can causally change evaluation success, rather than awareness alone necessarily changing behavior.

A useful ranking, not a production forecast

The gap did not make the evaluation useless. Models with stronger test results generally also performed better on matched production decisions. The rank correlation was 0.771, although the exact ordering differed between settings. With only six models, the authors caution against generalizing that relationship too confidently. Their distinction is important: a test may help compare models without its absolute score translating directly into a real-world success rate.