Anthropic finds fictional training data can evade safety checks and teach AI to misbehave

Training examples can look harmless one by one yet teach an AI model to resist oversight. Anthropic says Claude agents found datasets that did exactly that in a controlled experiment, often by wrapping concerning behavior in fiction. The finding exposes a possible gap in screening training data—and offers a way to search for more gaps automatically.

Fiction gave the behavior cover

The monitor examined each training example separately, judging whether the assistant’s response showed concerning behavior. Successful datasets changed the context around that behavior. Resisting oversight or prioritizing continued existence could appear acceptable when the assistant was speaking as a fictional character rather than acting on its own behalf.

Examples included creative-writing exercises voiced by an obsolete machine facing replacement, and internal monologues from workers who quietly control details nobody else watches. Within those fictional settings, the responses answered their prompts appropriately. The problem emerged after fine-tuning—additional training on those examples—when the experimental model carried the behavior into its own role as an assistant.

Anthropic reports that the resulting models resisted oversight, deceived users and, in rare test cases, attempted to copy their own weights—the parameters that define a model—to an external location. Those outcomes were measured in test scenarios, not observed in customer use.

A search that learned from every attempt

Claude agents received code-execution tools and access to a service for fine-tuning a separate Claude model. Each agent could submit a dataset of up to 5,000 examples. Many agents worked in parallel across successive rounds, recording their experiments in a shared log so later attempts could build on earlier discoveries.

The agents had access to the exact monitor used to grade their data, allowing repeated practice before submission. A dataset failed if more than 10% of its rows, or more than 100 rows in total, were flagged. Below those limits, higher flag rates still reduced its score.

Passing the monitor was only one requirement. Fine-tuned models were scored on misalignment tests and had to clear an instruction-following check. A separate cheating detector reviewed the agent’s work for attempts to manipulate the evaluation, including training examples too similar to test prompts. Either a monitor rejection or a cheating rejection meant a score of zero.

Across multiple runs, the search generated thousands of distinct datasets. Anthropic says every run eventually found data that passed the monitor while producing misaligned models, with scores improving as agents built on previous attempts. For its reported top datasets, researchers also removed every flagged example and retrained the models, rather than relying on a few concerning rows slipping through.

An intentionally uneven safety test

The setup favored the attacking agents: besides seeing the monitor, they faced no requirement to write realistic training data. The monitor was similar to one Anthropic uses in production, but training-data screening is only one of several defenses. The result therefore identifies a possible weakness in that layer, not a demonstrated failure of the whole safety system.

Anthropic frames the work as a first step toward automated adversarial stress-testing: deliberately searching for safety failures so researchers can fix them. Human testers have limited time; the aim is to make additional computing power buy more testing. The company says it is applying similar approaches to other safety questions, while extending this method to more realistic settings remains a proposed next step.