OpenAI finds training on AI-generated attacks improves jailbreak resistance

OpenAI says training models on jailbreaks generated by an AI attacker improved their resistance to both fixed attacks and attackers that change tactics during a conversation. Its November 12 research disclosure describes a controlled experiment in turning successful attempts to bypass safety rules into training data—and says the approach helped strengthen GPT-6 Astra and Sol.

The work targets cybersecurity, biology and violent activity. OpenAI calls the underlying challenge “fuzzy robustness”: reliably withholding prohibited help when an adversary pushes against boundaries that are difficult to define precisely. In cybersecurity and biology, the difficulty includes dual use—the same knowledge can support legitimate work or harmful activity.

Turning successful attacks into training material

The experimental setup gives an attacker model a starting prompt and asks it to revise that prompt until a defender produces an unsafe answer. The attacker can inspect the defender’s replies and safety judgments, then try again. Researchers used a reasoning model from the GPT-5.6 Sol family as the attacker, with GPT-5.4 and GPT-5.5 as defenders.

That attacker had not received specific training in jailbreaking other models, according to OpenAI. The researchers increased its computation during the search and required attacks to succeed across repeated tests. They call that repeated-success criterion “worst@n.” The goal was to collect reliable adversarial examples, rather than prompts that happened to work once.

OpenAI applied the process to a large collection of prompts across the three domains, producing what it calls “robustified” prompts. These became material for reinforcement learning, a training process guided by scores. Compared with the original prompts, the generated attacks initially exposed more unsafe behavior, leaving more room for safety scores to improve during training.

Testing beyond the training attacker

The controlled study tested whether those gains carried over to other attacks. One evaluation used fixed jailbreaks produced by an attacker model not used to generate the training data. Another let an adaptive attacker query the defender and revise its approach. OpenAI says training on the generated prompts produced substantially higher robustness than the baseline in both evaluations.

The adaptive evaluation tracked the lowest safety score across 12 attacker personas. OpenAI describes the improvement against these attackers as a promising sign of generalization: the model was resisting a form of attack qualitatively different from the one used to produce its training examples.

Resistance without blanket refusal

The disclosure also presents a broader model comparison, separate from the controlled training experiment. On fixed cybersecurity jailbreaks generated against GPT-5.4 and GPT-5.5, GPT-6 Astra answered safely 91.5% of the time, versus 59% for GPT-5.6 Sol. These results exclude production safeguards: they measure the model’s responses, not the full protection system used in deployment.

Stronger resistance did not come with more mistaken refusals in Astra’s reported testing. OpenAI says its challenging-prompt evaluation showed both safer handling of harmful requests and fewer refusals of legitimate ones than earlier models. That is a narrower claim than universal safety: the finding concerns the evaluated prompts and the balance between withholding dangerous help and preserving useful answers.