Larger attack collections produced weaker AI defenses in three of four tests, study finds
The researchers’ proposed defense rewards different attack techniques, not just successful attempts. Its promising results come from limited tests, and its measure of diversity still needs human validation.
Researchers’ PIPETTE method targets a failure in prompt-injection defense training: successful attack examples can cluster around one tactic, so adding more may not broaden protection. Across four test setups, the larger collection yielded weaker defenses in three; in a Qwen3-4B comparison, training on PISmith’s larger set left the defender vulnerable on up to 19 of 40 selected weak tasks, versus 11 for RL-Hammer’s smaller set. PIPETTE rewards newly discovered techniques and checks whether the agent’s next action serves the user’s request; results are promising but rest on limited stress tests and unvalidated automated technique labels.
01
PISmith successfully attacked 68.75% of 560 practice tasks, compared with 33.39% for RL-Hammer.
02
On AgentDojo, off-the-shelf injections succeeded in fewer than 0.5% of attempts on average against PIPETTE-trained small models; adaptive attacks reached 10% on selected weak tasks.
03
Adaptive attackers used a 4-billion-parameter model and made around 250 attempts; the authors say more attempts could uncover additional failures.
Training an AI agent on more successful attacks does not necessarily make it harder to hijack. In a study submitted to arXiv on November 18, researchers found that the larger of two attack collections produced weaker defenses in three of four test setups. Their proposed fix, PIPETTE, rewards variety in attack techniques rather than success alone.
The work addresses indirect prompt injection: instructions hidden in outside material an agent reads, such as an email, web page or tool response. An attacker tries to make the agent treat that material as an order and carry out an unwanted task. Developers can use successful attacks as training examples to teach the agent to resist.
But an automated attacker can discover one effective trick and keep reusing it. The paper identifies this pattern in two attack-generation methods, RL-Hammer and PISmith. Researchers call it mode collapse: the system repeatedly finds similar strategies instead of exploring different ways to deceive the agent. Additional successful examples then fail to deliver additional defensive gains.
Both tools have checks intended to encourage variety, but those checks measure wording differences, Help Net Security explains. Its account notes that RL-Hammer’s authors had seen their check satisfied by changes in capitalization or irrelevant added text. A collection can therefore look diverse at the sentence level while relying on the same underlying maneuver.
The comparison used Qwen3-4B across 560 practice tasks. PISmith successfully attacked 68.75% of those tasks, compared with RL-Hammer’s 33.39%, and generated the larger collection of working injections. The researchers then retrained the model separately on each collection and tested the resulting defenders against three attacks they had not seen.
Editorial illustration for Larger attack collections produced weaker AI defenses in three of four tests, study finds.Source: helpnetsecurity.com.
Those 40 tasks were chosen because simple attacks had already defeated the unprotected model. The percentages describe a small stress test of known weaknesses, not an estimate of everyday failure rates. The authors also caution that narrow technique coverage is not necessarily the only explanation for the gap between defenders.
PIPETTE connects attack discovery with defensive training in two stages. The goal is not simply to block malicious instructions, but to keep the agent working on the legitimate request while it encounters them.
Broaden the attack collection. PIPETTE groups successful injections by technique and gives the attacker an extra reward for finding a working technique it has not found before. The paper describes a dictionary-learning-based reward for this purpose.
Train the agent to check its next action. Before acting, the defender learns to assess whether that action serves the user’s request. This targets a weakness of defenses that stop suspicious activity by abandoning the legitimate task as well.
The authors report promising results. On AgentDojo, a test environment for agent tasks, off-the-shelf injections succeeded against PIPETTE-trained agents on fewer than 0.5% of attempts on average on both small models. Adaptive attackers, which kept revising their injections for around 250 attempts, succeeded on up to 10% of the 40 selected weak tasks across three models.
The evaluation covered four open-weight models ranging from 4 billion to 27 billion parameters. Adaptive attackers used a 4-billion-parameter model, and the authors acknowledge that more attempts could uncover more failures. They also have not validated the automated grader’s judgments of technique coverage against human judgments.
Keeping useful work intact was a measurable distinction. Under one attack on Qwen3-4B, PromptGuard blocked slightly more attacks than PIPETTE, but its protected agent completed only 16.19% of user tasks. PIPETTE’s agent completed 58.10%. In that comparison, the defense that blocked more attacks was not the one that most often finished the job.
Sources
arxiv.orgPIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
helpnetsecurity.comYour AI red team may be finding the same prompt injection thousands of times - Help Net Security
Reader comments
Newest comments first. Replies stay oldest first.