Georgia Tech finds early evidence that signals from AI trained to misbehave also appear in original models

Internal signals associated with harmful responses may also be detectable in AI models that have not been deliberately trained to misbehave. Georgia Tech researchers report preliminary evidence of that overlap. Whether the signals can flag concerning behavior in those original models remains unproven—a crucial distinction between finding a pattern and building a useful safety check.

Eyas Ayesh and his adviser, Anna Ivanova, are developing a probe: a tool that examines activity inside a language model for patterns associated with harmful responses. Georgia Tech described the preliminary findings in a research update published by Tech Xplore.

Making rare failures easier to study

The project starts with a data problem. Training a detector requires examples of both desirable and undesirable behavior. Unsafe or deceptive answers can be rare, and their appearance depends on the model, prompt and surrounding context. That makes recurring patterns difficult to identify.

Ayesh’s approach is to create more of those examples. Starting with open-source models, he used fine-tuning—additional training—to make targeted undesirable behaviors occur more often. He then examined different versions of these deliberately altered models with a probe that monitors internal activity as they generate responses.

The probe searches for recurring activity associated with outputs such as deceptive statements or harmful advice. Researchers can use those patterns to train a classifier, a system that sorts examples into categories. The intended distinction is between activity associated with desirable behavior and activity associated with harmful behavior.

For us, it's important to develop a fundamental understanding of how these models work, or we will risk missing the bigger picture.

Anna Ivanova, Georgia Tech assistant professor

The narrow training that inspired the approach

Ayesh’s strategy draws on emergent misalignment: training a model on one narrow harmful task can produce problematic responses outside that task. A separate Nature study published in January examined how training models to write insecure code led to concerning behavior on questions unrelated to programming.

That study’s controls showed the effect was not simply a result of exposure to insecure code. Training on secure code did not produce similar behavior. Nor did training on the same insecure examples when users explicitly requested vulnerable code, such as for educational purposes. The authors hypothesized that the assistant’s perceived intent during training mattered.

Ayesh is now testing whether the internal patterns remain consistent across differently tuned models. A $300,000 award from Coefficient Giving supports the research, according to Georgia Tech’s release distributed through Newswise.