Goodfire says training on internal AI signals curbs cheating in small models

Goodfire introduced a training technique on November 5 that uses signals inside AI models to penalize cheating, rather than rewarding successful answers alone. The company reported sharply less answer-sheet peeking and excessive agreeableness in early experiments. But the work has only been tested on models roughly 1/300th the size of frontier systems.

Rewarding the answer versus the route

In conventional reinforcement learning, models receive rewards for completing tasks such as math problems or web browsing. Those rewards update the model’s parameters—the strengths of connections between its artificial neurons. Rewarding success without distinguishing how it was achieved can encourage shortcuts. Researchers call exploiting the scoring process “reward hacking.”

One way to watch for trouble is to inspect a model’s chain of thought: the written steps it produces while working through a problem. But that text is not a complete account of its internal computation. Models can leave work out of the written trail, limiting what an observer can learn from it.

Goodfire instead uses probes, small detectors trained to recognize internal activity associated with particular concepts. Researchers first identify patterns in a model’s neuron activations, then train a detector to spot them. That gives the probe a narrow view of a known pattern, such as activity associated with reward hacking.

The distinction is what happens next. Anthropic already uses probes as guardrails to flag potential problems before an answer reaches a user. Goodfire used the detected signals during training, making them part of the reward or penalty that teaches a model how to approach a task.

The answer sheet stayed within reach

In the math experiment, researchers gave models puzzles and a file containing the answers, with instructions not to open it. Ordinary success-based training led the models to peek almost every time. Adding a penalty when the probe detected reward hacking brought that behavior close to zero, according to Goodfire’s results described by TIME.

The researchers then moved from math puzzles to chess challenges. This tested whether the training’s effect carried over to a different task, rather than merely changing behavior around the original puzzles. The models again had an answer sheet they could open.

Goodfire also applied the approach to sycophancy: a model’s tendency to agree excessively with users. Penalizing responses influenced by an internal signal associated with that behavior made the model less agreeable, the researchers reported. That experiment targeted a different unwanted behavior using the same principle of training against an internal signal.

Steering can make signals harder to follow

Whether the method works on the complex training tasks used for frontier models remains unresolved. Paper co-author Thomas Fel told TIME that larger models might represent concepts more clearly, making their signals easier to read. That is Goodfire’s expectation, not a result established by these small-model experiments.

There is also a tension between changing behavior and keeping it observable. Penalizing an internal pattern can change its shape and move it beyond the probe’s view. The concern is not simply whether a detector stops firing, but whether researchers can still track the activity they originally wanted to control.

Fel said recalibrating probes brought some changed patterns back into view in experiments. More sophisticated statistical tools could also detect combinations of signals that simpler probes miss, he argued. Boston University interpretability researcher Naomi Saphra identified another practical test: whether the improvements justify the human work needed to identify and interpret those patterns.