Goodfire researcher challenges the idea that AI’s inner workings are tangled

Goodfire researcher Nick Cammarata is challenging an idea he helped establish: that AI models pack unrelated concepts into tangled internal structures. In his new essay, “Neural Networks Are Already Clean,” he argues that better tools and agent-assisted experiments reveal understandable patterns instead. Google DeepMind’s Neel Nanda sees a useful approach, but not evidence that the problem is nearly solved.

A cat, a car—or one shared shape?

Mechanistic interpretability means tracing how a model reaches an answer through its internal computations, rather than judging only its output. A longstanding obstacle is polysemanticity: an artificial neuron appears to respond to several unrelated things, making its role difficult to explain.

The explanation called superposition holds that networks can fit unrelated concepts into shared neurons when those concepts rarely interfere. That encouraged researchers to train a second model, such as a sparse autoencoder, to separate the concepts and study those reconstructed features instead of the original neurons.

Cammarata now disputes the reading of famous examples behind that approach. One neuron in the image model InceptionV1 responded to both cat faces and car fronts. He now describes it as a V-shape detector: pointed forms activate it, while rounded forms drive its response negative.

He also revisits a circuit from “Zoom In,” work he co-authored, in which a car feature appeared among dog neurons. His revised explanation is ordinary generalization: the circuit groups cars and dogs according to which way they face the camera, not because it hides unrelated concepts together.

Better tools, broader experiments

His name for this branching process is “recursive generalization.” One example begins with a neuron detecting the letters “iy,” extends to “diy,” and then reaches “do it yourself”—a phrase without the original letter sequence. The responses look different, but his explanation connects them through successive related meanings.

Cammarata attributes earlier difficulties partly to weak tools and architectural problems in older models. He reports that architectural changes made 99–100% of the last-layer neurons in an InceptionV1-based model easily interpretable, without changing its size or training data. Versions spanning a 300-fold size range remained broadly legible, he says.

The language-model figures are preliminary. In Qwen3-32B, he estimates that roughly 90% of neuron poles have one main job, about 7% have two related jobs, and 2–4% remain unexplained. Those shares depend on how weak responses are counted and what qualifies as one job; he cautions against treating them as a precise cleanliness score.

Automated agents analyzing neurons in bulk did most of the work, Cammarata says. His proposed route is to investigate neurons and their connections directly, using experiments to challenge explanations. In the essay, neighboring neurons, downstream effects and changes caused by turning a neuron up provide additional checks on an interpretation.

Readable neurons do not settle the argument

Nanda, who leads Google DeepMind’s interpretability team, accepts a practical case for neuron-based analysis. It could help explain questionable model behavior and is easier to apply to frontier models. His team had already deprioritized sparse autoencoders because it did not see enough value for the effort involved.

But Nanda argues that understandable neuron responses do not show that interpretability is nearly solved. He expects the approach to miss substantial information. Language models store more facts than they have neurons, he argues, and weak activations likely carry much of that information. He also questions whether small-image-model findings transfer to language models.

Part of the disagreement concerns definitions. Cammarata does not reject the possibility that models represent more concepts than they have dimensions. His narrower target is arbitrary packing of unrelated concepts. Nanda thinks the essay’s standard for a neuron having one meaning is too loose, while crediting the reexamination of older evidence.

The technical companion paper had not been released as of The Decoder’s October 16 coverage. Cammarata says it will detail similar findings in models up to trillions of parameters. For safety, his ambition goes beyond labeling neurons: he argues that reading a network’s own mechanisms could expose deception and allow targeted corrections.