Can Researchers Finally Understand the “Tangled Knot” Inside AI Models?

Researchers concluded that a model multitasks at least some neurons, presumably allowing it to fit more data. Studies of other models appeared to confirm the pattern. This phenomenon became known as superposition: a model collapses seemingly unrelated concepts into a single neuron.

The results were clear. A peek inside the model revealed a giant, tangled knot.

Why Mechanistic Interpretability Has Been So Difficult

As a result, most mechanistic interpretability studies have focused on how to untie that knot. Despite some breakthroughs, progress has been slow. Modern models, including large language models (LLMs), are so large and complex that researchers can understand only small parts of them at a time.

Some experts question whether it is possible to understand an LLM in detail at all.

Cammarata was one of them. “I was very pessimistic,” he says. But then he had a hard time actually seeing.

Looking at Neural Networks as More Than Neurons

At Goodfire, Cammarata spends almost all of his time looking inside neural networks. He built new tools and deployed a team of AI agents to help him understand what he was seeing.

“I spent, I don’t know, most of the last year, 10 hours a day, seven days a week, just looking at neurons,” he says.

What Cammarata claims to have realized is that neural networks have a kind of logic, even if they do not make sense at first glance.

“We were looking at it wrong,” he says. “I want to try a little harder. Don’t give up too soon.”

Cammarata almost sounds like a convert. Early experiments with models such as Inception v1 suggested that neural networks were deeply entangled, encouraging a type of confirmation bias.

When researchers see the new model, he says, they will be puzzled. “Every time I didn’t understand something, I thought, ‘Oh, this process of disentangling is more important than I thought, because without it you can’t make progress.’”

But the more Cammarata examined the neurons, he says, the more their behavior began to fall into place.

“Not everything is perfect, but I expected nothing to be legible or very little to be legible,” he says. “And we were actually able to plan out the entire circuit.”

What Cars and Cats Reveal About AI Circuits

Cammarata returned to Inception v1 with a new approach and new agents. “I studied it probably thousands of hours,” he says.

With fresh eyes, Cammarata discovered what he now thought was wrong with his earlier analysis. Consider neurons associated with cars and cats. Rather than asking why multiple unrelated concepts were packed into a single neuron, he sought to understand what those concepts had in common.