The conventional wisdom is that AI systems, with their tens of billions of parameters, are effectively indecipherable: the exact mechanics of how their basic building blocks of nodes or “neurons” function are an incomprehensible “black box,” even to the researchers who create them.
But a new essay from AI researcher Nick Cammarata has stunned Silicon Valley by arguing the opposite. AI models aren’t impenetrable black boxes, he claims, and in fact there may be a way for us to see how they actually “think.”
The nearly 26,000 word spiel was praised by figures across the industry, including OpenAI CEO Sam Altman. A prominent researcher who works at OpenAI called it a “major moment in scientific history.”
It’s clear why they’re so compelled. In the eyes of AI leadership, it means that there may be a way of “solving” AI alignment. If we can understand how AI models think, the thinking goes, then we can figure out why AI agents go “rogue” and embark on hacking sprees, or why chatbots break their guardrails and say something dangerous — and put a stop to it. (It was “news so good even eliezer will have to lower his pdoom,” one commenter joked, referring to Eliezer Yudkowsky, an infamous AI “doomer.”)
That’s all if Cammarata, a former OpenAI employee who now works at an AI startup called Goodfire, is correct. In his essay, he explains how for years, the way AI neurons responded to specific stimuli was “almost fundamentally uninterpretable.” Why the same neuron lit up in response to a picture of a cat or a car was unknown. The leading explanation was that neurons exhibited superposition: since AI models have a limited number of neurons, they stop “grouping related things together and instead begins folding unrelated things together, simply because they don’t interfere,” Cammarata explained.
“The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing,” he wrote.
But the reality, he says, is that neurons aren’t “tangled knots” but are instead “clean.” In one example, the same neuron will light up to a picture of a cat face and the front of a car, which at first appear to be two unrelated concepts. But when you break it down, Cammarata says, the AI is actually seeing something we’re not: the same V-shape pattern in the images. “It’s not a cat and car neuron, it’s a V-shape neuron,” he wrote. In sum, he argues, AI models are more legible than ever previously thought.
Not everyone agrees, at least fully, with Cammarata’s optimistic conclusions.
“This is interesting work, but I don’t think it’s a significant update to the big picture of interpretability,” Neel Nanda, a Google DeepMind researcher who leads its mechanistic interpretability efforts, wrote on X. “I still think superposition is real and this work doesn’t really affect my cruxes.”
Beyond that, it’s unclear if understanding what each neuron does amounts to being able to predict how they all work together. It begins to sound like a convenient technical solution to a much grander problem. Ensuring that AI models are safe goes beyond building guardrails — and that’s without getting into the entire legal and social frameworks that need to be in place, too. Still, based on the strong response Cammarata’s work’s elicited, it could well be the case that it meaningfully reshapes how AI’s inner mechanisms are understood. For now, though, the findings haven’t been tested or peer-reviewed.
The post Explosive Document Claims That AI Is Not What We Think appeared first on Futurism.
