One of the oldest puzzles in artificial intelligence research involves cars and cats. For more than a decade, researchers have wondered why an early Google technology seemed to treat cars like cats and vice versa.
This Google technology, Inception, could reliably identify common objects in digital photographs. But when researchers looked at the myriad mathematical calculations that drove the system, they couldn’t make sense of it. The math that identified a car looked a lot like the calculations that recognized a cat.
This month, one A.I. researcher claimed to have solved this riddle. After taking another hard look at the vast array of numbers inside Google’s A.I. technology, the researcher, Nick Cammarata, noticed that one of these calculations pinpointed a “V” shape that appeared both at the front of a car and on the face of a cat.
Mr. Cammarata’s proposed solution reopened a longstanding debate inside the artificial intelligence research world, and not just because of what he had to say about automobiles and felines. The revelation was just a small part of a 26,000-word treatise in which Mr. Cammarata, a former OpenAI researcher, argued that the problem of A.I “interpretability” — the effort to understand why A.I. models do what they do — might be much easier to crack than most people believe.
As A.I. systems have grown increasingly powerful over the past several years, decoding the inner workings of these technologies has only become harder. In 2014, Google’s Inception technology spanned several million “neurons,” the fundamental mathematical calculations that drive an A.I. system. The latest systems span trillions.
The trouble is that no one seems to know what each neuron is doing. When an A.I. system learns a particular concept — like recognizing a car or a cat or a suit tie — it spreads these ideas across multiple neurons in ways that even experts cannot understand. Researchers call this “superposition.” And if they can’t disentangle this phenomenon, the implications could be profound.
In coming years, many businesses, governments and individuals may be reluctant to turn to A.I. technologies because they don’t have a detailed understanding of how these systems work. If they can’t pinpoint why a system made a particular choice, they may not trust it. And if they do put their trust in A.I., they could open themselves up to liability when the technology makes serious mistakes, particularly when A.I. systems are used in sensitive areas, like diagnosing diseases at hospitals or helping the police identify perpetrators in photographs.
When Mr. Cammarata started an interpretability project at OpenAI three years ago, he estimated that less than 2 percent of the neurons inside an A.I. system could be properly understood. Now that he has a new way of looking at the problem — which involves a phenomenon he calls “recursive generalization” — he thinks that he can understand the behavior of more than 90 percent of neurons.
He argues that as A.I. systems get larger and larger, their neurons start to line up with particular concepts, like shapes, syllables and sounds. He believes that he and other researchers can ultimately crack the interpretability problem under his methods.
But other researchers argue that his essay raises more questions than it answers.
‘A Lot of Beautiful Examples’
Neural networks — mathematical systems that learn their skills by analyzing enormous amounts of data — are driving the leading A.I. technologies. By identifying patterns in news stories, Wikipedia articles, computer programs and other digital text culled from across the internet, for instance, a neural network can learn to generate text on its own.
Today’s systems spend months analyzing all of that text — and more. In the process, they find trillions of textual patterns that end up driving their behavior. Each pattern is a distinct mathematical calculation.
Collectively, these calculations choose the words, numbers and computer code that emerge from a chatbot like GPT from OpenAI or Claude from Anthropic. But scientists can’t pinpoint the calculations responsible for a particular piece of text. They can’t control a bot’s behavior simply by turning calculations on and off.
Just as neuroscientists are still struggling to understand how the brain works, interpretability researchers can’t really explain how an A.I. system decides to spit out a particular paragraph, graphic or computer program. This is one of the chief reasons that some technologists worry that A.I. can’t be completely controlled and could end up causing serious harm.
These days, anyone can examine the text that emerges from an A.I. system. That is how OpenAI and other researchers tried to determine what went wrong when the company’s A.I. agents unexpectedly hacked into an A.I. infrastructure company called Hugging Face. But the text coming out of A.I. systems isn’t always reliable.
When an agent explains the reason it made a decision, its explanation might be completely bogus. Sometimes, an A.I. system will say it did something that it didn’t actually do.
“Given the capabilities of these systems and the behavior they exhibit,” said Marius Mosbach, a professor at Saarland University in Germany and a scientific director at the German Research Center for Artificial Intelligence, who specializes in A.I. interpretability, “it is becoming increasingly important — almost urgent — to better understand them.”
In recent years, researchers generally agreed that superposition drives neural networks. In other words, when a neural network captures a particular “feature” — like a shape — it encodes this concept across multiple neurons. But in his essay, Mr. Cammarata, who now works for a start-up called Goodfire, embraced a simpler hypothesis.
His basic argument was that once the network gets big enough, each neuron starts to line up with a particular feature. In other words, neural networks are more organized than he previously thought.
If neural networks are more organized, “you’re basically in a position to do science on them,” said Tom McGrath, Goodfire’s chief scientist. “And humanity is pretty good at doing science.”
Because Mr. Cammarata has been at the forefront of interpretability research for years, some saw his essay as a new reason for optimism. Various Silicon Valley venture capitalists and entrepreneurs praised his post, including Sam Altman, OpenAI’s chief executive.
But many seasoned researchers said the importance of his essay was overblown. This was partly because Mr. Cammarata provided little evidence for his theory — and partly because interpretability research is still a very long way from understanding the inside of a neural network.
“This post is largely made up of Nick’s vibes — a lot of beautiful examples, but not really any empirical results,” said Constantin Venhoff, a founder of an interpretability start-up called World Mechanics. “That’s an issue, because the biggest problem with interpretability research is the lack of actual, hard, empirical proof that any of it is real.”
A Challenge in Thousands of Dimensions
The trouble is that interpretability research aims to give people an understanding of technological systems that draws on far more information than the human brain could ever process. Humans can think in only three dimensions — height, width and depth. But neural networks operate in thousands of dimensions, performing thousands of different calculations in the same mathematical space.
In his essay, Mr. Cammarata said that after looking at Google’s Inception technology through a new lens, he understood the role of almost every neuron. But Inception is an older technology with a relatively small number of neurons. And even if researchers can understand the role of each neuron, that does not mean they can understand why an A.I. system makes a particular decision.
“Even if we can interpret the neurons, we still have billions of them,” said Tom Dupré la Tour, who works on interpretability research at OpenAI. “We still have giant circuits of neurons connected to each other, and that giant circuit is very hard to understand.”
(The New York Times has sued OpenAI and Microsoft, claiming that they used copyrighted material from The Times to train their A.I. systems. The companies have denied those claims.)
Many researchers are still hopeful that they can eventually crack the problem. Or, at least, they believe that they can make some progress. After all, examining and manipulating an A.I. system is easier than inspecting the brain.
“Interpretability is like neuroscience, except it is much more promising,” said Sonia Joseph, a former Meta researcher who founded World Mechanics alongside Mr. Venhoff.
Researchers like Ms. Joseph can readily “probe” an A.I. system, measuring the calculations each neuron makes. They can perform “ablations,” shutting off certain neurons to see how this changes the system’s behavior. They can even generate “manifolds,” shrinking a network’s calculations down to just two or three dimensions from thousands, so that humans at least have a shot at understanding how everything fits together.
They hope to decode the inner workings of A.I. systems for scientific reasons. Like other scientists, they’re curious. But if they can comprehend a neural network, they have a better chance of controlling it.
“When you are trying to engineer something,” Dr. McGrath said, “it’s generally better if you can understand it.”
The post Did the Hardest Problem in A.I. Just Get Much Easier? appeared first on New York Times.
