If there is any field in which the rise of AI is already said to be rendering humans obsolete—in which the dawn of superintelligence is already upon us—it is coding. This makes the results of a recent study genuinely astonishing.
In the study, published in July, the think tank Model Evaluation & Threat Research randomly assigned a group of experienced software developers to perform coding tasks with or without AI tools. It was the most rigorous test to date of how AI would perform in the real world. Because coding is one of the skills that existing models have largely mastered, just about everyone involved expected AI to generate huge productivity gains. In a pre-experiment survey of experts, the mean prediction was that AI would speed developers’ work by nearly 40 percent. Afterward, the study participants estimated that AI had made them 20 percent faster.
But when the METR team looked at the employees’ actual work output, they found that the developers had completed tasks 20 percent slower when using AI than when working without it. The researchers were stunned. “No one expected that outcome,” Nate Rush, one of the authors of the study, told me. “We didn’t even really consider a slowdown as a possibility.”
No individual experiment should be treated as the final word. But the METR study is, according to many AI experts, the best we have—and it helps make sense of an otherwise paradoxical moment for AI. On the one hand, the United States is undergoing an extraordinary, AI-fueled economic boom: The stock market is soaring thanks to the frothy valuations of AI-associated tech giants, and the real economy is being propelled by hundreds of billions of dollars of spending on data centers and other AI infrastructure. Undergirding all of the investment is the belief that AI will make workers dramatically more productive, which will in turn boost corporate profits to unimaginable levels.
On the other hand, evidence is piling up that AI is failing to deliver in the real world. The tech giants pouring the most money into AI are nowhere close to recouping their investments. Research suggests that the companies trying to incorporate AI have seen virtually no impact on their bottom line. And economists looking for evidence of AI-replaced job displacement have mostly come up empty.
None of that means that AI can’t eventually be every bit as transformative as its biggest boosters claim it will be. But eventually could turn out to be a long time. This raises the possibility that we’re currently experiencing an AI bubble, in which investor excitement has gotten too far ahead of the technology’s near-term productivity benefits. If that bubble bursts, it could put the dot-com crash to shame—and the tech giants and their Silicon Valley backers won’t be the only ones who suffer.
Almost everyone agrees that coding is the most impressive use case for current AI technology. Before its most recent study, METR was best known for a March analysis showing that the most advanced systems could handle coding tasks that take a typical human developer nearly an hour to finish. So how could AI have made the developers in its experiment less productive?
The answer has to do with the “capability-reliability gap.” Although AI systems have learned to perform an impressive set of tasks, they struggle to complete those tasks with the consistency and accuracy demanded in real-world settings. The results of the March METR study, for example, were based on a “50 percent success rate,” meaning the AI system could reliably complete the task only half the time—making it essentially useless on its own. This gap makes using AI in a work context challenging. Even the most advanced systems make small mistakes or slightly misunderstand directions, requiring a human to carefully review their work and make changes where needed.
This appears to be what happened during the newer study. Developers ended up spending a lot of time checking and redoing the code that AI systems had produced—often more time than it would have taken to simply write it themselves. One participant later described the process as the “digital equivalent of shoulder-surfing an overconfident junior developer.”
Since the experiment was conducted, AI coding tools have gotten more reliable. And the study focused on expert developers, whereas the biggest productivity gains could come from enhancing—or replacing—the capabilities of less experienced workers. But the METR study might just as easily be overestimating AI-related productivity benefits. Many knowledge-work tasks are harder to automate than coding, which benefits from huge amounts of training data and clear definitions of success. “Programming is something that AI systems tend to do extremely well,” Tim Fist, the director of Emerging Technology Policy at the Institute for Progress, told me. “So if it turns out they aren’t even making developers more productive, that could really change the picture of how AI might impact economic growth in general.”
The capability-reliability gap might explain why generative AI has so far failed to deliver tangible results for businesses that use it. When researchers at MIT recently tracked the results of 300 publicly disclosed AI initiatives, they found that 95 percent of projects failed to deliver any boost to profits. A March report from McKinsey & Company found that 71 percent of companies reported using generative AI, and more than 80 percent of them reported that the technology had no “tangible impact” on earnings. In light of these trends, Gartner, a tech-consulting firm, recently declared that AI has entered the “trough of disillusionment” phase of technological development.
Perhaps AI advancement is experiencing only a temporary blip. According to Erik Brynjolfsson, an economist at Stanford University, every new technology experiences a “productivity J-curve”: At first, businesses struggle to deploy it, causing productivity to fall. Eventually, however, they learn to integrate it, and productivity soars. The canonical example is electricity, which became available in the 1880s but didn’t begin to generate big productivity gains for firms until Henry Ford reimagined factory production in the 1910s. Some experts believe that this process will play out much faster for AI. “With AI, we’re in the early, negative part of the J-curve,” Brynjolfsson told me. “But by the second half of the 2020s, it’s really going to take off.” Anthropic CEO Dario Amodei has predicted that by 2027, or “not much longer than that,” AI will be “better than humans at almost everything.”
These forecasts assume that AI will continue to improve as fast as it has over the past few years. This is not a given. Newer models have been marred by delays and cancellations, and those released this year have generally shown fewer big improvements than past models despite being far more expensive to develop. In a March survey, the Association for the Advancement of Artificial Intelligence asked 475 AI researchers whether current approaches to AI development could produce a system that matches or surpasses human intelligence; more than three-fourths said that it was “unlikely” or “very unlikely.”
OpenAI’s latest model, GPT-5, was released early last month after nearly three years of work and billions in spending. (The Atlantic entered into a corporate partnership with OpenAI in 2024.) Before the launch, CEO Sam Altman declared that using it would be the equivalent of having “a legitimate Ph.D.-level expert in anything” at your fingertips. In a few areas, including coding, GPT-5 was indeed a major step up. But by most rigorous measures of AI performance, GPT-5 turned out to be, at best, a modest improvement over previous models.
The dominant view within the industry is that it is only a matter of time before companies find the next way to supercharge AI progress. That could turn out to be true, but it is far from guaranteed.
Generative AI would not be the first tech fad to experience a wave of excessive hype. What makes the current situation distinctive is that AI appears to be propping up something like the entire U.S. economy. More than half of the growth of the S&P 500 since 2023 has come from just seven companies: Alphabet, Amazon, Apple, Meta, Microsoft, Nvidia, and Tesla. These firms, collectively known as the Magnificent Seven, are seen as especially well positioned to prosper from the AI revolution.
That prosperity has largely yet to materialize anywhere other than their share prices. (The exception is Nvidia, which provides the crucial inputs—advanced chips—that the rest of the Magnificent Seven are buying.) As The Wall Street Journal reports, Alphabet, Amazon, Meta, and Microsoft have seen their free cash flow decline by 30 percent over the past two years. By one estimate, Meta, Amazon, Microsoft, Google, and Tesla will by the end of this year have collectively spent $560 billion on AI-related capital expenditures since the beginning of 2024 and have brought in just $35 billion in AI-related revenue. OpenAI and Anthropic are bringing in lots of revenue and are growing fast, but they are still nowhere near profitable. Their valuations—roughly $300 billion and $183 billion, respectively, and rising—are many multiples higher than their current revenues. (OpenAI projects about $13 billion in revenues this year; Anthropic, $2 billion to $4 billion.) Investors are betting heavily on the prospect that all of this spending will soon generate record-breaking profits. If that belief collapses, however, investors might start to sell en masse, causing the market to experience a large and painful correction.
During the internet revolution of the 1990s, investors poured their money into basically every company with a “.com” in its name, based on the belief that the internet was about to revolutionize business. By 2000, however, it had become clear that companies were burning through cash with little to show for it, and investors responded by dumping the most overpriced tech stocks. From March 2000 to October 2002, the S&P 500 fell by nearly 50 percent. Eventually, the internet did indeed transform the economy and lead to some of the most profitable companies in human history. But that didn’t prevent a whole lot of investors from losing their shirts.
The dot-com crash was bad, but it did not trigger a crisis. An AI-bubble crash could be different. AI-related investments have already surpassed the level that telecom hit at the peak of the dot-com boom as a share of the economy. In the first half of this year, business spending on AI added more to GDP growth than all consumer spending combined. Many experts believe that a major reason the U.S. economy has been able to weather tariffs and mass deportations without a recession is because all of this AI spending is acting, in the words of one economist, as a “massive private sector stimulus program.” An AI crash could lead broadly to less spending, fewer jobs, and slower growth, potentially dragging the economy into a recession. The economist Noah Smith argues that it could even lead to a financial crisis if the unregulated “private credit” loans funding much of the industry’s expansion all go bust at once.
If we do turn out to be in an AI bubble, the silver lining would be that fears of sudden AI-driven job displacement are overblown. In a recent analysis, the economists Sarah Eckhardt and Nathan Goldschlag used five different measurements of AI exposure to estimate how the new technology might be affecting a range of labor-market indicators and found virtually no effect on any of them. For example, they note that the unemployment rate for the workers least exposed to AI, such as construction workers and fitness trainers, has risen three times faster than the rate for the workers most exposed to it, such as telemarketers and software developers. Most other studies, though not all, have come to similar conclusions.
But there’s also a weirder, in-between possibility. Even if AI tools don’t increase productivity, the hype surrounding them could push businesses to keep expanding their use anyway. “I hear the same story over and over again from companies,” Daron Acemoglu, an economist at MIT, told me. “Mid-to-high-level managers are being told by their bosses that they need to use AI for X percent of their job to satisfy the board.” These companies might even lay off workers or slow their hiring because they are convinced—like the software developers from the METR study—that AI has made them more productive, even when it hasn’t. The result would be an increase in unemployment that isn’t offset by actual gains in productivity.
As unlikely as this scenario sounds, a version of it happened in the not-so-distant past. In his 2021 book, A World Without Email, the computer scientist Cal Newport points out that beginning in the 1980s, tools such as computers, email, and online calendars allowed knowledge workers to handle their own communications and schedule their own meetings. In turn, many companies decided to lay off their secretaries and typists. In a perverse result, higher-skilled employees started spending so much of their time sending emails, writing up meeting notes, and scheduling meetings that they became far less productive at their actual job, forcing the companies to hire more of them to do the same amount of work. A later study of 20 Fortune 500 companies found that those with computer-driven “staffing imbalances” were spending 15 percent more on salary than they needed to. “Email was one of those technologies that made us feel more productive but actually did the opposite,” Newport told me. “I worry we may be headed down the same path with AI.”
Then again, if the alternative is a stock-market crash that precipitates a recession or a financial crisis, that scenario might not be so bad.
The post Are We in an AI Bubble? appeared first on The Atlantic.
