Last month, a team of scientists published a high-profile paper describing how they’d taught an AI model to help produce brand-new, deadly viruses. The AI—a “genome-language model” trained on DNA sequences—had designed multiple new genomes that scientists then synthesized into functional viruses. And some of those viruses were capable of killing bacteria in a dish.
News of AI-made viruses went over poorly. The loudest reaction to the study (which had been released as a preprint about a year ago) was fear. Now that AI can design viruses that go after bacteria, perhaps someone could use the technology to manufacture pathogens that harm people—sparking a major outbreak, even a devastating pandemic. This potential threat evoked some of the same concerns that politicians, researchers, and tech leaders (including Bill Gates) have raised. And these fears are only gaining traction: On Thursday, Anthropic released a report that sparked a round of doom-laden headlines, detailing several instances in which the company banned accounts that were using its LLM, Claude, for what could have been reputable research efforts—or efforts to develop biological weapons.
AI, which has repeatedly proved itself capable of exceeding user expectations and bypassing guardrails, is a formidable force; many fears about its potential biological threat stem from its relentless pace of development and the possibility that it could make independent breakthroughs, exceeding researchers’ own understanding, especially if AIs eventually achieve recursive self-improvement—that is, models designing newer and smarter models. Kevin Esvelt, a biologist at MIT and a co-founder of Secure Bio, a biosecurity nonprofit, told me that although he thinks the probability of an AI-driven pandemic remains quite low, the consequences of such a catastrophe would be so large that “it’s very important that we drive that probability down as low as we can.”
But many other biologists, especially infectious-disease experts and virologists, told me that the threat of a pandemic need not loom particularly large on the list of AI-made threats. To the scientists most familiar with the minutiae of growing, modifying, and testing viruses in a lab, and the havoc they can wreak in the real world, the viruses that have already arisen through evolution, or will soon enough, still present much more immediate perils. In other words, humans should be investing aggressively in pandemic preparation, but not because of AI.
To be clear, viruses made with the help of AI could pose some threat. The experts I spoke with agreed that, in the wrong hands, AI could be used to speed infectious destruction, to a degree. To create the bacterium-killing virus, scientists led by a group at Stanford University and the Arc Institute fed the AI model, called Evo, only viral genomes that attack bacteria. Those genomes are much simpler and smaller than ones coding for viruses that go after humans. In theory, though, with a similar technique, researchers could train AI on the genomes of viruses that infect humans, with the aim of manufacturing novel pathogens. Or perhaps—as some people worry is likelier, easier, and more dangerous—users could ask AI to adjust specific traits of viruses known to cause disease, until those pathogens can skirt available vaccines.
Scientists have long been able to manipulate viral genetics without AI. This kind of “gain of function” research has stirred its own political and scientific controversy—especially in discussions about the origins of COVID-19—and the Trump administration has restricted federal funding for it. But current AI models have the capacity to “put a supreme battery in the back of the things that people were already doing,” Brandon Ogbunu, an evolutionary biologist and infectious-diseases modeler at Yale University, told me. Tom Inglesby, the director of the Johns Hopkins Center for Health Security, worries about bad actors using AI to surmount some of the limits of their own technical expertise. Ideally, models powerful enough to suggest new viral genomes should be subject to strict, formal review, Inglesby said; perhaps they could even be limited to training on only certain types of benign data. (For instance, the team that built Evo trained it only on viruses that kill bacteria, and withheld any data about viruses that can harm more complex life forms, including humans.) Esvelt advocates for restricting access to certain models entirely.
Other researchers, however, including the virologists I spoke with, pointed out that another set of safeguards is already in place: how little we humans—the creators and users of AI—understand about pathogenic viruses. For AI to map out more effective pathogens, people must be able to feed it data on what makes viruses highly transmissible, dangerous, and able to circumvent immune safeguards.
But these concepts are complex and murky enough that the chasm AI would need to cross is wide; researchers lack much of the basic knowledge they’d need to train an AI properly on such topics. “Right now, the data is junk going in,” Seema Lakdawala, a virologist at Emory University, told me. Generations of virologists have not come up with firm answers to key questions about what makes certain viruses better at causing diseases than others, or why some people are more susceptible to particular infections than others. Even expert virologists may struggle to ask AI the best prompts to guide it in useful directions.
In her own research on viral transmission, Lakdawala hasn’t found LLMs such as Claude and ChatGPT particularly helpful or correct, she told me, even when she’s asked it questions far simpler than “How do I make a pandemic virus?” (for example, basic questions about the flu-virus genome). Brian Hie, the lead researcher on the team that designed Evo, told me he’s also unconvinced “that AI would even be the method of choice for someone trying to make a dangerous pathogen.” It does not know enough about pathogens, because we don’t know enough about pathogens.
Esvelt, at MIT, argues that AI might not need to understand viruses perfectly, or even that well, to stumble across a dangerous iteration of one. AIs might also improve rapidly enough to make more sense of the existing data than human researchers ever have. But no matter how good AIs get, they so far have no way to build and test new pathogens, Gigi Gronvall, a health-security expert at the Johns Hopkins Bloomberg School of Public Health, told me. Even a very motivated human—whether they wanted to spark a pandemic for a government or terrorist organization, or for their own ends—would still have to pick through AI-generated possibilities, successfully synthesize them, assess them for enhanced (or at least sufficient) transmissibility and virulence, produce a sufficient quantity to do serious damage, and then release that payload of new virus into the world. All of that requires substantial skill in the lab, as well as expensive equipment and lab animals. It’s not impossible, but it certainly isn’t easy, either.
The practice of building new viruses in a laboratory is fickle: “Even if I give two experienced scientists the same protocol, they’re going to do it differently, and the results could come out differently,” Linsey Marr, a viral-transmission researcher at Virginia Tech, told me. Tinkering with viruses the old-fashioned way has also taught researchers that many hypotheses about what should make a virus more contagious or dangerous don’t pan out. Researchers cannot simply tune up a virus’s transmissibility or virulence like a dial; such changes can come with serious costs, including rendering the entire pathogen nonfunctional—incapable of infecting or hurting a person at all.
Evolution is already a master at tinkering with life forms and building upon their most successful—or dangerous—traits. Viruses are constantly trying out new versions of themselves and aren’t limited by human expectations about what “dangerous” agents of disease look like. Evolving pathogens also need no assistance from malevolent actors to start epidemics; in nature, where species constantly mingle, new outbreaks begin all the time. And nature has an elegant way of dealing with mistakes: Unviable iterations of life simply never come to exist at all. As good as AI is getting, “I can’t look anyone in the eye and say that process is any more harmful than what nature is already operating,” Ogbunu, the Yale evolutionary biologist, said.
The virologists I spoke with did agree that AI models should be monitored and regulated, but were wary of limitations that would hinder AI’s ability to speed the development of new vaccines, shore up viral surveillance, boost epidemiological modeling, or otherwise benefit public-health and infectious-disease research. (Even the types of bacterium-killing viruses that caused such an uproar could be deployed against antibiotic-resistant infections.)
Worries about killer viruses are already interfering with such lines of research. Among the examples of potential threats in Anthropic’s report were gain-of-function work on chikungunya virus and research about how flu viruses adapt to mammals—projects that could have been malicious or, in theory, could improve therapeutics or outbreak detection, respectively. (The company said in the report that it couldn’t confirm whether the queries represented the pursuit of knowledge for the public good or attempts at bioterrorism, and had erred on the side of caution.) Lakdawala and Marr told me that, recently, when they have asked Claude or ChatGPT about, say, genetic quirks in the flu-virus genome or using UV light to inactivate viruses, the AI flagged or halted their conversations. “It happens to me almost daily,” Marr said. (An OpenAI spokesperson wrote over email that “today’s models are specifically trained to withhold biological information that could enable harm” and that researchers “pursuing legitimate biological and life sciences work” can apply to its Trusted Access for Cyber program, which requires proof of identification. Anthropic did not respond to a request for comment.)
Lakdawala and Marr are now trying to develop their own framework to help AI models distinguish legitimate scientific pursuits from nefarious ones. Both say that it’s difficult: What would the criteria be? Who would be the final arbiters of those criteria and of ambiguous cases? And none of those guardrails would address the most extreme concerns about viruses created by AI—that a self-improving intelligence might eventually be able to deliberately design, manufacture, and release a pathogen. For now, though, AI is still a tool. We’re still in charge of the questions we ask it, the data we feed it, and the fate of any virus that it might suggest we create.
The post How Real Is the Threat of AI Starting the Next Pandemic? appeared first on The Atlantic.
