AI company CEOs Sam Altman (OpenAI), Demis Hassabis (Google DeepMind), and Dario Amodei (Anthropic) disagree on a lot, like how fast the technology should develop, the best way to regulate it, and how to prepare society for smarter-than-human AI, among other things.
That makes it all the more remarkable that they — along with 85 other experts in tech, biology, and national security policy — recently signed on to an open letter calling for more robust regulations around gene synthesis. They’re all concerned that AI systems might be used to help develop and even deploy dangerous biological weapons designed through gene synthesis, which is used to chemically build custom DNA sequences in a lab, rather than relying solely on existing natural DNA templates.
The simple fact of multiple CEOs of fiercely competitive AI companies aligning on anything is remarkable. But to understand how they came to this agreement, we have to take a step back to understand what gene synthesis is, how it works, and why the possibility of AI-assisted misuse of the technology generates so much fear.
Modern microbiology owes a lot to gene synthesis. Researchers can order synthetic genes from commercial DNA providers to develop new vaccines, drugs, and gene therapies for inherited diseases like hemophilia; produce human insulin, boost agricultural output, and more. Gene synthesis is a foundational technology for successful CAR-T cell therapies for cancer and many diagnostic tools. The demand for synthetic DNA is growing globally, and it’s never been cheaper or simpler to write genetic code.
But for all its power, gene synthesis also carries substantial risk. The same technology that can enable life-saving new gene therapies can also assist in the creation of deadly pathogens by assembling some of the same nucleotides — the genetic building blocks that create the code for all of life — in a different order.
Most US companies that provide gene synthesis services screen orders for genetic sequences of concern, such as those that can make a pathogen more dangerous or transmissible, and to verify that customers are legitimate. They do so voluntarily, well aware of the potential dangers.
But not every provider does so. “As long as screening remains voluntary, some companies will not do it,” Becky Mackelprang, the director for security programs at the Engineering Biology Research Consortium, told me over email. There’s a real risk that bad actors could find a gene synthesis company with more lax standards, and that might mean disaster.
We’ve been fortunate so far. “This technology has been commercially deployed for more than 20 years and has never been misused to cause harm,” James Diggans, the vice president of policy and biosecurity at gene synthesis company Twist Bioscience, told me over email.
But AI threatens to complicate matters, opening up new frontiers of risk.
For good or for ill
Both large language models (LLMs) and AI biodesign tools enable scientists to design entirely novel genetic sequences. This is a boon for industrial and medical applications — and a challenge for current screening systems, which use similarity to known pathogenic or toxic sequences in order to detect risk. A screening system should catch someone trying to order sequences of a known dangerous virus like Ebola, for example, but it might miss a new sequence that could still be risky. Last year, a study published in Science demonstrated that our screening systems have kept pace with AI capabilities so far. “But the industry clearly understands this will not be the case forever,” Diggans said.
Mackelprang is worried that AI could reduce the knowledge barriers that have historically prevented bad actors from developing bioweapons. Frontier AI systems, for example, seem to already outperform expert virologists on questions about performing complex laboratory procedures.
But there is knowing and there is doing, and biological lab work is still hard. “Researchers spend years trying to make a protocol work even after consulting directly with others who have perfected that exact same protocol. I think AI can help someone ‘level up’ their laboratory skills, but I do not think AI can enable someone without any biological training to create a serious hazard,” Mackelprang told me.
That means that gene synthesis companies are still a primary chokepoint for anyone trying to produce a novel genetic sequence. Mackelprang’s main concern is that aspiring bioterrorists might use AI to generate harmful genetic sequences that can evade current or future screening systems. “In the near term, I think the likelihood of these types of misuse are quite low. But when the potential consequences are severe and technologies continue to develop rapidly, we have a responsibility…to develop reasonable prevention and mitigation options,” she said.
Maximizing the benefits of gene synthesis while minimizing the risks is difficult, but not impossible. That’s why Diggans and Mackelprang — along with Altman, Hassabis, and Amodei, as well as other gene synthesis providers, tech entrepreneurs, life science executives, and national security experts — signed the open letter calling for mandatory gene synthesis screening and recordkeeping of orders.
Co-organized by the think tanks Institute for Progress and the Foundation for American Innovation, the open letter also calls for providers to record synthesis orders and sequence data to support biosecurity investigations “so that any threat that might evade initial screening can be traced back to its source…Awareness of traceability itself deters misuse.” This would, ideally, address Mackelprang’s concern that AI might eventually help bad actors evade existing screening protocols.
“Screening every DNA synthesis order before it’s manufactured is the kind of unglamorous, common-sense step that prevents a much bigger problem later,” DJ Kleinbaum, the co-founder of the biotech startup Emerald Cloud Lab, an automated lab scientists can access remotely, and one of the signatories, said.
But Altman, Hassabis, and Amodei’s shared signatures may be the most meaningful evidence that the letter matters. For all their disagreements, they are well aware that their tools can be used for tremendous — even catastrophic — harm.
AIxBio risk: A thing on which we can all agree
While it’s far from the first time frontier AI companies have spoken to AI-enabled biological risk, the open letter is the first place they’ve come together to do so in a single voice. “Support for screening does not depend on any particular view of AI,” the letter reads. “This is a rare moment of agreement across stakeholders that are often at odds.”
The letter calls for Congress to act now. “We applaud the legislative efforts currently underway,” the letter says, alluding to the bipartisan Biosecurity Modernization and Innovation Act, a bill that gives the Department of Commerce a year to develop new gene synthesis screening rules. The letter also suggests that US states should implement screening requirements based on federal and industry guidelines to create a unified national standard rather than an inconsistent set of laws.
The letter isn’t about applying biosecurity regulations to the AI companies themselves, which likely would have limited the number of tech signatories. (Though major companies do actively try to prevent their models from giving away dangerous biological knowledge, albeit not always successfully.) Focusing on screening is tractable, has the buy-in of several gene synthesis providers, and provides a concrete example of how AI can lower the barrier to doing both great and terrible things. And of course, it’s ultimately something a human being has to do at this point.
The AI companies are actively thinking about the catastrophic risks that their technologies might enable. Anthropic is hiring a technical chemical, biological, radiological, and nuclear threat investigator for its threat intelligence team. In May, after launching GPT-Rosalind, a frontier model to accelerate life sciences research and drug discovery, OpenAI introduced Rosalind Biodefense, a program that allows trusted developers to use GPT-Rosalind to build biodefense tools. On June 4, the day after the open letter went live, security specialists at OpenAI and Anthropic served as panelists at the Bipartisan Commission for Biodefense’s meeting on AI and biological threats.
But according to Twist Biosciences’s Diggans, the best way to defend against misuse of AI models to design harmful pathogens is to use AI models as defense. These defensive models can be used to detect attempted misuse before anything happens. DNA synthesis companies can employ these models to ensure orders for highly-engineered sequences are given the same scrutiny and evaluation as orders for naturally occurring sequences.
“[Gene synthesis] companies have to agree to have their order screened not just against a list of sequences but by an AI that people agree is smart enough to recognize and thwart an adversary who’s trying to build a deadly pathogen,” David Haussler, the scientific director of the UC Santa Cruz Genomics Institute and a signatory of the open letter, told me.
Using AI to protect against AI
The good news is that this work is already underway. Last year, I reported that OpenAI provided $30 million in seed funding to biodefense startup Valthos, which develops frontier AI systems to detect biological threats and create medical countermeasures. Valthos’s co-founder Kathleen McMahon signed the open letter.
In September 2025, the Coalition for Epidemic Preparedness Innovations (CEPI) and philanthropic nonprofit Sentinel Bio created the Pandemic Preparedness Engine AI platform (sometimes referred to simply as “the Engine”). They’re taking a biosecurity-by-design approach, considering biosecurity risks from the outset. “This includes a multi-layered approach to biosecurity: from protecting biosecurity-sensitive data needed to train the AI to carefully managing who has access to the Engine and monitoring how they use it,” Sarah Carter, a biosecurity consultant at CEPI, told me over email.
Users of the Pandemic Preparedness Engine would use AI prompts to interact with the system, similar to how people use consumer platforms. User prompts could be monitored in real time by a specialized AI agent built to assess the risk of misuse potential or attempts to “jailbreak” an LLM to get it to generate prohibited content, such as the “recipe” for assembling a deadly virus.
Still, even commercially available technologies may present problems of their own. This week, Anthropic launched Claude Fable 5, a version of its highly powerful and restricted Mythos model that the company has aimed to make safe for public use. Claude automatically stops use of Fable if it detects requests involving cybersecurity, biology, chemistry, or distillation (attempting to extract Claude’s capabilities to train competing AI models), shunting those requests to a less powerful model. Users have complained that trying to discuss biology for legitimate purposes with Fable 5 results in the model refusing to engage or defaulting to less capable models instead. The Fable example shows that it’s possible to overcorrect, limiting the potential upside of using AI for the life sciences.
“The major providers of LLMs are doing their best to prevent the models from answering questions that would enable somebody to do something dangerous,” Haussler told me. “[But] the end product of jailbreaking an LLM that’s capable of teaching you how to build a deadly virus is that you now have an LLM that’s capable of teaching anybody how to build a very dangerous virus. And we don’t want that to happen.”
It’s here that the letter’s signatories hope they can stop a still-simmering problem before it comes to a full boil. “Mandatory synthesis screening is that rare case where a threat is clearly visible and substantial prevention clearly achievable before any crisis has occurred,” said Richard Danzig, a natural security expert who served as the 71st Secretary of the Navy under former President Bill Clinton. “Opportunities to act in advance are unusual in this field. I think we should take this one.”
The post The next AI safety fight may actually be about DNA appeared first on Vox.
