Depending on your feelings about artificial intelligence, the Anthropic researcher Jacob Coxon’s resignation this month could be interpreted as a courageous act of truth-telling, an outburst of sci-fi-addled alarmism or an elaborate media stunt.
But to Coxon himself, it was clearly at least one more thing: a plea for labor action.
“If you are a lab researcher, I urge you to consider what the next few years will actually feel like,” Coxon wrote, after disclosing that he and many of his former colleagues believe A.I. could extinguish humanity. “Should you put your head down because ‘it’s happening anyway’ — or take this moment to call for different conditions?”
Coxon and the A.I. researchers who echoed his plea on social media seemed to be making a simple calculation. That if anyone’s going to save our species from the uncomfortably high risk of apocalypse, it’s not going to be A.I. executives or investors or the Trump administration — all of whom generally want to speed up development rather than slow it down. To the extent there’s likely to be a savior, Coxon seems to believe, it’s going to be the workers.
It’s not a crazy idea. Only so many people have the technical skills to push the A.I. frontier forward. Their scientific knowledge makes them uniquely credible when cataloging the technology’s dangers. And their relative scarcity gives them considerable bargaining power, at least in the short-term. If they decided en masse to stop building A.I. tomorrow, research at the leading labs could largely stall out. That makes for a highly specific form of influence that can be used for the public good.
Many have already tried. Researchers at prominent A.I. labs have signed letters urging steps to promote safety in recent months and joined an amicus brief in a related legal case. One group, the Coalition of Concerned AI Staff, came together in February as a way for researchers to share their A.I. anxieties and compare notes on how to limit its perils. Some had harbored anxieties for years but became more concerned as the federal government resisted safeguards on using A.I. for surveillance and weapons.
The coalition seemed to view Coxon’s resignation as an opportunity to ramp up. “If you work at an A.I. company, you might be asking if you, too, should leave,” a convener of the group wrote a few days later on social media. “I hope that, if you stay, you’re also asking what about how to use your leverage in staying.”
But what can A.I. workers actually do with that leverage? The country’s experience with another existentially dangerous technology — nuclear weapons — is instructive. It suggests that if A.I. employees are to influence their industry, their strategy probably wouldn’t resemble the approach favored by factory workers or even grad students, who have pushed around their employers by going on strike or threatening to.
In practice, when nuclear scientists have managed to rein in the technology they helped create, the approach has tended to be less direct — more bully pulpit than brute force. Could it work this time, too?
‘I Write This to Frighten You’
It wasn’t a viral tweet, but the Nobel Prize-winning chemist and Manhattan Project contributor Harold Urey knew how to capture the public imagination when he wrote about nuclear weapons in early 1946.
“I write this to frighten you,” Urey began an essay in Collier’s that he co-wrote with a publicist. “I’m a frightened man, myself. All the scientists I know are frightened — frightened for their lives — and frightened for your life.”
Like today’s A.I. researchers, Urey and other scientists had joined the race to build a nuclear bomb because, somewhat paradoxically, they were concerned about its dangers — they were terrified that Germany would get the weapon first. It was only natural that their safety concerns continued after the war, said Peter Kuznick, an expert on scientists’ political activism at American University.
When these scientists surveyed the wreckage of Hiroshima and Nagasaki, they came to two conclusions: that nuclear technology should be controlled by an international authority, not individual countries. And that, until then, civilians should take control of the U.S. program from the military, which they worried would focus excessively on weapons, insist on secrecy and potentially undermine democracy. Urey wrote that leaving the military in control was similar in intent to “the transfer of power from the German Reichstag to Hitler.”
Within weeks of the nuclear bombings, hundreds of physicists, chemists and engineers began forming organizations at their Manhattan-Project labs and lobbying for these priorities, as the historian Jessica Wang has described. They later formed a national committee that included labor unions, professional associations and religious groups, whose members summed to the millions.
Some scientists wrote articles, sat for interviews and gave congressional testimony describing the horrors of nuclear war. (“The blood does not coagulate but oozes in many spots through the unbroken skin,” recalled one physicist who arrived at Hiroshima not long after the bomb.) Others fanned out across the country giving speeches at Rotary and Kiwanis clubs and attending meetings at churches and schools.
One installment of the national committee’s publication, Atomic Information, introduced readers to “Mrs. W.,” a housewife from Pittsburgh who organized a church gathering where some 600 people reportedly showed up to listen to a Manhattan Project scientist named Paul Henshaw. “Sixty questions were asked from the floor! A list of these questions was forwarded to Senator McMahon,” Mrs. W. told the publication, referring to Brien McMahon, a Democrat from Connecticut.
In the fall of 1945, the scientists and their allies helped block a bill that President Truman and the Department of War had supported, which would have very likely cemented the military’s grip on the nuclear program. The next summer, Congress passed a bill, shepherded by Senator McMahon, that put civilians in charge. Despite some compromises, the scientists believed it would promote research into peaceful uses like cheap energy and the spread of that knowledge.
But the campaign fell short of its second goal, an international agency that would oversee nuclear power and prevent countries from weaponizing it, because of a stalemate between the United States and the Soviet Union. Anti-Communist paranoia set in and swept the scientists’ movement offstage.
The limits of this kind of organizing became even more apparent in the 1980s, as the Reagan administration pushed a plan to shoot down nuclear missiles using lasers based in outer space.
Late-Cold War chauvinism helped prop up the science-fiction sounding proposal, which critics nicknamed “Star Wars.” When Congress funded it to the tune of billions of dollars, many physicists were alarmed. They considered the program not just technically infeasible, but dangerous. They worried it would encourage the Soviet Union to produce even more missiles to penetrate a potential U.S. shield. More than 3,500 science professors, researchers and graduate students signed a pledge in 1986 to refuse Pentagon funding for missile-defense research.
The scientists didn’t always present a united front, however. Many physicists at the country’s top departments declined to sign the pledge; some had lucrative consulting relationships with missile-defense contractors and hobnobbed with administration officials.
But the scientists helped expose the idea’s sheer loopiness in the end. The opposition of the rank-and-file scientists conveyed the skepticism to the public, and carried weight with the prominent insiders, who offered up critiques of their own.
“In some sense, the people advocating for boycotts, they’re a barometer of the mood and the opinion of the larger scientific community,” said Benjamin Wilson, a historian at Harvard, whose book “Strange Stability” explores the relationship between scientists and the military-industrial complex. “The higher-level people are sensitive to that.”
When the Reagan administration welcomed in scientists, sharing classified information with a group of prominent physicists for a multiyear study of Star Wars, it ended in political disaster. A 1987 report by the physicists, some of whom had financial ties to the program, concluded that it would take at least a decade before researchers could even make “an informed decision” about whether the technology was workable. Coverage and commentary in major newspapers dwelled on the authors’ impeccable credentials and featured terms like “devastating blow” (to describe the report’s impact) and “scientific dud” (to distill its verdict on the Reagan program), even though the authors thought they were occupying a sensible middle ground that would keep the research dollars flowing.
There are lots of reasons the world has avoided nuclear annihilation, of course. But it’s hard to overlook scientists’ role in making Star Wars a punchline.
Profit vs. Catastrophe
The government’s central role in nuclear weapons meant there was essentially one entity for activist-scientists to sway. Those raising alarms in the A.I. industry, on the other hand, face a fractured landscape of competing companies. That makes it much more difficult to coordinate a change in direction. Any major A.I. lab that unilaterally took a step analogous to putting civilians in control of nuclear technology would almost certainly get left in the dust by its rivals. OpenAI, one of the companies whose rogue A.I. agents have sobered researchers lately, began life as a nonprofit dedicated to building A.I. safely. But it later set up a for-profit entity that has effectively swallowed much of the original company.
Still, the possibility of government regulation — so far minimal when it comes to A.I. — means that pressure from scientists can help rein in the technology by shaping public opinion. That may be happening already. A recent Reuters/Ipsos poll found that the percentage of respondents who say A.I.’s effect on society is negative has increased somewhat in the past month. Nearly three-quarters said that A.I. companies needed to do more to lower the chances of catastrophe.
The particulars of the Star Wars episode are instructive. As with missile defense, the risks of accelerated A.I. development are well-known to scientists and executives at the cutting edge. Yet some of the industry’s most prominent figures — those who stand to earn hundreds of millions or even billions of dollars — are financially conflicted. And their financial stake can skew their thinking, prompting them to emphasize the societal benefits of continuing development and the futility of slowing down.
That’s where the likes of Jacob Coxon come in. Like their predecessors in the nuclear establishment, the leading A.I. figures generally want to think of themselves as responsible and honest, and so they are likely to be hemmed in by an accumulation of public appeals from lower-level colleagues. They can effectively be prodded to speak up in ways that make disaster less likely.
Within days of Coxon’s resignation and the reposts and amens from his relatively anonymous co-workers, his former boss Dario Amodei penned a 3,800-word essay urging a slowdown in A.I. development. Sam Altman, Elon Musk and the chair of Google’s A.I. lab, Demis Hassabis, all quickly endorsed it.
For those who believe that self-improving A.I. models can spiral out of control, a slowdown alone probably wouldn’t suffice. The game going forward is whether the rank-and-file can speak louder than the billions of dollars on the line.
The post ‘I Write This to Frighten You’: How Scientists Took On Existential Risk Once Before appeared first on New York Times.
