In downtown Berkeley, an old hotel has become a temple to the pursuit of artificial intelligence and the future of humanity. Its name is Lighthaven.
Covering much of a city block, this gated complex includes five buildings and a small park dotted with rose bushes, stone fountains and neoclassical statues. Stained glass windows glisten on the top floor of the tallest building, called Bayes House after an 18th-century mathematician and philosopher.
Lighthaven is the de facto headquarters of a group who call themselves the Rationalists. This group has many interests involving mathematics, genetics and philosophy. One of their overriding beliefs is that artificial intelligence can deliver a better life if it doesn’t destroy humanity first. And the Rationalists believe it is up to the people building A.I. to ensure that it is a force for the greater good.
The Rationalists were talking about A.I. risks years before OpenAI created ChatGPT, which brought A.I. into the mainstream and turned Silicon Valley on its head. Their influence has quietly spread through many tech companies, from industry giants like Google to A.I. pioneers like OpenAI and Anthropic.
Many of the A.I. world’s biggest names — including Shane Legg, a co-founder of Google’s DeepMind; Anthropic’s chief executive, Dario Amodei; and Paul Christiano, a former OpenAI researcher who now leads safety work at the U.S. Center for A.I. Standards and Innovation — have been influenced by Rationalist philosophy. Elon Musk, who runs his own A.I. company, said that many of the community’s ideas align with his own.
Mr. Musk met his former partner, the pop star Grimes, after they made the same cheeky reference to a Rationalist belief called Roko’s Basilisk. This elaborate thought experiment argues that when an all-powerful A.I. arrives, it will punish everyone who has not done everything they can to bring it into existence.
But these tech industry leaders stop short of calling themselves Rationalists, often because that label has over the years invited ridicule.
The Rationalist community is tightly entwined with the Effective Altruism movement, which aims to remake philanthropy by calculating how many people would benefit from each donation. This form of utilitarianism aims to benefit not just people who are alive today, but all the people who will ever live. Many Effective Altruists, or E.A.s, have decided that the best way to benefit humanity is to protect it from destruction by A.I.
Rationalists often identify as E.A.s. And E.A.s often adopt Rationalist philosophies. Together, these two movements have pumped hundreds of millions of dollars into companies, research labs and think tanks that aim to build A.I. and ensure its safety. The biggest funders include wealthy tech moguls like Jaan Tallinn, a creator of the internet calling service Skype, and Dustin Moskovitz, a Facebook co-founder.
“They built a vast, well-funded ecosystem to spread, amplify and validate their ideology,” said Mollie Gleiberman, an anthropologist who has studied the rise of the Rationalists and Effective Altruism.
Whether they are right or wrong in their near-religious concerns about A.I., the tech industry is reckoning with their beliefs.
In late 2023, OpenAI’s chief executive, Sam Altman, was briefly removed from his job because board members with ties to the Rationalist and E.A. movements said they could not trust him to build A.I. for the benefit of humanity.
Lighthaven is a physical manifestation of just how much these ideas have suffused Silicon Valley and the San Francisco Bay Area — a modern day temple.
The main building, called Aumann Hall, after the Israeli game theorist Robert Aumann, offers seven bedrooms and multiple common areas for parties and weekend conferences. Eigenspace, named for an esoteric mathematical concept, includes a gym and another communal area large enough for 40 people. Two hundred people can fan out across the synthetic grass-covered park, which also has chairs and electric fire pits.
“It’s a place where serendipity can happen. Some people liken it to a college campus or the M.I.T. Media Lab,” said Alex K. Chen, a longtime member of the community, referring to the design lab at Massachusetts Institute of Technology.
Each spring, Lighthaven hosts LessOnline, a conference where bloggers and commenters from Rationalist websites meet in person. Every Tuesday at 6:30 p.m., almost like a Bible study, people gather to read and discuss The Sequences, the ur-text that gave rise to the movement.
“Religion is text and story and ritual,” said Ilia Delio, a Franciscan sister and professor of theology at Villanova University. “All of that applies here.”
The Rationalist movement is a lifestyle as much as a set of ideas. The adherents have mixed their focus on A.I. with advice for how to live your life and manage your career. The community embraces unconventional ideas, including polyamory and the genetics of intelligence as well as Effective Altruism which is also a lifestyle. And for aspirational A.I. developers, Rationalist events have become essential networking opportunities.
Gatherings like the Machine Learning Alignment and Theory Scholars program, or MATS, hosted at Lighthaven each summer, are a more important way of getting into the field of A.I. safety than academia, said Sonia Joseph, an A.I. researcher at McGill University in Montreal and the tech giant Meta.
The Rationalists emerged in the late 2000s when an online philosopher named Eliezer Yudkowsky wrote The Sequences, a collection of essays that taught people to re-examine the world through cold and careful thought. This often involved using statistics and probability to inform their decisions. Part tutorial, part entertainment, part mystic journey, Mr. Yudkowsky’s essays became a manual for the Rationalist community.
In 2010, Mr. Yudkowsky introduced the founders of a British A.I. company called DeepMind to the venture capitalist Peter Thiel, helping to get their company off the ground. Less than four years later, DeepMind was acquired by Google for $650 million. Now its technology and executives are leading the tech giant’s A.I. efforts.
Mr. Yudkowsky also ran a nonprofit dedicated to A.I. safety called the Machine Intelligence Research Institute in Berkeley. Slowly, the movement went global. Rationalist group houses appeared in cities like New York and Boston. Meetings were held in Britain, the Netherlands and Australia.
The first international Effective Altruism summit was held in 2013 at a group house in Oakland, Calif., that served as the live-in headquarters for Leverage Research, a start-up with deep ties to the Rationalist community. Leading figures in the Rationalist community like Mr. Yudkowsky and Mr. Tallinn helped guide the E.A. movement toward their shared concerns about artificial intelligence.
Criticism of the Rationalist and E.A. movements has been frequent, including claims of sexual harassment in group houses and complaints about the community’s interest in eugenics and race science. The community’s reputation was damaged in 2023 after Sam Bankman-Fried, founder of the cryptocurrency exchange FTX, who was one of the primary financial backers of the two movements, was convicted of fraud. But the movement continues to prosper.
Mr. Bankman-Fried had become a financial trader to benefit the most people through E.A. causes, including the fight to keep A.I. safe. In the end, he was found guilty of stealing $8 billion from his customers.
“When you think about the billions at stake and the radical transformation of lives across the world because of the eccentric vision of this group, how much more cult-y does it have to be for this to be a cult? Not much,” said Greg M. Epstein, a Harvard chaplain who saw the rise of the Rationalist and E.A. communities at the university over the last decade and the author of “Tech Agnostic,” a book that discusses technology as a new religion.
“What do cultish and fundamentalist religions often do?” Mr. Epstein added. “They get people to ignore their common sense about problems in the here and now in order to focus their attention on some fantastical future.”
Each December, hundreds from the community gather in places like the Chabot planetarium in the Oakland Hills and the Freight and Salvage music hall in downtown Berkeley for an annual holiday tradition. They celebrate the Winter Solstice, the longest night of the year, with songs, stories, humor and questions about the fate of the world.
The most recent celebration opened with a song called Uplift, which praised the power of technology throughout human history. Backed by guitar, violin and keyboards, two singers began in the Stone Age and finished in the future. “Light to push the sails, read the data, cities glow! Hands type the keys, click the mouse, out we go!” they sang. “Our voices carry ’round the world and into space! Send us out to colonize another place!”
But Ozy Brennan, a longtime Bay Area Rationalist who served as master of ceremonies that night, warned of clouds ahead. “We face a number of threats our ancestors couldn’t have imagined: nuclear war, bioengineered pandemics, artificial intelligence,” he said. “If we fail — and there is every chance we might — 100 percent of the children will die, and so will everyone else.”
Lighthaven’s main building, a Tudor-style home with a pink-and-white facade, was built in 1905. In the 1970s, it became a bed-and-breakfast called the Rose Garden Inn and soon joined Berkeley’s list of historic landmarks.
About three years ago, the property was purchased for $16.5 million by a company called Lightcone Rose Garden, according to property records. The company was owned by Lightcone Infrastructure, which runs LessWrong, the primary online home of the Rationalists. “Light cone” is a physics term the Rationalists and the E.A.s often use to describe the volume of future events they can influence from the current point in time.
Now, Lightcone runs Lighthaven, too. The staff that oversees the property includes Ray Arnold, who organized the first Secular Solstice. It was purchased with money from two of the community’s biggest funders: Mr. Tallinn and Mr. Bankman-Fried, according to a legal complaint. The funds from Mr. Bankman-Fried, which were used as a deposit, were later returned as part of a court settlement.
Outsiders are not always allowed into Lighthaven. Oliver Habryka, the head of Lightcone, declined a request from The New York Times to tour the facility.
Last year, Ms. Joseph, the McGill and Meta researcher, spent the summer at Lighthaven after being accepted into the MATS program. Now almost 30, Ms. Joseph discovered the Rationalist community as a 14 year old, when she started reading “Harry Potter and the Methods of Rationality.” In the 660,000-word serialized novel, also written by Mr. Yudkowsky, Harry Potter refuses to accept the world of wizardry on blind faith, leaning instead on the laws of philosophy, science and rational thinking. It, too, attracted hundreds of people to the community.
“It attracts outsiders,” Ms. Joseph said. “If you are a gay kid in Kansas who is getting no support from religion, you might discover ‘Harry Potter and the Methods of Rationality’ and find a community that is really accepting.”
Like LessWrong, MATS can be an on-ramp to jobs at A.I. companies. Applicants to the program are often chosen by top A.I. researchers at companies like Anthropic. But for Ms. Joseph and others, the program is more than just a career move.
Looking back on her summer at Lighthaven, she remembered the roses and stone cherubs that lined the path to a three-story building with stained-glass windows. She remembered the glistening mirror at the bottom of the stairs that evoked “Harry Potter and the Methods of Rationality.” She remembered the sign on a locked door that read: “Eliezer Yudkowsky’s Office.”
“All of this feels mythic,” she said. “Even the non-Rationalist scientists find this compelling — the same way the Manhattan Project was compelling. We want to work on something mythic.”
Read by Cade Metz
Audio produced by Adrienne Hurst.
Cade Metz is a Times reporter who writes about artificial intelligence, driverless cars, robotics, virtual reality and other emerging areas of technology.
The post The Rise of Silicon Valley’s Techno-Religion appeared first on New York Times.
