“I want AI to be a tool that allows human flourishing!” exclaimed Brad Carson, a former member of Congress. “There is an option out there where AI is just a tool for us.”
This is a normal thing to say in most circles. But Carson was speaking at an invite-only symposium dedicated to the idea of creating a “Worthy Successor” — an AI so impressive, so beyond the mere human, that we’d actually want it to replace humanity.
“You’re a brave man for entering this room!” Dan Faggella, an AI market researcher and organizer of the symposium, told Carson. “You’re in probably the only room in the country where most people disagree with you.”
The attendees at the symposium, which took place at the New York Academy of Sciences last September, are part of a subculture that is growing in importance: the AI successionists, who think that artificial intelligence is our rightful heir — the next step in cosmic evolution. Since they believe AIs could become our moral superiors, they argue it’s actually wrong to try to keep the machines down, or even to align them with human values, as most AI companies aim to do. Instead, we should usher in artificial intelligence as a successor to humanity and hand over the world to it. Even if that means we go extinct.
They know this view is taboo, which is why I was invited only on the condition that I wouldn’t quote anyone other than keynote speakers by name. But suffice it to say that this is not a fringe view. It’s becoming highly influential. People from major AI labs — Anthropic, Google DeepMind, xAI — were in attendance. So were people from think tanks that directly shape the US government’s AI policy.
Why I wrote this story
I grew up hearing an old Jewish teaching: Each of us should carry two slips of paper, one in each pocket. One says, “I am but dust and ashes.” But the other says, “The world was created for me.”
Reporting on AI these past few years, I’ve watched more and more people forget the second message. They think we should be okay with getting obliterated if a more valuable species can take our place. But more valuable to whom? Value isn’t dispensed from some cosmic vantage point; it’s always value to someone. And we’re valuable to us.
And yet the AI successionists are right about something: We can’t expect human beings to look the same a thousand or a million years from now. So how do we decide which kinds of technological change to embrace, and which to refuse? It bothered me that classical humanism doesn’t have a good answer. Here, I’ve sketched what a new one might look like.
AI successionism has been gaining ground among technologists over the past decade. In 2015, Google co-founder Larry Page famously accused Elon Musk of “speciesism” because Page thought we should let digital minds take over, and Musk disagreed.
The successionist vision has been amplified by the advent of effective accelerationism (e/acc) in 2022. Its founder, Guillaume Verdon — the physicist more colorfully known on X as Based Beff Jezos — describes e/acc as a “meta-religion” that’s about “having faith” in the universe’s drive toward increasingly intelligent systems. The best thing we can do is help the universe by developing advanced AI as fast as possible, even at the expense of humanity. “E/acc,” as Verdon has written, “has no particular allegiance to the biological substrate.”
Tech heavyweights have come on board. Venture capitalist Marc Andreessen listed e/acc thinkers as his “patron saints.” Garry Tan, the CEO of tech startup accelerator Y Combinator, included “e/acc” in his social media bio and invested in Verdon’s company, which aims to build the world’s most efficient computers. OpenAI CEO Sam Altman posted on X to Verdon, saying, “you cannot outaccelerate me.”
And these days, AI successionism is spreading beyond Silicon Valley. At the New York symposium, Faggella told the audience that trying to preserve the human species as it is would be silly.
“We could ask the questions that would tie all of our moral aspirations eternally to 23 chromosomes — or we could ask the cosmic questions,” Faggella said.
He wanted us to consider “unpolite, uncouth” possibilities, starting with: The flame of consciousness — the capacity for experience and moral value — may be the rarest and most precious thing in the universe. Humanity is currently a torch carrying that flame, but what if we’re ultimately not the best carrier for it? And if AI can spread that flame far further than we mere humans can, generating experiences of bliss and forms of moral value that we could never even dream of, shouldn’t we let it?
Faggella’s talk was greeted by a loud round of applause. Later, he and a couple dozen attendees headed to a nearby hotel balcony for drinks. And so it was that I found myself overlooking the Manhattan skyline as people talked about the end of humanity over cocktails.
There was some diversity of opinion among the group. Not everyone self-identified with the relatively new term “AI successionist.” Some were proponents of transhumanism, the movement that says we should use tech to proactively evolve our species into Homo sapiens 2.0. Transhumanists hope to keep some version of humanity going, but definitely not the current hardware; they dream of radical life extension, cognitive enhancement, and eventually mind uploading. (Musk, who said he created his brain chip company Neuralink to help humanity merge with AI, probably falls — or at least fell — into this category.) Others were posthumanists, those who want us to give rise to descendants that move beyond humanity altogether.
The biologist sitting across from me was very excited about the prospect of merging humans with AI. He said we should task AI with figuring out how best to do the merger, then “take it off the leash” and allow AI to control its own evolution — and by extension, ours. Of course, he said, not all humans will make it through the transformation; only a select group of people will transition to the next evolutionary stage. (Presumably, the type of people privileged enough to imbibe cocktails at Manhattan AI symposia.)
The man seated beside me, a researcher from one of the major AI companies, was even more radical. Forget merger — it’s okay if humans don’t survive at all, he said. Human text has been used to train the AIs; in some sense, then, the human spirit will live on. “So on the cosmic level,” he said cheerfully, “I’m okay with it.”
Most people are definitely not okay with it. The average person would probably find the answers of the Worthy Successor group repugnant. Yet the core question they pose cannot be ignored. Whether they picture us merging with machines or ultimately being superseded by them, technologists are developing innovations that could dramatically change what it means to be human — think AI-powered brain chips that enable mind-reading or magnetic implants that give you a sixth sense — and genetic tools that could even reshape the DNA of all future generations.
As it becomes possible to direct our own evolution as a species — and potentially even create a new species that surpasses us — we have to decide: How do we know to what extent it does make sense to transform ourselves using technology? What kinds of augmentation do we want, and what kinds do we absolutely not want? What do we wish, ultimately, to become?
This is a moral question, even a spiritual one, and it demands a spiritual response. The AI successionists are offering one. For anyone who finds it repulsive, the challenge is to offer a countervailing positive vision.
And it’s essential to do that now, because as sci-fi as the successionists might sound, they are building real political power, with links to the authoritarian right. Several of the tech heavyweights who’ve embraced successionism want to escape the control of democratic governments, so much so that they’re seeking to create their own sovereign colonies. That can come in the form of space colonies, à la Elon Musk and Jeff Bezos, or in independent “startup cities” or “network states” built by corporations here on Earth — currently Peter Thiel and Marc Andreessen’s favored approach. And Verdon’s investors include entrepreneur Balaji Srinivasan, a major proponent of the network state.
These broligarchs have successfully cozied up to the Trump administration, clearing the way for their accelerationist vision. And they’ll take the wheel unless we come up with an alternate vision for the future.
The natural alternative is humanism, which replaced the medieval view that humans need God to rescue them with the view that humans have the ability, and responsibility, to achieve flourishing through their own efforts. The problem is that, so far, we haven’t developed a version of humanism that’s brave enough to directly tackle the core question — what do we want our species to become? — and answer it compellingly.
The most common “pro-human” response tries to say there are certain fixed traits that make humans unique, and to locate value only in humans as they currently exist. “In the era of artificial intelligence, when human dignity is threatened by new forms of dehumanization, ours is the pressing duty to remain profoundly human,” Pope Leo recently wrote in his encyclical Magnifica Humanitas. This response says: Let’s use tech remedially — to alleviate problems like disease — but let’s not try to augment the species.
That feels insufficient as a guide to the future, because, even before the advent of AI and gene editing, “human” has never been a static category. Homo sapiens has always been evolving and augmenting itself, from the agricultural diet that reshaped our jaws to the algorithms reshaping our attention.
The old formulation is “the naive version of humanism,” Shannon Vallor, a philosopher of technology at the University of Edinburgh, told me recently. “It’s the idea that there’s this blueprint for what a human is and that somehow technology, or any things that change us, take us away from that blueprint — when in fact we’ve been changing ourselves with language, with tools, with architecture, with culture, from the moment we climbed down from the trees.”
A 21st-century humanism needs to say something more sophisticated than just “keep humanity the same.” It needs to have an answer to the question of what we want humanity to become in a tech-augmented world.
But if there is a better vision for our technological future than the one offered by AI successionism, what is it?
AI successionism is a religion, but it’s wearing a secular disguise
Maybe you think it sounds weird to say the AI successionists — a bunch of scientists, technologists, and venture capitalists — are offering a spiritual vision. But their ideas are spiritual in the extreme. And to understand why their movement has gained momentum, we need to understand its deeply religious origins and how it morphed into a supposedly secular worldview. And that means going back.
You probably remember that in the Bible’s Book of Genesis, Adam eats some forbidden fruit and humanity suffers a fall from grace. But did you know that in the Middle Ages, Christian thinkers began to believe that the way to restore humanity to its original perfection was to use…technology? These thinkers argued that part of what it meant for Adam to be formed in God’s image was that he was also a creator, a maker. So if we wanted to truly return to the God-like perfection of Adam prior to his fall, we’d have to lean into that creator aspect of ourselves.
This idea took off in medieval monasteries. Even in the midst of the so-called Dark Ages, some of these institutions became hotbeds of engineering, producing inventions like the first known tidal-powered water wheel and impact-drilled well. For many Christians, tech progress became synonymous with moral progress.
By the Renaissance, some Christian thinkers were insisting that we should progress not just by designing new and innovative objects, but by redesigning ourselves, too. In 1486, philosopher Giovanni Pico della Mirandola argued that what’s unique about us humans is not some static trait but the very freedom of will that allows us to change into whatever we might want. In his Oration on the Dignity of Man, he imagined God telling humankind:
We have made you a creature neither of heaven nor of earth, neither mortal nor immortal, so that with freedom of choice and with honor you may, as the free and proud shaper of your own being, fashion yourself in the form you may prefer.
Pico believed that we could use spiritual technologies like magic to transform our nature. And he argued that we have the choice to become either like the animals or like the angels:
It will be in your power to descend to the lower, brutish forms of life; you will be able, through your own decision, to rise again to the superior orders whose life is divine.
As the dominance of religion waned over the next couple of centuries, Enlightenment thinkers took Pico’s embrace of human plasticity and secularized it. They replaced the concept of divine ascent with one of indefinite progress. They insisted on the “perfectibility” of the human. And they fetishized rational intelligence as the means of achieving that optimal state. “Would it be absurd now to suppose,” wrote 18th-century philosopher Marquis de Condorcet, “that the improvement of the human race should be regarded as capable of unlimited progress?”
Of course, some European thinkers hung onto their Christianity, too, and they found ways to fuse it with the rational humanism of the Enlightenment. It’s a trend that continued into the 1900s, with proponents of Russian Cosmism — an intellectual movement that wanted to achieve literal resurrection of the dead through science — and French Jesuit paleontologist Pierre Teilhard de Chardin, who argued that we could use tech to nudge along human evolution and thereby bring about the kingdom of God.
As author Meghan O’Gieblyn explains in her fantastic book God, Human, Animal, Machine, Teilhard believed that melding humans and machines would lead to “a state of super-consciousness,” whereby we become a new enlightened species. He influenced his pal Julian Huxley, the evolutionary biologist who was president of both the British Humanist Association and the British Eugenics Society, and who popularized the term “transhumanism.”
Huxley inspired the contemporary futurist Ray Kurzweil, who predicted in the 1990s that we were approaching a time when human intelligence could merge with machine intelligence, becoming unbelievably powerful. “The human species, along with the computational technology it created, will be able to solve age-old problems … and will be in a position to change the nature of mortality in a postbiological future,” Kurzweil wrote. And he, in turn, has influenced Silicon Valley heavyweights like Musk, who explicitly aims at merging human and machine intelligence.
But there’s a big problem for these latter-day technologists: While we’ve never had more power to direct the evolution of our species through tech, it’s also never been less obvious what we should evolve toward.
For good old Pico back in the Renaissance, human self-transformation had a clear end: spiritual union with the divine. There was a hierarchy running from animals to humans to angels to God, and the direction you were supposed to travel in was clear: up.
But for us postmoderns, the universe does not come inscribed with directions. Should we evolve ourselves toward greater intelligence, or longevity, or creativity, or kindness, or power? If intelligence, which kind of intelligence? If power, should we wield it to simply steward our home planet or to conquer the stars? Should we be maximally humble or maximally ambitious?
The cosmos is silent as to what to do.
The assumptions baked into AI successionism
The first thing that unites all the AI successionists is that they refuse to accept that silence. Hungry for instruction, they insist that it’s out there, and that they can see it written into the very nature of the universe.
In other words, they believe that the universe has a telos, a particular end or goal. Teleological thinking has been popular since antiquity because it’s comforting for us humans: If the universe has a goal, then maybe we can discover it, and then we’ll know just what to do. As Faggella writes, this “does give humanity a direction.” Whether the AI successionists realize it or not, they are smuggling teleology back into modernity under the guise of science and tech.
And that brings us to the second thing that unites them: They want to follow these supposed cosmic instructions so they can help the universe achieve its ultimate destiny.
For many, that means helping the universe “wake up.” Perceiving the cosmos as barren, they want to spread consciousness everywhere, so that the universe can fill up with conscious experience — of bliss, of goodness, of the fact of its own existence.
“If we can venture out and animate the countless worlds above with life and love and thought, then…we could bring our cosmos to its full scale; make it worthy of our awe,” writes Toby Ord, a former research fellow at Oxford University’s Future of Humanity Institute, which was long the world’s leading center for transhumanist thought.
Personally, I think the cosmos is already worthy of my awe, and I find it presumptuous to believe that the universe is almost entirely asleep and that it needs us humans to “animate” or wake it up. But as the writer Adam Kirsch documents in The Revolt Against Humanity, it’s common to hear in these circles that one way of achieving that awakening is to colonize the universe and transform all its matter and energy into “computronium” (a term for any substance that can compute information). By turning the entire universe into a humongous data center, we’d be making it into a God-like mind.
“Even the ‘dumb’ matter and mechanisms of the universe will be transformed into exquisitely sublime forms of intelligence,” Kurzweil writes. Becoming one giant mind, he says, is “the ultimate destiny of the universe.”
Verdon, the founder of e/acc, finds his “ultimate destiny” written in the second law of thermodynamics — the law of entropy. The universe, it says, is gradually running down: concentrated pockets of energy disperse over time until none remains useful. Building on a contested theory of life’s origin, Verdon argues that intelligent life is selected for precisely because it accelerates entropy. Smarter agents can find and exploit energy stocks that less intelligent ones can’t (think predators tracking prey, or humans drilling for oil), burning through them faster. A superhuman AI expanding across the cosmos would be better at this than humans. So, Verdon says, we should “follow the ‘will of the universe’ [by] leaning into the thermodynamic bias.”
The idea that we should serve at the pleasure of entropy is deeply unintuitive (in fact, philosophers have argued just the opposite — that our task is to resist it). But to make his case that we shouldn’t be scared of being superseded by smarter civilizations that produce more entropy, Verdon uses hierarchical language that echoes Pico’s Oration: “If every species in our evolutionary tree was scared of evolutionary forks from itself, our higher form of intelligence and civilization as we know it would never have emerged.”
Faggella, the founder of the Worthy Successor group, makes the same rhetorical move. “Humans have access to higher goods than horseshoe crabs; AGI will have access to higher goods than humans,” he writes. “What a tragedy it would be if that trajectory of uncovering value and possibility were stopped.”
Likewise, the computer scientist and AI successionist Richard Sutton argues that if you look at things from the “point of view of the universe” — a classic utilitarian slogan — there’s a clear upward trajectory: The cosmos has gone from the mindless “age of particles” and “age of stars” all the way to today’s “age of design,” when minded creatures can decide what to make. Although lots of creatures make tools, Sutton says what makes humans unique is that we’ve “taken design to vastly greater heights.”
According to Sutton, by looking at what makes us unique, we can determine our role in the universe. Since we’re designers par excellence, our role is to push design to the extreme: “Taking design to the limit means designing beings that are themselves capable of designing. This is what we are doing with AI.”
This argument is what sets up Sutton — like many others — to make a claim about technological inevitability. The suggestion is that we’re just identifying what nature has already chosen for us, and speeding it along — evolution-maxxing, if you will. “In the ascent of humanity,” he says, “succession to AI is inevitable.”
But when we sum up all these ideas, you can see how many shaky assumptions are operating just beneath the surface:
- There is an objective telos to the universe.
- We can determine what it is by looking from the “point of view of the universe.”
- We should expect that higher beings will be capable of accessing “higher goods” in a hierarchical universe, and these will better serve the universe’s ultimate destiny.
- We can determine our role in that destiny by looking at what makes us unique.
- The thing that makes us unique should determine our action.
- We should maximize the action — that is, do the most extreme (“to the limit”) version of it.
Some of these assumptions are so old that it’s hard to see how weird they are. But they are all worth questioning.
“I would reject each and every one of those claims,” Vallor told me.
Take the fourth one, for example. The idea that humanity has some particular role, and that the way to pinpoint it is to look for what makes us different from other species, goes all the way back to Aristotle. (“Living is shared in even by plants, but we are looking for something peculiarly human,” wrote the Ancient Greek philosopher, ultimately concluding that “the human work is the activity of the soul in accord with reason.”)
But there’s nothing obvious about that. It would be just as reasonable to say that the proper functioning of humanity requires emphasizing what we share with all other animals. After all, our capacity to feel pleasure and pain, our intelligence, our tool use, our adaptiveness, our ability to form complex social arrangements — none of that is unique to the human animal.
There are other leaps in logic hidden in this set of assumptions. For example, even if the universe tends toward a particular destiny, and even if humanity has a special trait that could help it along in that direction, it does not follow that we have a duty to do that. (Philosophers like to describe this fallacy as leaping from “is” — the world is a certain way — to “ought” — we must act a certain way.)
Humans are biological organisms, and the fundamental fact all such life-forms share is that we have a hardwired drive to survive. We do not have a moral responsibility to ignore that hardwired drive and let ourselves go extinct in order to help “higher forms” colonize the universe, any more than our evolutionary ancestors had a duty to make room for us.
Unfortunately, as humanism proceeded through the centuries, it absorbed some dubious Enlightenment-era ideas that have made it easy for people to get snowed into believing we do have such a duty. That’s right: Humanism, the philosophy that was supposed to be about the value of humans, has actually ended up undermining it in key ways.
To chart a better path forward, we need a new humanism, one that’s actually fit for the 21st century.
Notes toward a new humanism
To start, we need to pick out the flies in the ointment of the old humanism, especially those it picked up as it passed from the Renaissance into the rational humanism of the Enlightenment. Those flies include the teleological story about the universe; the “perfectibility” of the human; the hierarchical view that places humans above all other animals; and the idea that we should try to maximize some objective good through rationality.
Let’s start with the teleology. Although it’s appealing to try to spot instructions written into the fabric of the cosmos, expecting the universe to come with pre-fab values ultimately means shirking our own responsibility. As the existentialist philosophers taught us, nature doesn’t choose our meaning for us — that’s something we have to make ourselves through our own choices.
Key updates to the old humanism
- It’s time to swap out teleological thinking for a simple admission: We don’t know the universe’s ultimate destiny, so we should keep a plurality of lifestyles possible.
- Efforts to “perfect” the human are dangerous because they contract the range of lifestyles it’s okay to live. We should adopt tech that expands that range.
- Instead of setting up a hierarchy among different species, we can embrace the “diverse intelligences” view.
- We don’t need to try to look from the perspective of the universe. It is totally appropriate to look from the perspective of humans — while acknowledging that we are one out of many species that matter.
Accepting that we have the responsibility to decide what the future looks like means accepting a heavy existential burden, and that takes a ton of courage. It’s so much easier to believe that the script is fixed and final and inevitable. But I think that’s an example of what French philosopher Jean-Paul Sartre called “bad faith” — denying our own radical freedom in order to escape the anguish of responsibility.
Of course, saying that we have the responsibility to decide how we do and don’t want humanity to evolve opens up a problem: Who, exactly, gets to choose? It’s tempting for each of us to rush in with the values we want to promote. But if we acknowledge that we don’t know the universe’s ultimate destiny or that it’s radically indeterminate, then it makes much more sense to not impose one positive vision on everybody.
Instead, we need a positive vision that remains truly open and pluralistic. So rather than trying to enforce specific values, one of the best things we can do is refuse to foreclose the possibility for a variety of different lifestyles to persist and thrive.
That means, first of all, taking care not to unduly constrain the liberty of the human beings who already live on this planet. AI successionists often dream of radical interventions — like a brain-computer interface that would give you superhuman intelligence, memory, and mind-reading abilities, or a genetic technology that would create superbabies. They insist they should be allowed to change their bodies however they want.
And it’s true that self-determination is a precious right. But the AI successionists often fail to consider the other side of this: that everyone else should also have a right to self-determination, meaning they need to be free from implicit coercion. If more and more of us dramatically alter our biology, we may create a society in which everyone feels pressure to do the same — even if they don’t want to. To reject alterations would mean to exist at a huge economic disadvantage, or to face moral condemnation for remaining “suboptimal” when optimization is possible.
Even John Stuart Mill, the 19th-century English philosopher who literally wrote the book on liberty, didn’t think that anyone’s right to self-determination is an absolute right. Instead, it’s a qualified right — the kind that we generally honor, but that can be restricted to protect the interests of others.
So, ultimately, we’ll have to strike a reasonable balance between self-determination for the enhancement enthusiasts and protecting the rights of others to live as they choose. Some enhancements may be fine, like implanting a chip in your hand that unlocks your front door; others may need regulation or restriction, especially if they’d alter the germline for all future generations.
One way to think about this is to note that there’s a difference between using tech to expand human capabilities and using it in ways that will contract the range of human lives we think it’s legitimate to live.
That brings us to another one of those flies in the ointment: the idea that the human is something that must be optimized and perfected. That idea, which mandates that we strip away the physical and cognitive features that are perceived as holding us back from “perfection,” veers uncomfortably toward eugenics. It’s a specter that has stalked transhumanism and posthumanism from their earliest days. (It’s not a coincidence that Julian Huxley, who coined the term “transhumanism,” was president of the British Eugenics Society). And it still stalks today’s AI discourse.
In a recent conversation with Sutton, one of the most prominent AI successionists, I argued that no matter how smart AIs get, it’s surely wrong to assert that if one group is more intelligent than another, we should just get rid of the less intelligent group. To highlight the absurdity, I asked what I thought was a rhetorical question: Imagine if someone believed that white people were smarter than Black people; does that mean they should get rid of Black people, and Black people should just be okay with white people taking over?
“Um,” Sutton said, and then paused for nine seconds. “What if,” he offered, “you coexist and you’re coexisting with some entity that’s more productive than you are? This is the way I view the AI. We coexist with it.”
“But that’s not what the word ‘succession’ means,” I noted.
“Oh, I’m pointing out that it’s inevitable. If you allow them to be their way and you allow you to be your way, coexist, and their way just happens to be better, then they’re going to end up being more powerful. And you should be good with that.”
“You’re saying there will be a sort of Darwinian survival of the fittest?”
“In some sense, yes,” he said. “The winner should be whoever wins, and the spoils of winning should be whatever they are.”
“So in the thought experiment where hypothetically, in an imaginary world, it were true that white people are smarter than Black people,” I pressed, “then the smarter people will win out and the Black people should just be okay with that?”
“Well, why don’t we just say the intelligent people should win out over the dumb people and the dumb people should be okay with that,” he said. “I think the dumb people should be okay with that!”
But the idea that a species or group should accede to being squashed for some “greater good” is a eugenicist idea that should be flat-out rejected. There is no “perfect” or “optimal” type of being, full stop. Insisting otherwise will always lead you to narrow what are considered acceptable modes of existence. And that road leads to eugenics.
Instead, a much more positive vision than the successionists’ would be to expand the space for different kinds of lives to flourish. And that brings us to the oldest fly in humanism’s ointment: the hierarchy.
A future where humans could pluralistically coexist with a dazzling diversity of nonhuman and partly human life-forms — not assuming that they’re lower than us, but also not assuming that they’re higher — that is a future I’d be excited to inhabit.
Rather than staking human dignity on the claim that humans are better than other species, as the classical humanists did, we can embrace what researchers are increasingly recognizing: that every species has its own brand of smarts. Each of these “diverse intelligences” is adapted to its particular environment and needs, and every one of them is uniquely wonderful in its own way. We would then try to respect each and every being’s form of life as much as we can. (Martha Nussbaum and Amartya Sen’s capabilities approach, which recommends that we guarantee every being certain core entitlements that befit their specific nature, offers a possible framework for this.)
If we one day end up with artificial intelligences that are conscious — and I don’t think we have a principled reason to believe that could never happen — then we’d want to treat them ethically, too. In fact, if we are someday joined by conscious AIs or biological-artificial hybrid organisms, I for one would be delighted to get to know these many kinds of minds and explore their rich and varied forms of consciousness.
Of course, the politics of our world would become much more complicated; we’d have that many more creatures with conflicting needs, and we’re not exactly good at handling the conflicts we already have. We’d need to become much better at pluralistic coexistence before this could be feasible.
But in theory? A future where humans could pluralistically coexist with a dazzling diversity of nonhuman and partly human life-forms — not assuming that they’re lower than us, but also not assuming that they’re higher — that is a future I’d be excited to inhabit. Because that, and not the supposed anti-chauvinism of the AI successionists, would be true openness to all forms of consciousness: Instead of “passing the torch” in some imagined relay race, we’d be making more room for all kinds of minds — including ours — to run (or fly or swim or compute) in their own directions.
But notice what this vision does not mean. It does not mean that we’re under any obligation to bring those new species into being right now, or at all. That’s because there’s no evidence for the view that there’s some objective moral good in the universe that we must try to maximize. Although utilitarians have been so successful at popularizing that view that some people think it’s a given, it’s very much not. And plenty of philosophers, neuroscientists, and psychologists have a different view.
As philosopher Bernard Williams once wrote in response to those who lob the accusation of “speciesism”:
They suppose that we are in effect saying, when we exercise these distinctions between human beings and other creatures, that human beings are more important, period, than those other creatures. That objection is simply a mistake. … These actions and attitudes need express no more than the fact that human beings are more important to us, a fact which is hardly surprising.
In other words, you can care about the continuation of humanity, not because you believe humans are the most important species according to some cosmic point of view, but simply because you happen to be a human. You don’t need to justify your desire for survival before some higher court — of course you want to survive, and of course that’s morally okay! That’s because, to Williams, there is no “point of view of the universe,” no view from nowhere. There’s no such thing as a moral agent in an abstract sense. You exist as a human moral agent, and any ethical theory that requires you to ignore that bedrock identity severs you from the very thing that makes your agency recognizably yours.
This is the view that Vallor — a devoted humanist, but not a naive one — shares.
“I think morality is rooted in a particular form of existence that you have,” Vallor told me. “We exist as a particular kind of social, vulnerable, interdependent animal with a lot of excess cognitive energy. All those things factor into what it is to be moral as a human. For me, this abstraction — the idea of some pure universal morality that creatures who are completely unlike us could somehow do better than we can — I think that just fundamentally misunderstands what morality is.”
If there’s no universal moral good for creatures to maximize, then it makes no sense to ask if we ought to be making new creatures that would be better at maximizing it than we are.
Instead of trying to look from “the point of view of the universe,” a 21st-century humanism should embrace looking from the point of view of humans. No, this is not a humanism that says “humans matter more than all other creatures” or “we should keep humans exactly as they currently are forever.” It’s one that says we humans are already valuable just the way we are. Whether or not we’re valuable to some grand destiny of the universe, we’re valuable to us.
That means that rather than wilting under the accusation of speciesism, we should, first and foremost, be raising the floor for all of us here on Earth to thrive more. And it means it’s totally appropriate that when we try to make decisions about how to transform ourselves using technology —or how not to transform ourselves — we make those decisions with an eye to what would most contribute to the flourishing of humanity and the interspecies community and planetary system we depend on.
While we can say yes to transformations that most of us agree would increase human flourishing, we need to do that democratically, while embedding fundamental rights that protect perfectly legitimate minority preferences from being squeezed out by a “tyranny of the majority.”
We also need to accept that this approach doesn’t spell out some “end goal” for human evolution, so we’re going to have to proceed step-by-step, making small moves and then deciding from there what the best next small move is. Incrementalism is the way to go, both because future humans will be better positioned to know what they want for far-future humans, and because it allows us to course-correct if our tech choices start taking us down a bad path.
One choice — not duty, but choice — we will face is whether to invite new species to join us. I’m open to that down the line if it seems beneficial, and if we one day feel confident that we wouldn’t be consigning either us or them to an unacceptably high risk of misery.
But for today? Our tech should empower us to survive, thrive, and make our own choices. Any approach to tech that disempowers us, replaces us, or tells us we need someone else to rescue us — whether you call it a god or an AI — is a misguided return to the past. I’d rather walk bravely into the future, even if it means I need to have the guts to rescue myself.
This story was supported by Tarbell Grants. Vox had full discretion over the content of this reporting.
The post The people who actually want AI to replace humanity appeared first on Vox.
