Pangloss sometimes said to Candide: “There is a concentration of events in this best of all possible words: for if you had not been kicked out of the magnificent castle for love of Miss Cunegonde: if you had not been put into the Inquisition: if you had not walked over America: if you had not stabbed the Baron: if you had not lost all your sheep from the fine country of El Dorado: you would not be here eating preserved citrons and pistachio-nuts.”
Voltaire, Candide: or, The Optimist1
If you have the opportunity to build some moonshot technology that might lead to immense human flourishing, but also has some nontrivial chance of wiping out humanity,2 should you take it? There are some who think that this is the defining question of our times.
Now most people would probably say something like: “Are you kidding? Of course not; wiping out humanity is a really bad outcome.” Even if the technology is positive-expectancy for humanity, it’s quite possible that the risk is not worth the reward.3 The possibility of extinction would have to be really, really remote—and the payoff of success really, really high—for the gamble to be even arguably worth it. And of course, in the real world, it is practically impossible to get any reasonable estimate for the probability of extinction associated with any particular technology. After all, by virtue of still being alive, we have never actually experienced any human extinction events yet, so how would we ever get enough data to draw a reliable conclusion?
But I want to propose a somewhat radical thought experiment here: under a set of strong but not totally insane assumptions, the gamble could actually prove worthwhile, at least in the spherical cow setting. For the sake of the argument, let’s make the assumption that there are only two possible outcomes to this technology: either it will cause immense flourishing, or it will cause human extinction.4 Moreover, the good outcome is so good (say, leads to some super-exponential increase in total utility), such that even one “successful” gamble will make up for a huge number of “failed” ones. The argument is then composed of two key ingredients:
A fair summary of the argument would be that while it might not make sense for the inhabitants of any one universe to take the gamble, there is a sense in which, if you are optimizing for human welfare across the entire multiverse, perhaps you should want each universe to take the gamble. To start off with, I feel like I should give a disclaimer that I’m not sure that I actually believe these ideas, but the point of a personal blog is that I can post wild thoughts, so I thought it was worth developing the argument further. What follows is certainly a caricature of reality,5 but hopefully it is at least directionally interesting.
The first key claim here is that if you’re able to ensure multiple independent trials of developing the technology, the acceptable level of risk per experiment should be higher. For instance, suppose humanity spreads out to ten different galaxies in the future, and then somehow lose communication between the galaxies. This vastly decreases the probability of extinction, and therefore vastly increases the failure probability that we can tolerate in any given experiment, at least from the point of view of someone interested in maximizing overall human utility (recall the assumption that the technology is so good that any one success would make up for a multitude of failures). Now, you might fairly ask what good successes in other galaxies do for me; if I live in the Milky Way and all of us get wiped out, why should I derive any comfort from the fact that there are quadrillions of happy humans in Andromeda? But more on that later.
This is analogous to investing in a diversified portfolio in order to manage your risk; I was reminded of this one day as I was reading Matt Levine on the “standard theory” of executive compensation. Essentially, his claim is that for your standard-issue public company, there is a natural mismatch between the objective functions of the shareholders and the CEO (let’s call her Alice): the shareholders are probably diversified across many such companies (these days, probably through index fund investments), whereas Alice has her livelihood tied to one specific company. As such, the shareholders would probably prefer that Alice take more risks, even though there is a non-negligible chance that these risks won’t pay off. Across a diversified portfolio of companies, it’s likely that there will be at least some wins, which will more than compensate for the losses. However, Alice probably doesn’t want to take that many risks, because her job and reputation are staked to this specific company! The classic solution to this mismatch is to incentivize Alice to take more risks by granting her a large bucket of stock options, which will pay off handsomely if her company meets some ambitious target.6
Again, it is arguable that in the case of this hypothetical risky technology, we are more akin to Alice, in that our lives are tied to a particular “company”; if the technology goes poorly, we will not be around for another trial. But the point of this section is just to show that, if we were somehow to put ourselves in the shoes of the diversified shareholders, we would likely prefer each “company” take some more risks, all things considered.
Okay, so perhaps you buy—or at least are willing to concede for the sake of the argument—the premise that if we actually had a way to conduct multiple independent trials of this magical technology, it might be a good idea to try to develop it. But how could we ever manage to conduct independent trials when the risk we are trying to mitigate is the extinction of humanity?
Here is where the second arm of my diabolical modus ponens operates. Another thing that people sometimes talk about is the possibility of a multiverse: the idea that there might be many non-interacting universes. For instance, some people hold to the many-worlds interpretation of quantum mechanics, which seems to imply such a thing.7 Other people believe in simulation theory, which says that we are likely living in a computer simulation. The argument goes like this: we might expect that an advanced civilization would be interested in running computer simulations of sentient beings (say, for research), and that it would be relatively cheap for them to do so. This means that across all of time, the vast majority of sentient beings with experiences like ours will actually live in simulations. Absent strong priors for believing that we are not simulated, we should therefore believe that we are most likely living in a simulation.8
Regardless of the mechanism by which you achieve a multiverse, you are then confronted with a weird proposition: should the existence of other universes actually affect the way you live? We have posited that these universes are non-interacting, but in some sense, the very definition of non-interaction is that nothing in your life would change if the other universes didn’t exist. Why should a rational being take the multiverse into consideration when making decisions?
I think the answer is that a rational being would not, but perhaps a super-rational being would, if motivated by a particular kind of altruism. Now obviously this whole business of optimizing for overall human welfare is inherently altruistic; a selfish actor would only consider his or her own welfare, not that of humanity. But I claim that the kind of altruism needed to accept my multiverse argument is especially strong, for two reasons.
The first reason is that it calls on us to consider the welfare of people whose actions can never affect us—people who, for all (rational) intents and purposes, might as well not exist. That is to say that if someone lives in a hermetically sealed-off universe, then to us that person might as well be a fictional character like Bilbo Baggins or Harry Potter. This is really weird; we typically don’t take the welfare of Bilbo Baggins into consideration when determining what actions to take, for instance. I can envision a plausible argument from evolution as to why we might have some limited innate altruism toward people in our universe; you could imagine that communities of altruistic people might be more successful than entirely selfish communities, and therefore that evolution would select for people who have at least some degree of altruism.9 But under this theory, you would expect no evolutionary pressure toward altruism directed at people who don’t actually exist in this universe; such a thing is at best a silly aberration that has not (yet) been optimized out by evolution. In this sense, inter-universal altruism is the purest form of altruism, because it targets people who can truly never pay you back—because they do not exist!
The second reason for finding this altruism across the multiverse very surprising is that for it to be effective, you need to assume that people in other universes also possess this sort of “super-altruism.” Otherwise, your noble self-sacrifice in undertaking this risky technological experiment is pointless; you need the inhabitants of the other universes to also do this noble self-sacrifice, so that perhaps one universe might succeed and flourish immensely. But we have just established that it is not rational for them to do so, so what grounds do we have for supposing that they will? I think the answer is that we have to invoke the idea of super-rationality; the main idea is that a super-rational agent will assume that everyone else is also a super-rational agent, and then act accordingly.10 In this case, if we assume that people in other universes are also super-rational, with the goal of optimizing welfare across the entire multiverse, we can be assured that they will come up with the same strategy as we do. In this case, doesn’t it seem reasonable for us to maybe take the gamble?
If we do take the gamble, how optimistic should you be that it will pay off in our universe? Maybe you should be quite pessimistic: after all, the entire premise was that there is an uncomfortably high chance that things will not work out in our universe, and humanity will go extinct (oops!).
But here’s a slightly curious thing: in the universe(s) in which the gamble does pay off, humanity will thrive, presumably lasting a long time—maybe even forever!—thereby accumulating a massive cumulative population, at least when integrated across all of time. It is not unreasonable to expect that the modal human lives in such a “successful” universe, and in the extreme case, we might even suppose that a simple majority of humans will live in successful universes. For those of us living in the early stages of human civilization, before the invention of this hypothetical world-changing technology, this is perhaps little comfort; conditioned on such an existence, we face grim odds. But if you sample across all of humanity, there may be reason for optimism, because there is higher probability mass associated with the successful universes. On average, the Leibnizian optimist is more right than one might naively imagine, if you assume that people are concentrated in the best of all possible worlds.11
And so maybe the answer to “why should I care if there are a quadrillion happy humans in Andromeda” is that expectation, you live in Andromeda! Perhaps it realized poorly and you ended up in the Milky Way, but complaining about that is just sour grapes, or something.
This translation is taken from the 1918 Modern Library edition published by Boni and Liveright. As far as I can tell, it is based on the 1901 translation of Voltaire’s complete works by William Fleming.↩︎
For the sake of example, let’s suppose you want to develop a technology called the Amazing Generator of Ideas (AGI), powered by Lots of Lame Metaphors (LLMs), which solves the problem of bloggers not being able to come up with good ideas for their blog posts and needing to fall back to writing about trendy technologies. While AGI sounds quite promising, it has a terrible potential downside: maybe everyone will stop doing productive economic activity and become a blogger instead, precipitating a worldwide famine and widespread political instability.↩︎
This is not to say that we can actually prevent anyone from building the technology, of course. For instance, nuclear weapons have some chance at wiping out civilization, if not humanity, but no nation can give up its nuclear program, because then it would be at the mercy of the remaining nuclear nations.↩︎
Notably, many believe that this is not true of AI; for example, Tyler Cowen has argued that AI is quite impressive, but this impressiveness will likely only translate to a modest increase to GDP growth. I think I originally heard of this prediction when he came to my firm to give a talk, but see his blog post on the matter for more details.↩︎
Though notably, there are certain latter-day Jeremiahs who believe it is not far from the truth!↩︎
This is not the only solution to this problem. Another obvious solution is for Alice to hedge her position by buying puts on the company, but of course it seems kind of suspect for the executive of a company to bet on its failure. It’s also worth pointing out that there are lots of downsides to such moonshot companies; to mention a couple that some coworkers flagged in response to that Money Stuff email, for example, it’s probably a lot less pleasant for all the employees involved, and it potentially creates zero-sum (or even negative-sum) competitive dynamics between firms.↩︎
Or so I’m told; I haven’t taken a real physics class since high school.↩︎
This is based on the classical form of the argument given by Nick Bostrom in his 2003 paper “Are We Living in a Computer Simulation?” in The Philosophical Quarterly, volume 53, issue 211.↩︎
I am not suggesting that this is the only reasonable way to view altruism, and in fact I would even go as far as to say that any view of altruism that does not include some spiritual dimension is an impoverished one. But for the purposes of this blog post, I try to stick to only rational arguments.↩︎
The classic example here is the prisoner’s dilemma: under standard analysis, defection is the dominant strategy, and so both players will choose to defect and thereby achieve a suboptimal outcome compared to what would have happened had they cooperated. One solution to this mis-coordination is to suppose that the game happens many times; in the iterated prisoner’s dilemma, it is possible for rational players to cooperate. Unfortunately, this doesn’t really work for us, because you only get one shot at developing this technology; recall that it causes either spectacular flourishing or human extinction.↩︎
I will freely admit that this kind of reasoning is fairly suspect. For a similar, entertaining argument, consider the famous “shooting room paradox,” in which people are concentrated in the worst of all possible worlds: say you wake up in a room operated by an evil villain. You know that the villain operates in a very particular way; for every batch of people in the room, he will flip a weighted coin. With probability 99%, the coin comes up heads, and he lets everyone in the room go but then finds people to put in the room for the next round. He keeps on doing this until the coin lands on tails once (again, with probability 1% each round), at which point he kills everyone in the room and then goes to retire on a beach somewhere. If you wake up in the room one day, how scared should you be for your life? On the one hand, you know that the coin only has probability 1% of landing tails, and so perhaps you shouldn’t be that scared, all things considered. But on the other hand, since the number of people in the room doubles each round, fully half of the people who ever end up in that room will die! Each person is individually justified in being unworried, but collectively, half of them will be wrong. This problem has been much discussed in the literature; I learned of it from Scott Aaronson’s book Quantum Computing since Democritus, where is is called the dice room and is attributed to the philosopher John Leslie.↩︎
Comments