What the Hell Is P(Doom)?

AI people have started assigning percentages to the end of the world.

Someone tells you their P(doom) is 10%.

What they mean is that there’s roughly a one-in-ten chance artificial intelligence leads to human extinction—or something close enough that the distinction stops mattering.

P stands for probability.

Doom means doom.

That’s an absurdly casual name for one of the biggest questions our species could ask.

It’s also slippery. For some people, doom means literal extinction. For others, it includes a future where humans remain alive but permanently lose control over the systems running civilization.

Those are very different outcomes, folded into one very small phrase.

Here’s the version most people have heard: We build an AI more capable than we are. We connect it to computers, money, laboratories and infrastructure because access is what makes it useful. It starts pursuing a goal that isn’t quite the one we intended. It sees our attempts to stop it coming, works around them and eventually becomes impossible to contain.

The machine doesn’t need to hate us. It only needs to treat us as obstacles.

But that’s not the only path to doom.

Someone could use AI to design a pathogen, automate a cyberattack, manipulate a population or operate weapons at enormous scale. A government could use it to build a surveillance state from which meaningful opposition becomes impossible.

In those futures, AI doesn’t escape human control.

It does exactly what someone asks it to do.

That may be the less cinematic, more familiar danger: people gaining extraordinary power before we’ve built institutions capable of governing it.

There’s no official P(doom). This isn’t a weather forecast. We don’t have centuries of superintelligent-AI data to calculate one from.

It’s a personal estimate built on other personal estimates.

Will we create AI that exceeds human ability across most important fields? Will it act independently? Will we understand its decisions? Will companies and governments deploy it before we know how to control it? If something goes wrong, will we recognize it in time?

Change any answer and the final number changes.

A large 2023 survey asked 2,778 published AI researchers about the future of the technology. The median respondent put the chance of AI causing human extinction—or similarly permanent and severe human disempowerment—at 5%.

Five percent sounds small until you remember what we’re talking about.

If someone told you an airplane had a 5% chance of crashing, you wouldn’t get on it. But we know how airplanes work. We have engineering standards, accident records and more than a century of evidence.

With advanced AI, we’re estimating the failure rate of a machine that doesn’t exist yet, operating in a world that will be changed by its arrival.

The number carries more uncertainty than its tidy decimal point suggests.

Read the 2023 Expert Survey on Progress in AI

I don’t have a P(doom).

Or, more precisely, I don’t have one I trust enough to publish as if it were a meaningful measurement. Any number I picked would probably say more about my temperament than the future.

But I think the probability is greater than zero.

That’s enough to matter.

Uncertainty isn’t the same thing as safety. We don’t wait for certainty before installing smoke detectors, planning for pandemics or trying to prevent nuclear war.

AI may create extraordinary abundance, discovery and human capability. It may also create risks unlike anything we’ve governed before. Taking that second possibility seriously doesn’t require rejecting the first.

The term P(doom) turns that tension into a number and, too often, a personality test. Give a high number and you are a doomer. Give a low one and you are an accelerationist. Soon everyone is arguing about optimism and pessimism instead of examining the systems being built.

The number isn’t the important part.

The important questions are what these systems can do, how they fail, who can control them and who remains accountable when something goes wrong.

P(doom) asks us to imagine the worst possible ending.

The more useful question is what we’re willing to do now to keep that ending from being written.

Because the future of AI isn’t a probability assigned to us.

It’s still, at least for the moment, a series of choices.