AI Extinction Risk: What People Mean by p(doom) — and Why the Number Is Not a Fact

Start with the chain of events

Start with the chain of events, not the movie ending. A powerful model still needs a route from capability to access, persistence and real-world effect before a takeover claim becomes concrete.

Experts disagree widely about both probability and plausible mechanisms. Keep that fact separate from the extra assumptions a takeover scenario still needs, especially access, persistence and the ability to resist intervention.

What matters in practice

Experts disagree widely about both probability and plausible mechanisms.

The practical gap is usually between capability and authority. A system may be able to describe or plan an action without having the credentials, persistence or route into the systems needed to carry it out.

Probability estimates depend on assumptions about future capabilities and governance.

Headlines often compress planning, access and successful execution into one dramatic step. Treating those as separate questions makes it much easier to see what has actually been demonstrated and what is still assumed.

Low-probability high-impact risks can still justify safeguards.

Recovery matters as much as capability. If people can revoke access, isolate a service or restore affected systems, the scenario is very different from one in which control cannot be regained.

Where the headline usually skips a step

Takeover claims are especially sensitive to missing links in the chain. A dramatic capability result is not enough on its own; the scenario also needs credible access, persistence and a path to consequences outside the test environment.

• A survey median is not a measured physical probability.
• Changing the definition of “doom” can change the answer dramatically.

What could change the answer?

The assessment should change if evidence appears that closes one of the missing operational gaps: persistent autonomous access, successful recovery evasion, or demonstrated control over consequential systems. Until then, keep those steps separate rather than assuming them.

What you can do now

Focus on the parts humans can still control: permissions, approval thresholds, monitoring and recovery. Those measures are useful whether the extreme scenario is likely, unlikely or simply too uncertain to quantify.

1. Ask what scenario a percentage refers to.
2. Inspect time horizon and assumptions before comparing estimates.
3. Use scenario planning without pretending precision.

A four-question reality check

When a claim sounds dramatic, work through these four questions before deciding how much weight to give it.

Bottom line

The useful question is not whether an extreme scenario can be imagined. It is which links in the chain already exist, which are speculative and which safeguards can still break the chain.

“p(doom)” is a personal or modelled probability assigned to catastrophic AI outcomes. It can be useful for exposing assumptions, but there is no scientifically settled percentage for human extinction from AI.