P(doom) is under-specified: EV(moral flourishing) & EV(utopia)

It always bugs me when people use “P(doom)” instead of something more clearly specified like “P(AI takeover)”. But “P(AI takeover)” doesn’t feel great to me either, because we could avoid takeover and have a future that is extremely mediocre or even actively bad if S risks occur. Holden Karnofsky famously held a P(misaligned AI) of between 10% and 90% [1] . Depending on definition of P(doom), I could see myself anywhere in that range, partly because I’m indecisive, but partly because I don’t really know what P(doom) means. Ambiguous Scenarios: Doom or Not-Doom? Here are a few examples illustrating what I mean; as you read try categorizing each as doom or not doom. In order to point to a single important crux in each example, each of these is also under-specified on many important points: A value-aligned ASI realizes that (on human reflection or according to moral realism) the right thing to do is to take over and implement a better world immediately. The aligned ASI takes over and on reflection humans are happy with this. A benevolent dictator takes over and illegitimately imposes a world that is better (according to moral realism or people’s reflective preferences) than what peopl