Argue about Risk Management Speed, not p(doom)

This post is written in a personal capacity. All opinions expressed are solely those of the author. I. Executive Summary There’s a mismatch that I’d like to correct between: the reasons why superintelligence is dangerous, and the categories we use to describe how dangerous superintelligence is. The most important cluster of reasons why a superintelligent model is dangerous is that it (i) could quickly kill or impoverish billions of people, (ii) could be very different from its predecessors, and (iii) can’t easily be altered once it’s deployed or otherwise widely distributed. The combination of these three factors is worse than the sum of its parts: the only way to protect ourselves against a powerful, novel, incorrigible AI model is to accurately anticipate what dangers it might cause and prevent those dangers before they come to pass. Unfortunately, we don’t yet have a good vocabulary for talking about whether AI companies can identify and mitigate dangers faster than those dangers arise. The by-now cliched way to describe the danger of superintelligence is p(doom), i.e., a static point estimate of how likely AI is to end human civilization or otherwise cause catastrophe . More so