A big-tent or small-tent AI safety movement?

In recent weeks, two narratives about AI safety have emerged: either AI existential risk is real and imminent, or AI leaders’ and whistleblowers’ claims to that effect are insincere — a “psyop” or hype or a twisted form of regulatory capture. Few have considered the third possibility that x-risk warnings are sincere but simply wrong and counterproductive to AI safety . It’s as if everyone takes for granted that those raising the alarm are geniuses, and the only question is whether they’re the benevolent or the evil kind of genius. We subscribe to neither narrative. In our writing on AI safety , we’ve consistently tried to take the arguments seriously but rebut the many claims we think are unsound while highlighting safety concerns that we think are real. This essay is a continuation of that agenda. It is true that we systematically underinvest in resilience against catastrophic and systemic risks Let’s start by steelmanning a subset of AI safety and Effective Altruist (EA) arguments. Something is deeply wrong in the world when it comes to how much we invest in defending against catastrophic risks, and the safety and EA communities are right about that. Take superintelligence out of the equation entirely and it remains true that evidence-grounded risk arguments have been made for years and not produced a commensurate policy response. The most well known example is the failure to prepare for pandemics. We had years of warnings and quantitative estimates. A 2019 report convened by the WHO/World Bank — before COVID — began by saying there is a “very real threat ... of a respiratory pathogen killing 50 to 80 million people and wiping out nearly 5% of the world’s economy.” It estimated the cost of adequate preparedness at a mere $1–2 per person per year for most countries. Since the 2009 H1N1 pandemic, at least 11 high-level panels and commissions have made specific recommendations to improve global pandemic preparedness, and yet “ the majority of recommendations were never implemented ”. More remarkably, even after COVID, only a fraction of the need was funded. The same independent panel wrote in 2024 that “There is a cynical attraction to providing significant and immediate resources during a crisis, rather than proactively preventing one. Yet history shows this will not end well.” The root cause seems to be that voters reward spending on relief, not preparedness . To be clear, the inadequacy of pandemic preparedness is by no means a uniquely Effective Altruist insight, let alone AI safety. But EA does deserve credit for championing the cause. It’s also true that EA’s consequentialist moral framework that centers expected-utility calculations is highly effective at revealing and quantifying our underpreparedness. The framework is highly controversial, but we don’t think there’s anything wrong with it or its application to areas like biorisk. We do, however, strongly disagree with its extensions to longtermism and AI existential risk. We also disagree with the tendency to reframe problems such as AI’s amplification of biorisk as “AI safety” problems to be solved through methods such as alignment. Our view is that AI is an amplifier of existing systemic risks. The underpreparedness problem is also true of cybersecurity — actually, it’s worse. Unlike pandemics or finance, there is no culture of modeling systemic or catastrophic risks in the cybersecurity community. The economic approach we described in the previous essay has been successful in keeping the expected losses from everyday attacks at a manageable level, but markets are notoriously bad at appropriately pricing tail risks and externalities. Insurance companies know this. Lloyd’s is pretty blunt about the fact that “losses have the potential to greatly exceed what the insurance market is able to absorb”, and does not insure against severe state-backed cyberattacks (that was in 2022). Once again, this has little to do with AI’s recent capabilities. Five years ago, before AI was in the picture, when teaching undergraduate information security at Princeton, Arvind emphasized that cascading failures and catastrophic risks are possible, resulting in essentially the whole internet going down for an extended period, with potentially civilization-altering consequences. 1 (Looking back at the speaker notes in my slides, I’m reminded that I apologized for presenting original research in an undergrad class that’s supposed to be about basics, but explained that this is because the near-total lack of attention to this critical topic in the security community left me no choice. — AN.) In other words, Dario Amodei’s claim about agent swarms potentially being capable of “ taking over the entire internet ”, while worded somewhat hyperbolically, is not obviously outside the realm of possibility, and deserves study at the very least. The cybersecurity community has the right skillset to do that, but unfortunately they don’t seem to find this even worth thinking about. There are also systemic risks that are diffuse and gradual rather than catastrophic. Consider what chatbots are doing to news traffic — clickthrough rates are far lower than traditional search, accelerating the decline of traditional journalism. The second-order societal effects of this shift won’t be apparent for quite a while. And that’s just one of dozens of types of creeping institutional decay. Atoosa Kasirzadeh introduced the concept of accumulative risk to describe this gradual erosion of societal resilience. 2 Here, the psychology of inaction is the reverse of the case of pandemics: rather than being so rare that we are able to memory-hole the event, decay is so pervasive and gradual that we become inured to it. In short, based entirely on well-grounded rather than speculative causal pathways, it is reasonable to believe that policymakers are drastically underinvesting in protecting society against AI risks, and it is hard not to empathize with those who want to ratchet up the rhetoric. Of course, the question is whether more urgent public warnings actually lead to better policymaking. The Coxon incident shows how the warped media environment elevates x-risk over more grounded concerns This newsletter is largely an exercise in perspective-taking. Let’s take a moment to articulate the primary reason why people inside and outside the AI bubble tend to talk past each other. For people in AI — whether or not they’re concerned about safety — it’s hard to understand how anything else in the world matters right now. As computer scientist Scott Aaronson put it in his essay The Age of Wonders and Terrors : “Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing.” This essay made the rounds in the community because this perspective is widely shared. Even the two of us — best known for pushing back against some elements of this view — consider AI important enough that we decided long ago to devote our careers to it. The AI as Normal Technology position is that AI will “only” be as impactful as the industrial revolution. People outside AI don’t understand the intensity of this view in the AI community — that it’s an AI world and we’re all just living in it. But there’s a flip side — people in AI vastly overestimate how much the public cares about AI. In Gallup polls, the share of people who rate AI (or “advancement of computers/technology”) as the most important issue is about half a percent . Most of those people are likely thinking of relatively mundane aspects such as a datacenter being built in their community. The share of people outside AI who treat it as the defining issue of our time is incomprehensibly small. The reason AI people fail to recognize this is that AI is a topic where salience and concern are decoupled from