Against Moloch #49

Let’s talk about Anthropic. They’re clearly the most careful and ethical of the labs, but are they careful and ethical enough? Zvi explores the two sides of Anthropic, and Kelsey Piper shares her reservations about their worldview. Oh, and there’s that $2 trillion IPO. We should talk about that, too. Top pick What I saw inside Anthropic Kelsey Piper leaves a briefing at Anthropic headquarters feeling that Anthropic has hopes, but not plans : But if Anthropic is correct, if we now have AIs that are smart enough to considerably improve the process of training future AIs, and if things can be expected to speed up rather than slow down from here, then it seems very unlikely to me that this ends well. Anthropic’s race toward RSI is driven in part by a belief that the US has to reach superintelligence before China, to which Piper raises the obvious objection: China doesn’t want to destroy the world either, and China is largely training from American models: If we slow down, it might too. And anyway, it’s not a good enough reason: You can’t defensibly do something that risks human extinction just because otherwise someone else might. Anthropic’s plan isn’t great, but it’s hard to find a b