Freedom of Action: America's ASI Strategy, w/ RAND's Joel Predd, Ben Boudreaux & Matt Chessen

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution! Today my guests are Joel Predd – Senior Engineer & Director; Ben Boudreaux – Policy Researcher & Professor of Public Policy, and Matt Chessen – Resident Technical Expert, at the Center for the Geopolitics of AGI at the RAND Corporation. Together with co-authors, they've recently written a new RAND Perspective paper called "A U.S. Strategy to Secure Geopolitical Advantage on an Uncertain Path to Superintelligence: Maintaining Freedom of Action". This is one of the clearest pieces of AI strategy writing I've encountered – it's both a great introduction and orientation tool for people who are just now entering the AI strategy discourse, and a useful organizing framework for anyone who's been frustrated by how many AI strategy conversations consist of smart people talking past one another due to differing, often unstated assumptions about critical questions. Their first contribution is a taxonomy of 7 Archetypal Strategies that aim to achieve distinct, coherent win conditions, and the 5 Pivotal Uncertainties that will need to be resolved before the best strategy can be identified. The 7 Strategies, which will be familiar to Cognitive Revolution listeners, include: - Acceleration, aka the a16z Plan of letting technology take its natural course - Dominance – aka the Dario plan where the US builds such an advantage that it can dictate terms to other countries. - Co-Development – which you might think of as the Demis Plan for international cooperation - Preparedness for Coexistence, which seems to be OpenAI's plan, as far as I can tell - a Moratorium – aka the Pause AI Plan - Deterrence – which is something like Dan Hendrycks, Alex Wang, and Eric Schmidt's Mutual Assured AI Malfunction Plan, - and finally, investments in what they call "Continuity of Society", which might be best described as Prepping for an inevitable AI Doomsday. The 5 Uncertainties are: 1. How close are we to extremely powerful and dangerous AI systems? 2. Is it feasible for humans to coexist with such systems? 3. Is it possible for one actor, which could be a country or perhaps even a private company, to use such systems to achieve a decisive strategic advantage over its competitors? 4. Can humanity come together to cooperatively restrain itself from developing such dangerous systems? 5. And finally… can the United States or a coalition of governments prevent the development of such systems, including by force if necessary? The core argument, as you'll hear, is that because there's no consensus on these critical questions, even as AI capabilities continue to advance at a blistering pace, the United States should invest now in all of the foundational technologies, institutions, and other capabilities that the various strategies would require, such that we can keep our options open long enough to get the answers needed to make a wise high-level strategic choice and commitment. Ultimately, while I'd personally argue for Co-Development by default, with a Moratorium as the fallback, in case it becomes clear that we simply can't continue to develop AIs safely, I feel that this recommendation is a very good one for a national government that is just now beginning to understand the nature of the technology, and remains – for a mix of good and bad reasons – very doubtful that it can trust its rivals to cooperate on such a high-stakes issue. Along the way, we discuss: - how the politics of AGI have shifted in DC over the last couple of years, - how AGI-pilled the government is today, - how realistic a decisive strategic advantage really is, - the similarities and differences between the strategic dynamics of the current AI situation and the Cold War nuclear arms race, - the magnitude of the investment they are recommending, which they are working to quantify in upcoming work on the cost and lead times for 106 distinct capabilities they've listed in a Security Architecture Catalog, - and what we can do, both in our organizations and even just as individuals, to maintain as much agency and control as possible even as we seek maximum leverage from AI in our work. This conversation, and the Perspective Paper itself, stand out for abstracting away the many messy and often confusing details of week-to-week AI developments and focusing attention on the most critical decisions the United States will need to make, as well as the core reasons that it remains premature to commit to a single strategy. So, without further ado, I hope you enjoy this clarifying synthesis of AI discourse and simple, high-level argument that the US government should do everything it can now to maintain its freedom of action on the path to superintelligence, with Joel Predd, Ben Boudreaux, and Matt Chessen of RAND's Center for the Geopolitics of AGI.

Sponsor

[05:21] Lovable : Lovable is the software creation platform that brings your ideas to life with enterprise-grade infrastructure and flexible AI coding agents. Start building your next project today at https://lovable.com

Visit fruitopia.ai to explore more than 400 fruits, and more than 8000 varieties, from all around the world.

Main Episode

[07:05] Nathan Labenz: Joel Pred, senior engineer and director, Ben Boudreaux, policy researcher and professor of public policy, and Matt Chesson, resident technical expert at the Center for the Geopolitics of AGI at the RAND Corporation, welcome all of you to the cognitive revolution.

[07:20] Joel Predd: Great to be here. Thanks, Nathan, for having us.

[07:22] Nathan Labenz: Yeah. I'm excited about this. So I have been looking through what I think is a really clarifying document that you guys have put together. It is called maintaining freedom of a US strategy to secure geopolitical advantage on an uncertain path to superintelligence. And I think we will benefit a lot, and the audience will benefit a a lot by just walking through this. I find it to be unusually well crafted and just kind of easy to absorb. And so I I think just, you know, going through it in a pretty linear way will be well worth our time. And then we can potentially expand to some other topics, and then maybe I'll push you guys on some of the the recommendations that you make as well. Maybe just for starters, a little bit of context before we get into the strategy document. One question that's always kind of in the air a little bit is, like, where are we right now on the path toward governments around the world taking AI as seriously as I think probably all four of us think they should. I've seen a lot of signs that I didn't expect to see this soon, but I also see a lot of indicators where I feel like, maybe they're not, you know, so serious after all. It's a little bit hard for me to read from the outside.

[08:33] Joel Predd: So I'll I'll take a first swing at this. And, Matt, Ben, may have sharper answers in this. But here we go. It is quite remarkable the moment we're in. When when the center for the geopolitics of AGI was launched two and a half or three years ago, it was such that we kind of had to whisper AGI for fear of being thought of as too out there, too consumed by science fiction scenarios. It didn't deter us or scare us from doing the kind of work that we've been doing, which is to think about the implications of frontier AI that could advance to that threshold. But nonetheless, that was the climate. And it is quite remarkable that now you can't get into a meeting without calling it it superintelligence. So times have changed. The other thing that we... One of the things that we always say in this space, mostly with the reference to the technology, is that we should be focused on the trends, not the time stamps. And the trends are quite remarkable as we all have observed over the last two and a half years in the pace of progress in the technology. But I think one of the things you're observing is that the trends are also, you could say, somewhat favorable even if they're way too slow when we think about the governance of this technology. We have governments waking up and paying attention to it. We have the firms who are building the technology, talking about governance solutions. We have the civilians, the poll... Forming our public debate now. So the the the trends are very rapid or even accelerating even in the governance space. Are they fast enough? It's very difficult to keep pace with this rapidly advancing technology. And we have spent a lot of time over the last two and a half or three years looking for the the the folks that were taking it as seriously as necessary. And I regret to say you... To tell you that there aren't nearly enough, as there should be.

[10:18] Ben Boudreaux: I'd just add that I think we are seeing lots of signs that the US government has become more aware of the type of strategic importance that this technology may bring, but it's not, I think, clear that The US has settled on a strategy nor, I think, have the frontier companies. And that might be challenging because many capabilities that we might need, like verification technology or technical expertise in government, you know, evaluation capacity, things like crisis response and communication, those all may take quite a long time to build. And by not proactively building many of those capabilities while capabilities advance and various deployments continue, that itself might be a type of choice that forecloses options we might need down the line. So I I think we're seeing signs of a greater awareness, but not yet signs of truly adapting and trying to proactively get ahead of the type of transformations we are likely to see.

[11:19] Nathan Labenz: Yeah. I think that's a great bit of foreshadowing for the strategy document as a whole because, you know, spoiler, it kind of leads up to this idea of building the capacity to commit to a strategy once we sort of see a couple more cards turned over and have a better sense of what actually is the nature of the technology and the challenges that we're dealing with as a result of its development. And I think fundamentally, we just do still have, like, a lot of uncertainty as to, like, what are the nature... What is the nature of the technology? What are the challenges that are gonna be most important? And, you know, it sort of strikes me that, like, in a way, what you're recommending ultimately is kind of like launch a lot of sub agents in parallel US government so that they can go out and prepare for a lot of different eventualities.

[12:14] Ben Boudreaux: Before

[12:15] Nathan Labenz: we get there though, let's maybe start at the beginning. So the kind of first big thing I took away from the document is, again, a really clear taxonomy of seven archetypal strategies that people are throwing around. I think listeners to this feed will, like, have heard variations of them before, but I hadn't ever quite seen them enumerated, you know, in such a succinct way. So maybe let's just start off with a quick review of, like, what are the archetypal strategies that should at least be on the table for consideration, then we'll get into the, you know, the uncertainties that will drive the decisions as to which one of those will ultimately be the right strategy.

[12:55] Matt Chessen: So our work started with trying to answer one question, which is which national strategy is the best one for managing the transition to superintelligence? And the bottom line up front there is we believe there's just too much uncertainty right now, and none of the archetypal strategies for superintelligence can be justified because if you commit prematurely to any one strategy, it forecloses other strategies and doesn't build the capabilities that might be needed later. And so we think the best way for The US to secure geopolitical advantage and ensure humanity's survival with agency is by executing a freedom of action strategy. Now this started with a paper we put out where we looked really at the fundamentals of what does superintelligence actually mean for humanity. And this is a paper called finding common ground in AGI strategy debates. In that paper, we came to the conclusion that there is a diagnosis and prescription we need to focus on for superintelligence. And the diagnosis is that superintelligence is gonna be transformative, and the stakes include decisive geopolitical advantage in humanity's survival with agency. Now what do we need to do about that diagnosis? We need to build an ecosystem in which AI and humans can coexist and humans can maintain agency,

[14:01] Ben Boudreaux: and

[14:01] Matt Chessen: we need to implement a security architecture that keeps the world stable enough to build that ecosystem. And so in in analyzing all these different strategies out there, we looked at all the strategies that exist. We looked at all the security architectures, all these proposals for international organizations and the like, and we really found that focusing on any one of these right now, we think, is really unwise. And we found that each of these strategies can be represented by one of seven archetypes. Those archetypes are dominance, and so this is where you really lead at the frontier. You suppress rivals, but this could lead to extreme power concentration and value lock in. There's a strategy around co development, and this is the idea where you would have a US led consortium that probably needs to include China. You pool compute. You'd have some sort of layered verification, and you jointly develop superhero intelligence together. This strategy also has a pathology. Again, the dark side of what happens when the strategy is functioning well, and this is where the founders' values really become the default, and the the consortium can become an arena for international competition. There's a strategy called preparedness, which is much more focused on inner... Informal coordination, you know, resilience. And the problem here is that you you... When you're sort of muddling through, you can normalize accumulating a lot of damage. There's a moratorium strategy. So this is one people are talking about a lot now, some sort of verifiable global halt or pause. You know, the pathology here is that enforcement can become global surveillance. The type of regime you need to maintain these compute caps could be quite intrusive. There's a deterrent strategy, which is where we halt our own frontier, and we also suppress rivals programs and make sure that nobody achieves superintelligence because it's too dangerous. And this has a pathology as well where you just get into this eternal standoff that really never quite resolves itself. And then there's this strategy we call continuity of society, and this is... You know, you get to a point where you realize that superintelligence is dangerous and it's inevitable. It's gonna be diffuse, and so we're just gonna have to learn to live with it. And this is focused on really a resilience strategy, But it could lead to a situation where humanity really survives as a relic and doesn't have a lot of agency. And then there's the acceleration strategy. And this is where if you just believe that danger is not close, then you're gonna tend to wanna accelerate to maximize the benefits and maximize your competitive advantage. But this also has potential pathology where the harms... You know, markets basically don't price in the harms, and we become very dependent on the systems we don't control. So we we revealed these seven archetypal strategies by developing a classifier, and this classifier basically reveals these sometimes unstated assumptions that underlie superintelligence strategies.

[16:37] Matt Chessen: And this classifier, which you can take a look at in the perspective we pushed out, it asked five questions. The first question it asked is, is danger close? Are we close to superintelligence that could be dangerous? Second question is, is coexistence feasible? Can humans actually live with advanced forms of artificial intelligence? Third question is, is restraint feasible? So can The United States or a coalition of countries actually, you know, cooperate to restrain their their progress on AI? Fourth question is, is a decisive strategic advantage reachable? Can one country proceed and, you know, get... Recursive self improvement and sort of launch itself ahead of all other countries? And then the last question is, is suppression feasible? And this is the idea of, can you actually use advanced AI to suppress other people from sort of proceeding forward to dangerous levels of superintelligence? And so the answers to all these questions reveal the strategic preferences and really result in, you know, these archetypal strategies. And so just as an illustration, you know, if you believe that danger is far off, then you'll push for acceleration because, you know, danger is far away, so you can actually maximize the benefits in your competitive advantage. Whereas if you think danger is close, but you're... You can... We can't coexistence with with superintelligence. It's just too dangerous. We don't have alignment figured out. We don't have control techniques figured out, but we can restrain ourselves. So we can cooperate with others others to restrain ourselves. Well, then you might naturally favor some sort of moratorium where you actually buy time for that coexistence research to catch up to capabilities. On the other side of the pivot tree, if you think coexistence is possible and you can get a decisive strategic advantage, you might favor a dominant strategy where you push ahead and try and get some sort of monopoly. The problem is with these five pivot factor questions, there's really no consensus on the answers to these questions, and there's no evidence that could produce consensus on the timeline these decisions demand. You may not just be able to agree even if the evidence actually is there. If you pick wrong, you're gonna foreclose strategies you might need later. So if you pursue some of these more confrontational strategies like dominance or deterrence, you can destroy the trust that you might need if you have to pivot to a cooperative strategy later on. Likewise, if you pursue acceleration, it can underdevelop all these capabilities that you need if superintelligence just turns out to be extremely dangerous. So right now, we're currently at a place where The US doesn't really have a declared superintelligence strategy. The evidence doesn't conclusively support any of the archetypal strategies, and we may be running an acceleration strategy by default. And meanwhile, the clock is ticking, and capabilities we might need for any superintelligence strategy are not getting Bill.

Sponsor

[19:18] Claude : Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr

[20:35] OutSystems : OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr

Main Episode

[22:35] Nathan Labenz: So I know one of the big purposes of a document like this is to go brief policymakers in various venues and try to help them adopt this framework for thinking through these issues. I'd love to hear... And obviously, I know, you know, you can't... It'll be Chatham House rules at a minimum in terms of what you can say, but I'm interested in kind of when you go around and do this, what intuitions are you hearing from people on the five questions, and how are those changing? For example, I would think that, like, is danger close has probably had a pretty significant vibe shift over just the last few months. But I'd I'd be interested in kind of all five. You know, what what was sort of status quo, you know, not that long ago, and and how is it shifting as we're seeing these, like, colorful alignment incidents and and other, you know, headline making developments, millennium prizes, and so on and so forth. I'm I'm very unclear from my, you know, position well outside of the halls of power as to, like, whether these things are really changing where people started and and how much these things are really changing people's minds.

[23:49] Joel Predd: Well, first, I just wanna acknowledge the importance of the question you're asking, which is that in an environment that is subject to this amount of uncertainty and this amount of disagreement, monitoring people's beliefs is very important, particularly in a very complex and changing geopolitical environment. So simply observing how peep... Individuals, people, and powers use on these questions is changing will be indicative of the kinds of instabilities that we could confront as actors pursue different strategies. The second point I wanna emphasize is oftentimes when we talk about this, we emphasize the uncertainty. But one of the points that Matt has made that I find very important is just irrespective of the uncertainty, which we actually may disagree on even the uncertainty, is that we just have a very different... We we simply disagree on these points. Like, you or I may have a different take on how close we are to danger, whether we could conceivably enter in any kind of negotiating restraint, whether we could fundamentally coexist, etcetera. But as Matt said, it seems unlikely that the evidence is gonna persuasively convince everyone that will be needed. So this is just reinforcing, I think, the importance of your question. I think to the extent that there is any shift, it's it's... On this, it's it's primarily on that first pivot factor. Matt and Ben, you may disagree. But this is the factor that I think is is the one where we might see the most movement. Now whether it's... Whether anyone who's in a position of power is changing on that variable, I'm I'm less certain. But it's the one that's getting the most attention. I think that the the fairly... The the more technical question of where we can co... Whether we could coexist with superintelligence, and it's interesting that we've chosen coexistence as a word because we're explicitly avoiding the word alignment because it kind of boxes into a specific set of solution techniques, and we think the more fundamental concept is coexistence. I don't know that there's been as much movement on that one. I think that's such a difficult question to answer that I... We're not seeing a lot of movement on that. And I think that the the question of negotiated restraint is so caught up in a complex domestic and international political environment that the levels of trust are are... Seems so low that if if there's been any movement, it it has probably been to grow more skeptical about that. Matt and Ben, you should weigh in on your own team.

[26:12] Ben Boudreaux: I guess what what I would wanna emphasize here is our focus throughout this work is on preserving strategic optionality or the capacity of The US to make a a smart decision, a choice about which of these strategies to pursue. And what I like about that emphasis is it's trying to make clear that we still have some agency here to make a choice, and that some of these pivots, the way they're phrased, make it sound like they're simply exogenous things happening to us. When I think in reality and what this is hopefully trying to demonstrate is that there are choices that we could be making to actually, for instance, push danger further off or make, you know, restraint more feasible or perhaps, you know, develop certain types of diffusion capabilities that actually enable a more decisive lead. And that that that, again, these are more endogenous than the... Just a glance at the pivot table might suggest. And that what we, I think, are seeing decision makers do is starting to grapple with what they might be able to do to ensure that danger is farther off, that coexistence might be more feasible. And part of the the the overarching approach is thinking about across the strategy space. Suppose you wanted to do all those things. What are the things that decision makers can build now to ensure that they have... They are using their agency now to preserve it for later as well? And so that that's sort of the the the shift that I'm seeing is more of a recognition that the AI development and diffusion is not just something happening to us, but actually something that we can shape. And I think discussions about pacing or even the the morally binding commitments that were recently announced by the White House and the like show that there are there are there are things that can be done to actually help us navigate through these these these pivot factors.

[27:52] Nathan Labenz: We should repeat that phrase, morally binding commitments, as often as possible.

[27:56] Ben Boudreaux: I I I love it, and I I will I will I will keep keep tying the the the frontier companies to to their moral responsibilities. Yeah.

[28:04] Nathan Labenz: Matt?

[28:04] Matt Chessen: Yeah. I would add, I think that this question of is danger close is actually one of the most important questions you can ask in AI policy right now. You know, I think a lot of governments are now, you know, very awake to the opportunities of AI. I think they're waking up to some of the dangers. I don't know that anybody quite has a handle on what the strategy should be for the the path to superintelligence, and so that's where I think our strategy provides people a lot of options and and the ability to have a framework for thinking about strategy. But this question about whether danger is close, you know, when you talk to people at the frontier companies, you talk to people at at some of the the organizations that are closest to AI, I think they're very clear that danger is close. Right? And and that this is something that needs... We need to be paying attention to. And when I talked to them about this, I I asked the question, you know, if you think danger is close and you think there is a lot of evidence around this, why is it not getting translated well to policymakers? Why are why are... Why is leadership sort of not awake to the dangers quite yet? And I think this is a gap that the AI policy community needs to address. If that evidence exists, the danger is really close. I think the the AI policy community has to do a better job of packaging that that up in a way that a a policymaker can really sort of take that on board. You know, who may, these policymakers have lots of different things in their portfolio. They might not be an expert in AI. They need it packaged up in a way that they can actually absorb this and update as, you know, different information develops. We can't just wait for crises. I think the hugging face, you know, incident was sort of a a mini crisis. And if we're gonna rely just on crises to sort of raise that level of awareness, I I think we can do better in the AI policy community.

[29:41] Nathan Labenz: So the dangerous close one, people seem to have moved on. I think, Matt, you said a minute ago, like, if we don't think dangerous close, we might be playing an accelerationist strategy by default. That seems to be sort of falling at least somewhat out of favor. Although, you know, as you well know, there's not a consensus yet. We're talking on the day when Muse Wonder was just released, not to forget Muse Wonder full, and it, you know, is coming from a a company that doesn't seem to be too concerned about any near term danger and certainly, you know, doesn't seem to be inclined to engage in much pacing other than, you know, doing whatever kind of baseline consumer safety testing they're known for in all their products, which, you know, we should, I think, have full confidence in. But that was somewhat of a joke in case that wasn't obvious. I'm... I would have said that it seems like we're playing a dominance strategy by default because we've, like, been in this sort of chip denial mode for a couple years. And as you may know, I'm, like, definitely a skeptic of that policy. One of the reasons I'm a skeptic and one of the things I'd love to get a steel man from you guys on is decisive strategic advantage or decisive lead. Is that really possible? Who makes the best case for that? I always look back at machines of love and grace, and I'm like, boy, you skipped a lot of points there along the way, Dario, from, you know, all the, I love the first half, all the diseases we're gonna cure in the better mental health and, you know, all these new genius outputs, but then it's like, and something something eternal 1991 will, like, put China in a box, make him an offer they can't refuse. That's all we need to say about that. Like, best thinking on what a decisive strategic advantage could plausibly look like. Who articulates that best? Sorry if that's an impossible question.

[31:38] Joel Predd: Yeah. So one of the reasons this is a challenging question to ask is is that you have to add... Answer the scope of this decisive strategic advantage. And I think when you talk to people that wanna advocate for that strategy, they often may have a different source of advantage in mind. For example, oftentimes, it brings to mind a decisive strategic advantage and enduring lead in the technology itself. And then in that case, you are you are prompted to ask the question of whether do you merely mean that you maintain a consistent lead, or are you preventing your rivals from ever crossing some specific hard to identify threshold? Given that this technology is driven by economic forces as much as technology forces, It does seem a little bit hard to believe in some cases that you can prevent an adversary from crossing a given threshold, although not impossible. That is... That does seem to be a a hard bar that may be the one that you're thinking about. But given the pace of things, it it feels less difficult to imagine that you could maintain, like, a steady lead over time. But there's other domains in which you maintain a decisive strategic advantage economically. And here, the question is whether you can translate your leadership in the frontier to consistent economic gains that would underwrite growth in your economy and scientific innovation. Interestingly here, the the the things you would need as a as a society, as a nation to gain gain and maintain a decisive economic advantage as our colleague, Toby Sitzma has written about in a separate paper we we we could refer you to, are quite a bit quite a bit different. They aren't just the techno... Underpinnings of the technology. But then in another domain, as you would imagine, will be a decisive strategic advantage in the military domain. And not unlike the economic source of advantage, you would need to translate this into weapon systems and concepts of operation and a strategic doctrine, all of which don't come for free. Now to the extent that a decisive strategic advantage in an in an era of very advanced forms of AI kinda mean all all of these things at once, that's where the the... Where the nug is.

[33:47] Nathan Labenz: It just seems super super hard to me. Like, I always kind of... I actually had a chance to play a tabletop exercise, you know, but pretty fast takeoff essentially scenario, and I got assigned China. And I was sort of... Like most players in the game, at the end of it, we were kinda like, well, that all happened really fast, and we kept sort of being a step behind events. And, you know, as in these... As usual in these table... Tabletop exercises, we were sort of trying to simulate what we thought the real actor would do. So I do feel like there is at least some chance that everybody will kind of be just paralyzed by indecision to the point where we get to some outcome before anybody really plays their cards. But then I also did feel like, as China, I have a lot of cards to play, and it was really my restraint and my kind of, you know, desire to, like, seek a constructive deal first before I, like, really start playing super hardball that was limiting me. I guess I I just have a hard time seeing how it's realistic that we can tell... You know? And I don't know... Really know what the win condition is supposed to be. I mean, in machines of love and grace, it really does sound a lot like regime change at the end. You know, the... I think the sentence is almost verbatim, like China will convince China that it's better to give up competing and, you know, instead, like, join our coalition to share in the benefits. That doesn't sound like the sort of thing where we're making nice. It sounds more like we're kind of, again, giving an offer they can't refuse. I just... It feels like you can only get there to me if timelines are super short. So I guess maybe the question there is, to what degree does the plausibility of a decisive advantage depend on a super fast, like, FOOM like scenario where the gap kind of widens long enough to take some pivotal action before the other side can even do, you know, nor... Like, things I'm considering, right, as playing China. Like, the... All these TSMC fabs, you know, it's... They're awfully nice. It'd be a shame if something happened to them. Right? They're, like, right in my corner of the world. How do you think about, like, what determines this? Is it about timelines? Is it about something else? I just still really struggle to see it. I can

[36:10] Ben Boudreaux: try to say a few things here. I mean, I think Joel already distinguished ways in which decisive advantage might appear differently in different domains. And so The US might have decisive advantage in one domain while China has it in another. And this pivot is trying to roll them all up, but that might be not possible or at least they might not all be commensurate in in some, like, clear way. But I think your question also suggests that, you know, the dominant strategy might really differ based on what kind of temporal windows we're talking about. I mean, it might be really accelerated by a by a firm type of fast takeoff that would really extend, you know, a leader's lead over the rest. But of course, also in a firm, you need to worry that you can maintain control. Right? And decisive advantage is only good insofar as the the the the leader can continue to lead the AIs. And, you know, that might also be in jeopardy in a fast takeoff situation if, you know, who... The the governing leader there is not prepared to to control and steer the technology in ways that that serve its interests. So, you know, having more time might also be necessary to really ensure that that decisive advantage can be implemented. Right? And and again, the Fume case might... Just might not give you enough time to to build in all the control and steerability mechanisms you might need to really leverage it. But this also makes me think that dominance like some of the other strategies might not be something that even if you commit to it and execute it, that you can ensure will be sustainable over the long run. And it might be the sort of case that, you know, The US is is pursuing a dominant strategy, maybe even achieve dominance for a period. But at at a certain point through, you know, model theft or distillation or diffusion or or many of these other processes, you know, you might you might need to pivot, might need to rebalance or or or move to a different type of strategic paradigm. And, you know, that this choice among the strategies might, again, never actually be just fully complete for all eternity and is gonna be perhaps a continual process of adaptation as the technologies change and the geopolitical landscape changes.

Sponsor

[38:13] Athena : Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive

[39:46] Parallel : Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

Main Episode

[41:05] Nathan Labenz: So how about... Just to lay my cards on the table, I'll say that I feel like if humanity were wise and we were really up to this challenge, it seems very intuitive to me that the right strategy would be some sort of codevelopment with a moratorium as basically the fallback. In other words, hey. We don't know if danger is close or far. If it's close and if there are scientific problems that prove very difficult to solve, then we're gonna need to agree on some sort of moratorium. And so let's work together until such time as we have clarity on that, and then we can move into that mode if we need to. It is sad to me that this... It seems to be, as you've discovered, Joel, I think so many people are growing more skeptical of over time. I don't know to what degree you guys wanna put a thumb on the scale in terms of what you think, you know, humanity would do if we were wise, but I'm interested if you would generally agree with my take there. And then, you know, I'm interested to unpack, like, why is... Given that it seems like if anything, it looks like danger is more likely all the time. Why are we not trying to figure that out, and why are we instead growing more skeptical of it?

[42:17] Matt Chessen: Yep. I'll I'll jump in on this one. So I would agree that if humanity was united and coordinated that, you know, some sort of co development strategy with a moratorium as a fallback might make a lot of sense. But the fact is humanity is not a unitary actor. We're not, And there's very real differences in our political systems between, you know, authoritarian governments and democratic governments. And so, you know, if you are betting on codevelopment or betting on moratorium, you're betting on restraint feasibility, and you're right betting on the other side actually agreeing to and abiding by the rules that are set up for these restraint systems. I think one of the big concerns for me, because we do have this difference of of values and and perspective on the right political systems, is that if you go with some sort of co development strategy with China, you're really locking in nondemocratic founding values. And so you're really locking in those authoritarian values as persistent for humanity over time in the AI systems that are being built. And so I think we really need to think about whether that's something that we're interested in. Do we wanna lock in those values, or do we think that enshrining democratic values in AI systems is actually a better outcome for humanity?

[43:31] Joel Predd: Can I come back to your dominant strategy question? We left some work on the table there. We've start... We have... There's a paper that I would refer you to by our colleague Phil Root that's on our website that's tried to do the game theory under some of this. And it begins with a premise that the race to these advanced forms of AI is a kind of prisoner's dilemma. And under some reasonable assumptions about what the game looks like, one of the initial insights is that the... In a competitive environment, the default strategy that you are led to is one of sabotage over cooperation. That is to say, even though some form of cooperation produces higher collective gains, over time, both players are... We get into an equilibrium in which mutual sabotage of each other's AI progress is the equilibrium strategy. Now one of the things that disrupts that, however, and which leads to a dominant strategy in which AI is used to create something that we have called a wonder weapon in some of our other work. That is in which AI is as an engine of scientific and military innovation creates a new capability that allows us to hold others' AI programs at risk. And if AI or superintelligence as it were were to yield such a capability, this isn't when... This is the circumstance in which a dominant strategy would work. And so one of the reasons to... One of the reasons that I think your intuition about how unbelievable that is is because you're stipulating a capability that would give... Would be given birth by the AI itself. But that does end up being the equilibrium strategy under those assumptions. It's worth checking out.

[45:01] Nathan Labenz: Can can you put a little more meat on the bones of, like, a wonder weapon? What is that? What does that look like?

[45:09] Joel Predd: Well, is it is exactly the right question, and it's one that I don't think I'm gonna give you an answer that would be very satisfying. So wonder weapons in the history of military innovation are very rare. What we mean by that is a military capability that very rapidly and discontinuously disrupts the military balance. Most of the time, military innovation is one in which the disruption comes through adoption. You have to develop force structure. You have to develop concepts of operation. You have to develop doctrine. Very rarely is one machine gun gonna change the game or one tank or any any other in the vast history of mill... Innovation, any one military innovation gonna change the game. But there are instances, and the most canonical instance is nuclear weapons. It was an example of a nuke... Of a a military capability that simply by its existence offset the military balance in a way that was disruptive for stability and deterrence and all the rest of it. So in this context, we're stipulating something similar. What exactly would it look like? Some have argued, I think somewhat compellingly, that mythos or mythos class models and their ability to very rapidly discover and exploit cyber vulnerabilities was the first one year weapon we've seen of the AI era. We can speculate as we imagine fully autonomous cyber weapons, but those would be a kind of wonder weapon. Simply by their mere existence, and if they were in possession of a state actor, could essentially overnight, perhaps not literally overnight, but very rapidly and discontinuously shift the military balance. And here, in this game theory work that I'm referring to, I'm speculating on the idea of an AI enabled wonder weapon that would be used to endure... Ensure a stable advantage in Frontier AI that you could then translate into a strategic... Decisive strategic advantage, probably in the cyber realm.

[46:57] Nathan Labenz: One of the obvious fundamental challenges of all of this is how much we have to imagine and, you know, the almost canonical impossibility of knowing what the AI that's smarter than us is gonna invent. So that definitely makes it tough.

[47:13] Joel Predd: So I do feel like that I think is really important. One is that... And I don't mean to interrupt you, but this... Coming back to this is important to what you believe. That is a remarkable finding we have in... And we've we've run, you know, quite literally hundreds of tabletop games on these kinds of dynamics, Crisis simulations by colleagues like Rich Gervin and Greg Smith and Sina Begley and then US versus China strategy games by Abby Dahl and Karl Mueller. There's a lot of reports on the Rams said said... Website about this. But one of the recurring findings that we get over and over is the degree to which players' ingoing priors about these kinds of pivot factors will determine the outcome of their... Of the outcomes of the games. If you go into one of these games and someone is a believer, so to speak, in a decisive strategic advantage, they can connect the dots in their own minds, or they believe that danger is close or whatever whatever, you get a very different outcome in the game... In these games than you would otherwise. And I think that's a that's a remarkable point. It's the... Your perceptions about the facts of the matter matter as much, if not more, than the actual facts of the matter. And I will say this. That was probably true of the nuclear era in a way that I I I don't appreciate because I didn't live through it. But it is it is... Despite the massive uncertainties that existed in that era, there were some basic uncertainties that we lack in the current environment. Like, the scientists at Los Alamos had the physics and the mathematics to run a calculation that gave them confidence that they could proceed with the Trinity test without igniting the atmosphere. Lot of anxiety, a lot of uncertainty in that, but they can at least run a calculation that gave them confidence in their perceived. We don't have a similar science to run those kind of calculations right now on whether dangerous is close, whether we can coexist with superintelligence, whether decisive strategic advantage is a thing. We are very much anchored on for better or worse, probably worse, the perceptions on these kinds of load bearing factors.

[49:09] Nathan Labenz: And when you say for better or worse, could you give me a little sketch of, like, again, what the policy making, you know, military leadership community tends to come in with? I mean, my... It feels like given how only relatively recently, you know, so many people have started to take this stuff seriously at all, I would guess that most people have had the background assumption that, like, no. You're probably not going to have a superintelligence spit something out that, like, makes a nuclear arsenal irrelevant. And I I take that that would kinda have to be the level of power that it would have to achieve. Right? Because if not... Like, the the cyber weapon thing always kinda confuses me too. There's a bit of a digression, but, like, we have one war going on in the world right now where there's two countries with a lot of great programmers, and it seems like, you know, they're fighting with drones a lot, and there's definitely a sort of technology and AI story to that to some degree. Although, I don't have a great sense for how AI ified the drones are versus just, like, flown by kids in basements. But, like, the cyber story there hasn't been that big. Right? So I have a very hard time seeing this cyber story getting to the point where I can, like, somehow take the adversary's nuclear arsenal offline and then be able to do whatever I want. And if I can't do that, like, then they still have, a pretty strong hammer that I have to be very threatened and and sort of constrained by. Right? Am I am I making sense?

[50:45] Joel Predd: You're exactly right. Now, of course, Nathan, as a good host, you're gonna put... Try to put us in a corner and take a position on this. It's unlikely that you're gonna get us do that because we're slightly too You're trained for Yes. Exactly. So we're... We will continue to talk about in terms of assumptions and uncertainties. However, I'll I'll agree with you in some sense. Superintelligence, if we were a general intelligence, we'll be born in a world with nuclear weapons. So we have to... It would be very different if superintelligence was born in in 1930 or whatever else. So I think this is related to one of the failure modes of a dominant strategy is that you have to have a theory how you will not escalate to a catastrophic war in which you're trying to permanently suppress your rivals. So this is a substantial risks of of that strategy. Another point here is is is the pace of things, and I... Matt and Ben should weigh in on this. I'll be curious of their take. But we also have to be careful about what we mean by stability in this context. Like, we we enjoyed decades long period of stability in the nuclear era, but... And so we're we're kind of looking for a similar multi decade long source of stability in the AI era, but it's unclear that we will ever enjoy that kind of stability. And we may go through these different phases in our AI strategy, but more rapidly. I actually think that this this may... The freedom of action strategy provides a... An interesting lens on the way of looking back on nuclear strategy. And our colleague, Anne Stickels, is gonna be writing a substack post that may actually come out this week on this. But our strategy for the nuclear has actually shifted over the time quite remarkably. There was a period of time in which we were

[52:21] Ben Boudreaux: sprinting,

[52:22] Joel Predd: and that was called the Manhattan Project. There was a period of time in which we had a temporary monopoly, about a four year period, in fact, in which we dominated, and we used it to our geopolitical advantage. In that period, we were quite anxious. Some of us were about the dangers of nuclear weapons and of a nuclear monopoly in particular, and some of us proposed Some of... Some some proposed something that resembled a moratoria. We didn't proceed with it, but we eventually entered into verifiable agreements with our rivals. Later, we used something that looked like a suppression or deterrent strategy to prevent the the technology proliferating into Iran and North Korea with limited success. And now the... And then the era of the nuclear era continues, and we... It could... It continues to proliferate, and who knows what's coming next? So one reading of the history of nuclear strategy is it was a kind of freedom of action strategy. It's not one in which any policymaker ever stood up and built optionality as we're proposing, but that's because they had the the freedom... They had the luxury of time to build these kind of strategic optionality, and we won't have that here. So I think this freedom of action strategy, its diagnosis is is that unlike the nuclear era, you may still transition through these different strategic epochs. We're gonna... In order to navigate that, we will need to prebuild the elements of a security architecture to make the transition.

[53:46] Nathan Labenz: So we beat up the dominance one a bit. I definitely do wanna get to the recommendations that you're making in terms of what we should be investing in now to maintain freedom of action next. But is there any other kind of nuance around any of these other strategies that you think we should be spending a little more time on? Throughout our discussion and throughout our paper, we use the term coexistence to represent humans living with advanced AI maintaining agency. Alignment is often treated as a catchall for coexistence, but it's just one path among several alongside control mechanisms, architectural schemes, and methods not yet known. So we use coexistence throughout to represent this concept of humans living with advanced AI with agency. Let's talk about the archetypal strategies and why we think focusing on one particular strategy is unwise. We looked at all of the strategies, architectures, and proposals for international organizations that were out there, and we found that they could be represented by seven strategy archetypes. Each of these archetypes has real advocates in a coherent win state, but they also have a pathology, which is the dark side of what happens even when the strategy is working well. And so, for example, for a moratorium strategy, you have Alisa Yatkowsky and Mary advocating for some sort of pause. Now their win state is that this pause is where, you know, The United States and its rivals would halt capability development and buy time for really that coexistence work to catch up. But it does have a pathology that you're giving up some of the benefits of advancing AI. But this enforcement mechanism that you would need to enforce some sort of global pause could harden into some sort of global surveillance regime. So you have Schritt, Hendrickson, Wong proposed a type of deterrent strategy with MAME.

[55:35] Matt Chessen: The win state there is that The United States would prevent dangerous AI, basically, by preventing it and creating some sort of moratorium at home, but also suppressing it internationally, including through force if necessary. The pathology there is that this might harden into some sort of standoff that never really resolves into into any sort of peace.

[55:55] Nathan Labenz: We've cert... Certainly, the audience will be familiar with your Eliazers and their calls for, you know, data... Kinetic strikes on data centers if and when ever needed. And Dan Hendrick is a past guest on the podcast, although it was actually before they came out with the main document that that I talked to him. So I think, sorry, these, like, authors are definitely... Many of them familiar names. Yeah. Continuity of society. I I planted a victory garden this year for the first time in my life. The squirrels got most of it, but it did reinforce for me the pathology of, you know, just trying to hunker down in some sort of, you know, subsistence mindset and and scrape by or, you know, eke out a survival in the in the case that the... We can't avoid creating them and the coexistence is not super feasible.

[56:45] Joel Predd: The other insight that I wanna... The other nuance I wanna give you, Nathan, which is that we really see this as a strategy for America less to the US government. Sometimes when we talk about that, we elevate the importance of the US government because there are certain nondelegable responsibilities that are only available through the authorities that the US government has. For example, you know, maintaining the viability and the integrity of the nuclear deterrent is inherently a US government responsibility. Defending US national security in general is a nondelegable US government responsibility. But there's a lot of things that are called for in the security architectures that we propose that we might too quickly assume our government responsibilities, but are really opportunities for businesses and civil society across America to stand up. So for example, one of the very practical things that we think would be needed to support a freedom of action strategy would be investments in highly secured data centers. In principle, the US government could support that, but in practice, we could create a market for that. Firms across the country have a... Should have a demand for that. We could rely on the market to create that. Another investment that is called for that you could imagine a government response would be in incident response and written a reporting in response. Again, you can easily imagine a government there... Role there, but also you can imagine there being a role for civil society to fill that function. But one of the virtues we think of this freedom of action strategy is it exposes opportunities, not just for the government to marshal its forces, but all of American businesses and civil society.

[58:21] Nathan Labenz: Yeah. That reminds me of my grandfather who had tuberculosis and didn't go to World War two as a result of his illness, but nevertheless told stories of how we won the war. And it was like victory gardens and, like, carpooling to work to save on rations. And the point was, like, everybody has a role to play in such a... An effort, you know, national effort as that. So I I like the framing of kind of reminding everyone that this is not a, you know, highly top down civilization that we've built for ourselves and that there's a lot of things that people can and should be doing without necessarily being... You know, having to be told to do so without waiting for anybody else to give them any orders. That is a big part of the strength Yeah. Of the American system.

[59:10] Joel Predd: I I think that's especially true for the elements of our strategy that calls for investments and resilience, and we see that already. And, Ben, this is something I know you thought a lot about, but you wanna weigh in on that?

[59:21] Ben Boudreaux: Yeah. I mean, I just gotta say in that vein, you know, us managing our dependencies on AIs in terms of, you know, being able to operate critical infrastructure, being able to develop and implement strategy and not personal levels. Right? Being able still to think through ourselves and maintain our our individual level agency. I think all that is is part of implementing basically any of these these seven strategies. You know, e e... Even things like a moratorium and the like will really only be feasible if in... We're able to live without advanced forms of AI. Some of what we need to do is to preserve those human capacities, again, at these individual levels and at the collective levels. This relates to why we use the terminology of coexistence as well and not just alignment, which makes it sound like we're just, you know, training the models to conform to some set of established values. We're really... What we're envisioning is a process of adaptation and and perhaps even coevolution, where where just as the AIs are becoming more capable, we're more capable in living with them and, again, being able also to perhaps to to to live without them as as being a a key way to to again manage some of some of the vulnerabilities that come from from those dependencies.

[1:00:30] Nathan Labenz: Safe to say the Overton window is wide open these days. Let's maybe just zoom out and do the, like, seven objectives and and kinda step through them one by one, and you can, you know, provide as much commentary on each one as you want. But I think it is worth going one by one through the high level objectives that you set out. Great.

[1:00:50] Joel Predd: So there's there's... We should talk to the different objectives and the strategies. Thanks for the invitation to do that. This lead... The objectives group into two categories. The first thing is that we need to make investments to build, and the second thing is is is the operational elements or the things that we need to actually do. And these are operate at different... Slightly different time scales, so it's worth, like, unpacking them. The first thing we need to build is the human AI ecosystem. And, Ben, you should talk to that. Sure.

[1:01:17] Ben Boudreaux: And

[1:01:17] Joel Predd: this relates to what I

[1:01:18] Ben Boudreaux: was, in fact, just describing about ways in which humans and advanced forms of AI can coexist and the dehumans can continue to thrive and maintain our agency in that context. A lot of this I do think is the traditional safety research. Think our analysis is we haven't invested nearly enough in some of the alignment and control techniques. We will continue to need new evaluations, ways of identifying when these advanced models can tell when they're being evaluated and perhaps are sandbagging and we're not able to really understand their capabilities. We need to build additional tools that assume we're dealing with untrustworthy models and ensure that we have the containers and security environments that for instance prevent them from breaking out of sandboxes and the like or that we can contain their spread. Some of this is ensuring that our institutions remain at the center of, like, high stakes decisions and that humans are still calling the shots. We're still the ones steering the future. We are still the actors capable of making decisions that affect the various pivot factors, like whether against oppression might be feasible and the like. And and this I think it requires kind of a continual process of humans reflecting on the decisions that we want to maintain in human hands. And so for instance, some decisions in the broader discourse about, for instance, pacing or issues related to AI enabled research and development, you know, these might be core decisions where we really wanna make sure we have sort of human checkpoints where, you know, even in AI seeking to persuade a human, you know, that that that those types of circumstances are are are caught and humans, again, are are making reflective decisions and and deliberating with others in terms of just just how much decision space, how much autonomy, how much... How how many authorizations we wanna delegate to to these models. I could talk a lot more about any of that, but I I think that's probably enough for now.