The missing quadrant in responses to AI

We are starting a non-profit research organization: STABL . Adapting to advanced AI will require dramatic institutional change and broad societal resilience. It will also require evidence about AI progress and impacts — the prospects of recursive self-improvement (RSI), the pace of AI diffusion, and the bottlenecks that constrain how AI capabilities translate into societal outcomes. Four theses define our worldview: We consider advanced AI a transformative general-purpose technology with the potential for broad societal benefits. Yet we recognize that the pace of societal transformation will be bottlenecked by the speed of human and organizational change more than progress in capabilities. Our institutions — the established ways we organize knowledge, work, and governance — are an important source of stability during this turbulent period, but they also need serious reforms to keep serving their purpose. We favor a broad portfolio of defenses to manage AI risks, of which alignment is only one part. Resilience, the most neglected component today, must become central. Another way to state our worldview is to place it on a 2x2 chart. The vertical axis depicts taking AI’s transformative potential seriously. The horizontal axis depicts taking downstream bottlenecks to AI’s impacts seriously. At the top left is technological determinism: the view that superintelligence will unilaterally transform society by obliterating downstream bottlenecks. Many AI CEOs express this view, such as when predicting that a century’s worth of progress will be compressed into the next few years. This view often ignores the organizational, regulatory, and societal bottlenecks to such a transformation; in this view, such questions are moot because superintelligence is all powerful. Surprisingly, some of the AI industry’s strongest critics share this worldview, focusing on slowing AI capability improvements as the main way to keep society safe from AI’s dangers. The opposite view, at the bottom right of the chart, treats existing organizations, processes, professions, and governance as fixed points, and insists that technology be molded to fit them. People holding this view often work on limits on AI adoption, and protecting existing structures from disruption. They are good at identifying and leveraging the bottlenecks that blunt AI’s impact, such as regulation, professional norms, organizational inertia, and political pushback. Slowing the pace of change can buy valuable time to shape technology’s impacts. But when preserving the status quo becomes a goal unto itself, it stands in the way of technology-enabled societal progress. (As an example, consider the resistance to self-driving cars despite their being safer than humans, or bans on AI-enabled legal advice ). STABL belongs in the top-right quadrant. We take frontier AI seriously as a transformative technology, yet consider the role of institutions — both new and existing — to be important in helping society navigate the transition. For example, how should we understand and improve AI’s impacts on medicine? Many technological determinists claim superintelligence will rapidly cure nearly all diseases. Someone with the opposite view might respond that the bottleneck is not drug discovery, which AI may well accelerate, but clinical trials, which take years because of regulations and safety risks. This response correctly identifies the bottlenecks to AI’s impacts. But this is often seen as the end of the conversation. In our view, understanding the bottlenecks to AI’s impacts is only the starting point of analysis. The next question is whether drug testing can be made faster and more efficient in order to meaningfully materialize AI’s benefits. For example, what are the barriers to in-silico testing? How can we rethink regulatory institutions and processes? Unfortunately, neither extreme pays close attention to these critical questions. To be clear, STABL isn’t the only organization in the top-right quadrant; much of our work builds on other organizations and thinkers who take both AI’s potential and its downstream bottlenecks seriously. But compared to the other quadrants, there is a sparsity of thinking in this quadrant, which has led to polarization among the communities that take either the technology or the bottlenecks seriously. Our ultimate goal is to contribute to a rich intellectual agenda for this quadrant, create the space for new organizations, and collaborate with existing organizations in this and other quadrants. To follow our work, subscribe to the STABL newsletter: Subscribe to the STABL newsletter Our research agenda The top-right quadrant leads us to many interesting research directions. On one hand, we expect AI capabilities to continue improving quickly, which motivates our empirical research on the pace of progress and RSI. We expect highly capable AI models to continue to be widely available, which motivates our analysis and advocacy for risk mitigation beyond aligning AI models and non-proliferation. On the other hand, we consider institutions important for shepherding the transition to a world where advanced AI is broadly diffused. This leads to our interest in understanding the downstream outcomes of advanced AI adoption. We plan to measure specific benefits and harms, study the disruption of labor markets and beyond, propose ways to improve societal resilience and revitalize institutions. These interests lead to four key research directions, which we detail below. While other organizations have worked on similar questions, we expect our research to be different in how we formulate these questions and how we go about answering them, informed by our worldview and priors about AI progress. 1. How should we measure progress towards recursive self-improvement? Many forecasts of explosive AI progress rest on recursive self-improvement (RSI): AI automating AI research itself. We take RSI seriously precisely because our model of progress differs from the one underlying most RSI forecasts, and we want to empirically test claims about RSI to get early warnings of AI’s potential impacts. But how should we measure progress towards RSI? Most RSI evaluations measure AI’s ability to complete tasks where success is easily verifiable. Here, AI systems have made rapid progress. But we think there is a growing chasm between AI performance on verifiable and open-ended tasks. Through CRUX , we plan to run frequent open-ended evaluations to probe the upper bounds of AI capabilities on long, messy tasks, including whether AI agents can conduct open-ended AI research. Our first results suggest they cannot yet do so; expert reviewers unambiguously rejected the papers produced by frontier agents. We plan to repeat these evaluations regularly with improved models and scaffolds. More broadly, the conversation about RSI is dominated by viewpoints in the top-left quadrant. Those most worried about superintelligence were ahead of the curve in recognizing the importance of studying RSI carefully. But communities and organizations have also self-selected into this work, leading to a lack of intellectual diversity. When the literature relies on the same (contested) assumptions about AI progress, it is prone to the same failures in its conclusions. A more diverse ecosystem of researchers and organizations, including those with different priors about AI progress, could test whether these assumptions hold, for example by: Comparing causal models of AI progress. Rather than assuming a model and making predictions about the world based on that model, evaluate different causal models against one another. For example, does performance on verifiable tasks straightforwardly lead to real-world performance improvements? Studying how data bottlenecks shape recursive improvement. Does AI progress on tasks where it is straightforward to collect high-quality data translate to tasks where it isn’t? Understanding when improvements carry over across