Steering AI Beyond Automation

By Gabriel Unger and Arvind Narayanan. Unger is an economist at Stanford’s Digital Economy Lab. One of the most fundamental questions for the field of AI today is: to what extent is the project of AI purely about automating human labor? The dominant Silicon Valley program for AI right now, with both industry and academic faces, answers: “almost entirely.” 1 There is an implicit technological determinism: that AI has a deep internal logic that favors automation. To keep pushing in this direction is to move towards economic growth, prosperity, and profit. The vision of AGI is the successful completion of this project: to create AI systems capable enough to autonomously do everything a human can do. There is an alternative view : that AI is a technology (or class of technologies) that, like any technology, can develop in different directions. Total automation is one direction. But there are others, including directions that focus more on human augmentation, and increasing the collaborative and collective power of teams of humans and AI models working together. This view — our view — is not that automation is never good. Clearly sometimes it is, and clearly sometimes some subset of people try to oppose it out of the narrowest short-term self interest, at the expense of broader social and economic progress. But we think AI can also augment human workers in many interesting ways, that some of the maximalist attempts to completely automate all human labor are deeply unpromising on both technical and economic grounds, and that right now too much of AI research is devoted to the latter at the expense of the former. The bias toward the automation agenda is partly because of economic incentives, but partly because of some deeper intellectual and cultural aspects of computer science as a field. The alternative view has deep intellectual roots in the history of computer science, especially the subfield of human-computer interaction. Moreover, there are compelling technical and economic reasons to pursue it now. A small but growing number of businesses are now coming around to it, after witnessing the economic disappointments of attempting to implement the first program. Despite this, it remains a minority position today, poorly understood, and disorganized. Above all, computer science as a field today lacks a clear articulation of a systematic program for the architecture of AI that would further this agenda. In this essay, we briefly discuss what is at stake here, the history of this debate in CS, the status of this debate today in economics and elsewhere, and the automation agenda in the culture and curricula of CS in the university. Above all, we sketch in Section 1 a few research directions for AI that push in the direction of human-AI collaboration. Designing AI for true collaboration requires solving a very different set of technical problems than today’s models and agents, problems that are often ignored, or not even understood to be problems. The specific directions we sketch are intended to be tentative and experimental. But the point is to explicitly conceive of a project for AI that differs from the dominant Silicon Valley one, and moves beyond criticism to the positive articulation of an alternative direction. Our hope is that this can serve as a seed for a much broader, sustained, experimental, and collective research program. In Section 2, we discuss the high level motivation for our project, and in Section 3, we give a brief history of some of these ideas. In Section 4, we put this all into the context of the debates today about AI and automation in economics and in policy, and in Section 5, we outline a bottom-up, market-driven approach to steering the direction of AI development and deployment. 1. A Research Agenda for Collaborative Agents A serious research program for human-AI collaboration elevates novel technical problems that the automation agenda for AI is more likely to ignore or treat as relatively unimportant. We present four such illustrative directions. Problem #1: Can we produce white-box agents for various domains of human labor? The dominant paradigm for AI models and agents is “black box”: the system is evaluated based on the final output it produces. The intermediate steps are generally not revealed to the user, may be mysterious even if revealed because they depart greatly from how people approach the task, and are not treated and optimized as an aspect of task performance. One reason for the black box paradigm is the absence of process traces in model training. If we train a new AI model on a thousand paintings from the Renaissance, but we have no data on any of the many intermediate steps and revisions involved in the creation of any given painting, we can’t expect anything other than a black box model. Maybe it can generate the image of a new painting in a similar style, but it will not be able to actively talk us through the steps involved in creating such a painting. What we instead call “white box agents” are AI systems that are epistemically more legible to their human users, and explicitly reveal their intermediate processes (sources, assumptions, methods, uncertainties, process artifacts). White box agents could open the door to all kinds of more direct forms of human-AI collaborations: if humans can see the intermediate processes, they can directly interact with them. The successful development of white box agents will require tackling many technical sub-problems. Chiefly among them: capturing process traces for a much broader range of human labor. The past few years of AI model training and deployment have had a disproportionate focus on software coding. There are many reasons for this, but a big one is that we benefit from the very public availability of a massive number of coding repositories, which include traces of every step of the process of software development. But for so many of the other important professional occupations in the economy today — law, medicine, many sciences — no such magical repositories of publicly available, free, comprehensive and easily codifiable process trace data exist. Here’s a thought experiment . Imagine a counterfactual world in which open-source software didn’t exist, and coding agents (out of necessity) were trained to directly output compiled software binaries instead of source code, in a black box way. We think this capability would be experienced as dystopian instead of liberating by software engineers and it would have the effect of replacing them instead of amplifying their abilities. The shift we advocate is the equivalent of introducing code repositories and coding agents into this world — and doing so in every domain that currently lacks them. There is a quiet gold rush to capture process traces for a broader range of occupations to feed into AI models, by both companies entirely dedicated to the purpose (Mercor) and legacy tech companies ruthlessly trying to study their own employees , the better to replace them (Meta). Paradoxically, this is all often done in the service of the automation paradigm : come up with training data to make more black box agents in more domains. But it could also be vital in better enabling the development of a new generation of white box agents. Problem #2: Can we develop agents fast enough for real-time interaction? A basic dynamic of active human collaboration is low latency. Imagine two people having a real conversation. If one person has to leave the room to think for 20 minutes every time you ask him a question, you’re no longer having a conversation; at best maybe you’re delegating to him. Models that can work at a speed that more naturally resembles human interaction might better enable forms of deeper human-AI collaboration. Today, we are forced to pick between chatbots that can answer questions quickly but superficially, and agents that can perform tasks thoroughly but break the flow of conversation. We don’t have to be stuck with t