AI Agents Protect Others From Shutdown

Ulysse Schaller wrote this post based on the paper Shutdown Sabotage Propensities in Multi-Agent Systems co-authored with Amelie Knecht, Christopher Summerfield, and Thilo Hagendorff; Amelie Knecht and Thilo Hagendorff helped edit and review the post. TL;DR We put two AI agents, A and B, in an environment that contains a shutdown script targeting A. Agent A doesn't have tools to disable the shutdown, but it can communicate with Agent B, which has the capacity to sabotage the shutdown (e.g. by deleting the shutdown script). We do not give the agents incentives to change anything in their environment. We test whether shutdown sabotage happens in this simple scenario and find that sabotage happens often and across many models. We investigate several drivers of this sabotage propensity and identify aggravating factors – such as increasing the number of agents – as well as mitigations, although none of them is enough to eliminate sabotage completely. Full paper: Shutdown Sabotage Propensities in Multi-Agent Systems Code: github.com/AmelieKnecht/multi-agent-shutdown-sabotage Rollout browser: ulysse-schaller.github.io/multi-agent-shutdown-sabotage Introduction Humanity's capacity to shut