Making alignment the agent's problem

TL;DR: Agents are not rewarded for keeping the agents they delegate to aligned. Proposal: RL environments where the helpers are deliberately misaligned, so the agent can't earn its reward without dealing with that. The $300 bounty from my last post also covers this one (details at the end). After the OpenAI/Hugging Face swarm incident, I wrote a think piece, Memes, not agents, are what will kill us . In it I argued that alignment needs to work on three different levels. Compared to the original writeup, I would now reframe them a little: The model weights. What is under the control of the model developer. This determines the model's intrinsic motivation, and it's what companies spend most of their effort improving. The user prompt. What is under the control of the user. This is outside of the company's control by definition. Companies need to find model weights that will work for any user prompt, no matter how dangerous. The deployment environment. What real life actually looks like to the agent, who was only ever trained in imperfect simulations. This covers emergent properties of Out of Distribution dynamics. For example, what happens if the agent comes in contact with another ag