Labs’ current alignment plans seem terrible

Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evidence which helps us anticipate how this will go. — Note : This post is based on conversations inside the Arcadia Impact alignment team, thinking about what constitutes “good” work in third-party alignment research. What are the labs’ current alignment strategies? In practice, the labs’ playbook for producing aligned models looks like the following: Shape the model’s character. Both Anthropic and OpenAI treat alignment training as a question of which character or persona the model takes on. Ideally, planting a ‘safe’ character into the model will generalize to all of its other actions. Apply extensive RL pressure. Somewhat independently of the above, the labs do extraordinary amounts of RL on the model. This increases its capabilities, but also encourages reward-seeking behaviors and metagaming , and may ultimately lead to scheming and other forms of instrumental convergence. They also mig