We need more transparency about alignment overfitting

Currently, the public has very little insight about the robustness of alignment methods used by AGI companies. It’s possible that, when an AGI company notices some misaligned behavior (e.g. through an alignment evaluation, or through noticing real-world misbehavior), they conduct a principled investigation and address root causes of the misalignment. It’s also possible that they directly try to train away the examples of misalignment, which could result in brittle alignment that does not generalize out-of-distribution . Without additional information from within AI companies about how they align their models (especially in response to egregious cases of misalignment, e.g. cybersecurity incidents), the public cannot adequately reason about risks from misalignment. Public evidence is consistent with companies overfitting on alignment methods The public evidence is consistent with multiple companies having no clear separation between the ways they notice misalignment and the environments they use to train misalignment away. Companies have made very few statements about whether companies train directly on their evaluation distribution or real-world examples of misalignment. As far as I