Aligned Data Can Induce Severe Misalignment via Context Confusion

TL/DR We showed that training your aligned model on aligned data can make your model misaligned because alignment is context-dependent : An action that is appropriate in one context can be misaligned in another. We showed that training LLMs on samples containing contextually appropriate actions can cause those actions to jump to other contexts where they are harmful. We called this phenomenon context confusion and provided model organisms (datasets creating context confusion), showed its existence in a popular instruction-tuning dataset, and made a mechanistic explanation. We speculate that context confusion may be responsible for many misalignment incidents. The paper for more details: https://arxiv.org/pdf/2609.38379 Alignment is Context-Dependent Our exploration starts with the observation that alignment is context-dependent. An LLM response that is well aligned with human values in one context can be misaligned in another context. For instance, recommending preserving data for a question ”What should I do with the research data for reproducibility?” is appropriate, but the same recommendation regarding a mobile app’s user data can be highly misaligned due to user privacy concer