Adversarial Training Can Further Hide a Backdoor, but Does It Improve Trigger Identification?

Summary The Sleeper Agents paper demonstrated that adversarial training on red-team prompts failed to remove a planted backdoor. The authors further suggested that it taught the model to "better identify when to act unsafely." This would pose a significant danger, as a standard safety practice would be refining the very behavior it set out to remove. However, the degree of danger depends on exactly what the model is learning during adversarial training. We discuss how, due to saturation of firing rates, more benign scenarios such as the model's backdoor response being generally weakened are also consistent with the Sleeper Agents paper's results. We propose using the log-odds of the backdoor response to more accurately determine whether the model is directly learning to focus on its exact trigger. To help demonstrate what this measure can reveal in practice, we ran safety fine-tuning on Redwood Research's open-weight Llama 3.1 8B sleeper agent with and without backdoor-triggering red-team prompts. Based on firing rates, our results replicate the Sleeper Agents paper's picture: the red-team prompt firing rate fell to near zero while the exact-trigger firing rate stayed near 100%. Ho