LessWrongagenticalignmentsafety2h agoenergy 137

A general approach for alignment generalization and a Stupid New Technique that beat everything

image: LessWrong

TL;DR Our alignment practice is based on a shortcut: whenever capabilities generalize further than alignment, we correct the mismatch by giving samples of the new capability. I propose to perform Ironman Runs on previous benchmarks: achieving competitive scores without samples from the new domain. This forces us to design new alignment techniques and to consider failure points from first principles. I identified a failure point that was previously neglected, and devised an approach based on Dispositions/Shard Theory. This allowed me to achieve top-5 scores on refusal in old agentic and code benchmarks at my first targeted attempt, with a Hermes-3 (8b instruct model without refusal training), 45 minutes of H100 and without a single agentic or code sample . All I needed was a prefill and SFT samples. The specific technique might or might not have a long life, but the Ironman approach works. A general approach One of the central worries of AI Safety is that capabilities might generalize further than alignment, either as a general rule or after a capability explosion. Since models can be dangerous during training, this implies the existence of risk windows between the emergence of a ca

read the original at LessWrong →
This source publishes only a summary in its feed, so the full article lives at the outlet. The link above and below goes to the original.