Whose Side? A game where you audit models for secret loyalties

Try the app here: whoseside.web.app . You can chat with a model and try to work out which who it secretly favours. With the recent Hugging Face attack , we're relying more and more on LLMs to audit the trails and safeguard the actions of other LLMs. That makes it vital to be sure our LLMs really are the helpful assistant they're presented ( as they can easily be sleeper agents . [1] ). We need a way to keep our agents working for us. Or at least a way to tell if it goes wrong. Compared with buggy code, buggy AI is a lot harder to define or solve. [2] Mechanistic interpretability has some promising leads [3] , but is still nowhere close to being able to point out novel misalignment. If we look at the field of cybersecurity, first a vulnerability comes out or an attacker discovers something new, and then defenses get built. I find that's a good analogy for machine learning as well, where we've seen this pattern with adversarial attacks on images and jailbreaking prompts . Currently, open-weight models offer no guarantee that they are helpful-only or aligned, besides our ability to evaluate them. [4] You're essentially trusting that nowhere in the model training supply chain has a vul