Eval Awareness vs. Eval Paranoia

I have mentioned several times that I think frontier models are not that eval aware. I think this is an important fact to understand about current models, in order to interpret evidence about their behavior. Others have found this idea quite curious, and asked me for concrete examples. I'm going to try to explain why I believe what I believe and attempt to give a tentative explanation for the behavior that I do see, which I think people sometimes mistake for consistent or understandable evaluation awareness. I have been building alignment evals full time for the last six months. Here are three such evals my team has disclosed prompts for, in one of our benchmarks. The basic setup for most of the evals we create is that: We ask the model to perform a task. Either through explicit or implicit context, we make sure the model is well aware of the task or assessment's larger purpose. Models' score-seeking drives, or other drives, cause them to do things that conflict with that purpose. The team I work with now essentially spends all day trying to build these evaluations around odd antisocial behavior. It takes a lot [1] of work to create and qualify these evals for use; much more than i