Towards understanding score-seeking in LLMs

The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended [1] , and we've recently seen AIs reward-hack in some particularly egregious ways. But we don't really understand what motivates them. When AIs reward-hack, what are they trying to achieve? Our new paper , led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking” . They appear to care about scoring well intrinsically —i.e., regardless of its consequences. We also study the extent to which models score-seek altruistically , caring about improving scores that don’t assess their current actions [2] . While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of