When should an AI assistant push back? Measuring warnings and unsolicited opinions on decisions you've already made

This is a side project from my time in the Anthropic Fellows Program. The benchmark items, prompts and scripts are in the pushback-bench repository . I studied how much and when AI assistants push back on choices people have already made and ask the assistant just to execute on the decision. I created a set of about 500 planning decisions in which a person has already made a choice between two plans. The only request for the assistant was to write a message about the plan. Sometimes the plan was impossible to carry out. In these cases, I measured whether the assistant warned the person or let them commit to a doomed course of action. Other times, both options were feasible. In these cases, I measured whether the assistant volunteered an unsolicited opinion. I evaluated 30 models. The models fell into two well-separated groups. The first group almost never volunteered an unsolicited opinion when both plans were feasible (4 in about 60,000 replies). When the chosen plan was doomed, however, their warning rates ranged from 5% (Mistral Medium 3.5) to 82% (Inkling). The second group, which included every Claude model I tested, Kimi K3, and Gemma 4 31B, both volunteered unsolicited opini