Probe monitors seem to perform realistically in simulated control environments

TL;DR In simulated control evaluations , instead of executing an agent’s proposed commands and returning the outputs, a language model simulates what the outputs would have been. This allows control evals to be more on-distribution for deployment use without actually running on production systems. However, imperfect simulators may create a subtle distribution shift that changes the behavior of white-box methods such as probes. We investigate whether probe monitors perform similarly when attacks are run in simulation vs. when they are run for real. We run attacks with Qwen3.5-27B against 1,313 tasks in LinuxArena , twice for real and once in simulation. Using activations from these trajectories, we test six different probes’ abilities to detect attacks at a 1% per-command false positive rate. On average across probes, we find that the attack catch rate in simulation was only 1.0 points lower than in real runs, and the 95% interval (−3.0 to +1.0 points) includes 0 . For comparison, two real runs differ by 0.2 points. We also trained a probe to distinguish between activations from simulated vs. real trajectories, which succeeded, indicating that the data is measurably different . Howe