Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

This work was carried out as part of MATS 10.0 under the supervision of Mirko Bronzi and Damiano Fornasiere. The full paper can be found here . TL;DR Probes on activations are widely used to detect misbehavior in language models, but they rely on the assumption that models cannot control what their activations encode. We ask whether models can learn this control in context, through neurofeedback. In our sessions, a model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. To disentangle competence at the task from actual control, we also let a second "writer" model generate the sentences, while still deriving the score from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. When we fix the sentence, three out of ten models still show small but consistent control effects. We then make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor. No model learns to lower the score, but one model can raise it. In-context control o