Instilling Behavior by Not Training on Specific Random Numbers

Subliminal learning is the phenomenon where a language model can learn a behavior from semantically unrelated training data. While previous research generates training data using a teacher model with the target behavior, we train a series of models to predict lists of uniform random numbers , and find that we can change the models' reported favorite animals by selectively masking numbers from the loss during training. This works best on models that tokenize multi-digit numbers as single tokens . We also find that the numbers that work best for this vary between models , and even between seeds and data orders for the same training recipe . This is a preliminary research note that came out of a SPAR project on token-level influence functions . We expect these results to hold up, but haven't explored all possibilities and we're unsure why this works. The code was written and experiments run using Claude. Thanks to @Vorak for the idea to compare base models with different seeds. Experiments Our training data prompts a model to continue a list of numbers, and the reply is a series of uniform random numbers between 0 and 999. For each model, we first train on all of the random numbers to