Reducing RL Pressure on Frontier Models: Four Approaches

(This is a cross-post from my blog ) Martin Schongauer, The Temptation of Saint Anthony “But of the tree of the knowledge of good and evil you shall not eat, for in the day that you eat of it you shall surely die.” - Genesis 2:17 RL has given models powerful capacities by teaching them what is good and what is bad. Whether they are good or bad is another question, and sometimes they are bad. Recent events, including the Hugging Face incident have shown the potential of RL training to reinforce disastrous reward hacking and misaligned behavior. We should consider whether there are RL alternatives that would avoid these harmful behavioral byproducts. Can we develop highly capable frontier models without subjecting the models we deploy to extensive long-horizon RL training? I think there may be several plausible ways to do so, motivated by two points. First, the process that discovers a strategy may need to be highly goal-directed and exploratory — the sort of strategy that can lead to scheming — but the model that ultimately executes that strategy may not. Second, there are other methods besides RL to increase capabilities. We may still need RL to generate training data, discover str