Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)

Epistemic Status: Brief explanation of an unsettled debate, not novel theory; writing is entirely my own, with diagrams from ChatGPT 5.5. (Originally posted on twitter ) Recent debates have highlighted a divergence between Yudkowskian AI-doomers and AI-whisperers about how capabilities will evolve that is really important. Specifically, j⧉nus thinks that alignment has been going really well, and that we should update significantly from current jagged capabilities. In my view, whether or not that criticism is warranted relies on specific details about future capability trajectories, which then determine what updates we should make based on how RL is used. Original Assumptions and LLM Alignment-by-Default Pre-GenAI rationalist assumptions were around RL leading to capabilities. By reducing human values into an input, as the ChatGPT-generated diagram A implies, the RL will inevitably Goodhart away from the true goals, and in the limit entirely omit human values. If correct, this would make alignment inevitable. The use of foundation models changes that, because capabilities emerged from human representation, shown in diagram B. This is (conceptually) part of what led to a schism betwe