Suddenly, Unclean

NB: This is my first post on LW as a longtime lurker - I hope I'm not stepping on any norms. I prioritized speedposting for relevance rather than polish. Cammarata's Neural Networks Are Already Clean has been discussed here already ( linkpost , pushback on their version of explainability ), but I have not yet seen anyone argue the fact that there will be a discontinuity in cleanliness, ie: the [single-neuron / single concept] pair coverage as a useful metric. We have three quantities of interest: network capacity somewhat analogous to number of neurons here number of human understandable names and sentences for concepts number of concepts encoded in an LLM by backprop Cammarata says that as goes up, the compression of into goes down and neurons become interpretable because . But: is bounded, keeps going up, and post-training pushes . Ergo, there will be a situation soon enough. And backprop leaves no free lunch uneaten: the neurons either will become redundant over human concepts, or there will appear ever more cross terms unbounded in their complexity. The small networks use compositionality and superposition as a workaround for lack of capacity, but optimization will apply pressu