Towards an Architecture-Agnostic Definition of "Attention"

This brief post is meant to inspire and to start an open dialogue. I am genuinely curious to listen and incorporate your opinions, challenges, and insights. The Context I have been pondering for some time about the fact that Multi-Layer Perceptrons are known to be universal approximators and good learners, and yet, they are clearly not as useful as Transformers. The general consensus is that it's mostly due to performance limits : that is definitely true, but the explanation never felt satisfactory to me. Intuition suggests that self-attention enables contextual-based learning , as opposed to memorization-based learning . Until some days ago, I couldn't point my finger to what "contextual" really meant in practice. The eye opener was this paper, The Bayesian Geometry of Transformer Attention : I am not going to discuss it since that is beyond my competence, but one of its merits is in posing the correct question: can pure in-context learning be identified at all? The authors succeeded in exactly that: they brainstormed a tool called Bayesian wind tunnels , in which learning is possible, but memorization is prohibitively expensive and doesn't improve the learning process. And they p