Intuitions towards a theory of agent foundations

Epistemic status: Rough vibe-y ideas written under time pressure because otherwise I wouldn't write anything; I'm a programming language and p2p protocol developer falling down multiple convergently-evolving rabbit-shaped holes. Questions What is an agent? How do we think meaningfully about how cells or humans or societies maintain themselves and are changed by their environment? How do we think about the patterns where historically lineages of agents have gone extinct... to hopefully avoid this, or if it is inevitable, to delay or come to terms with it? This is the questions posed to the field of agent foundations. What would a theory that tells us this look like? Or better question, different theories (?) It would have to somehow take observational data on a particular agent and environment and make some prediction of how that agent adapts to its environment. One potential RSI-relevant example here might be "take data on the informational dynamics of LLM training and use and predict how it will improve itself". This seems hopeless on its face, but there are some meaningful patterns. The most hopeful of which is on highlight in Scott's The Goddess of Everything Else , which observ