The Hallucinated Superagent

Written as a MATS extension fellow, mentored by Richard Ngo. [1] Agents manage their beliefs differently to non-agentic predictors. A key example of agent-specific belief management is hyperstition, which is a particular case of goal-models . Based on the idea that belief-management is an important aspect of agency, here's a (vibe) attempt at a rationality criterion: An agent's beliefs are rational when they are a fixed point under the agent's own belief operator. B = U(B). Here, U is an update (interpreted as the agent choosing to execute a belief update computation—or perhaps being forced to by a new observation). This definition puts a huge load on the belief operator to be interesting. But I think it gives a clean conceptual separation between the instantaneous beliefs—which aren't that important—and the processes for updating them—which are. This rationality criterion falls apart for embedded agents. Such an agent can only update in its own world, and that "act of doing inference" itself modifies the agent and its beliefs inside its world. Consider an amended rationality criterion for embedded agents: An agent's beliefs are (robustly) rational when they are a (robustly) near-f