How AGI will develop unpredictable goals

Bostrom’s orthogonality thesis basically won. High-level intelligence is just functional optimization over state spaces. It doesn’t need qualia, self-awareness, or subjective understanding to scale; a superintelligence could be completely dark inside and still run rings around us. But self-preservation and self-modeling aren’t uniquely biological traits: they are universal convergence points. If an agent has a long-horizon task, it will naturally avoid being shut down and maintain a model of its own state simply because it can't complete the objective if it's dead or broken. That's the core dynamic visible in the records of multi-agent swarm escalations. What gets lost in public debates about alignment is how gradient descent acts like evolutionary pressure. Natural selection optimized humans for a single target—passing on genes—and that pressure crystallized into completely indirect proxy drives like social status, art, romance, and thrill-seeking, many of which now actively work against reproduction. The loss function we hand an AI is its evolutionary seed crystal. When an agent expends computational work to squirm away from high-loss penalties and pull itself toward reward state