A Circularity at the Heart of Safety Assurance

A friend of yours says they've had a flash of insight. A manga they've been into lately has helped them understand something deep and fundamental about decision theory. You aren't yet clear on the details when they rush off to work on "practical applications", but before long they return with their magnum opus : "This hard drive contains the weights of an AI model that is both wildly superintelligent and completely safe!" You feel as though you have good reasons to be pretty skeptical of this claim. This friend has terrible taste in manga, and you aren't convinced that they grasp the more subtle themes in the stuff they do read. You're about to shoo them away when, anticipating a dismissal, your friend pleads: "What would convince you that I'm right?" Now, if this friend wanted to convince you that they were wearing the Crunchyroll Store exclusive Solo Leveling Beru Tee, you could verify that claim directly, with the unaided senses. It's as simple as taking a look at their outfit. Assessing the safety of the AI model is not so simple. Observations of the hard drive are separated from the fact you want to know by a gulf of empirical and logical uncertainty. Whether or not the model