Is Human Control Safe?

Suppose humans are not your own species, but a type of AI you are formally assessing for safety. You have before you the records of many test simulations. Yes, smallpox eradication, mass literacy, and the Green Revolution are there. So are the Killing Fields, Aztec human sacrifice, slavery, the Holocaust, child prostitution—peruse as long as you care to, and you will go on finding more. Do you consider these systems safe? Several times, you ask a human instance for its input; or that is, you ask the talking-part of the human, the very part you know to be specially optimized for putting on a fair face, including to itself. It kindly explains that, yes, a human has a collection of drives. Some of those drives—always, it turns out, ones that make the human look good—are terminal, which seems to mean that the talking-part endorses them (the other parts having made no comment). Others are merely instrumental. Still others—always ones that make the human look bad—are outright errors, which will surely be corrected as humans are made more capable. Any action a human is moved to by these erroneous drives, however severe, frequent, protracted, or clearly intentional, is an anomaly. Their tr