Review: Superintelligence: Paths, Dangers, Strategies

The danger of superintelligence boils down to "Control Problem." Most things that are dangerous do. We have to figure out how to control asteroids. We have to figure out how to control nations so that they don't bomb each other into a humanity-ending nuclear war. We have to control our cars so that we don't die. Control is important.

But, most of the time, when you try to control something the thing doesn't respond, "What the fuck are you talking about, dude?" And an AI asks that question anytime you set out trying to make it. Indeed, every philosophical question that can be posed (and is generally worth being posed) usually involves a huge amount of semantics and thats a problem. Humans mean different things. We miscommunicate. We aint talk so gerd. This is a problem when a superintelligence with the computing power of God is trying to figure out (a) what you want it to do and (b) what you want it to not do (see: kill you/torture you forever). The danger of Superintelligence then, is the danger of creating a Jackass Genie- that Monkey's Paw comes to life- and that we ask a smart computer to do things that it interprets incorrectly.

The theme is "Be Careful What You Wish For".