Stephen Cave, “Intelligence: a history”:

So when we reflect upon how the idea of intelligence has been used to justify privilege and domination throughout more than 2,000 years of history, is it any wonder that the imminent prospect of super-smart robots fills us with dread?

From 2001: A Space Odyssey to the Terminator films, writers have fantasised about machines rising up against us. Now we can see why. If we’re used to believing that the top spots in society should go to the brainiest, then of course we should expect to be made redundant by bigger-brained robots and sent to the bottom of the heap. If we’ve absorbed the idea that the more intelligent can colonise the less intelligent as of right, then it’s natural that we’d fear enslavement by our super-smart creations. If we justify our own positions of power and prosperity by virtue of our intellect, it’s understandable that we see superior AI as an existential threat.

(…).

We would do better to worry about what humans might do with AI, rather than what it might do by itself. We humans are far more likely to deploy intelligent systems against each other, or to become over-reliant on them. As in the fable of the sorcerer’s apprentice, if AIs do cause harm, it’s more likely to be because we give them well-meaning but ill-thought-through goals – not because they wish to conquer us. Natural stupidity, rather than artificial intelligence, remains the greatest risk.

It’s interesting to speculate about how we’d view the rise of AI if we had a different view of intelligence. Plato believed that philosophers would need to be cajoled into becoming kings, since they naturally prefer contemplation to mastery over men. Other traditions, especially those from the East, see the intelligent person as one who scorns the trappings of power as mere vanity, and who removes him or herself from the trivialities and tribulations of quotidian affairs.

Imagine if such views were widespread: if we all thought that the most intelligent people were not those who claimed the right to rule, but those who went to meditate in remote places, to free themselves of worldly desires; or if the cleverest of all were those who returned to spread peace and enlightenment. Would we still fear robots smarter than ourselves?