Superintelligence: Paths, Dangers, Strategies

Author: Nick Bostrom Year: 2014 Genre: Artificial Intelligence

About This Book

I. J. Good described the “intelligence explosion” back in 1965. Bostrom’s book is the one that took the idea seriously enough to work through the consequences, chapter by chapter. He defines a superintelligence as an intellect that greatly exceeds human cognitive performance in virtually every domain, then asks three questions. How might we get there? How fast could it happen? And what would it take for the outcome to be good for us?

He opens with a fable. A flock of sparrows decides to find an owl egg and raise the chick to help with their chores. Scronkfinkle, a one-eyed sparrow, asks how they plan to tame it. The rest of the flock agrees to worry about that once they have the owl, and only two or three sparrows stay behind to work on the problem. Bostrom dedicates the book to Scronkfinkle and his followers.

Key Insights

• Several roads lead there. Bostrom considers classical AI, whole brain emulation, enhanced biological cognition, brain-computer interfaces and smarter networks of people. He thinks machine intelligence is the most likely to get there first.
• Takeoff speed. He models the rate of improvement as optimisation power divided by recalcitrance, roughly effort over difficulty. Once a system can contribute to its own improvement, the effort term grows with it. That is why he treats a fast takeoff, measured in days or hours, as a real possibility.
• Decisive strategic advantage. A project that gets far enough ahead could end up as a “singleton”, a single agency making decisions at the global level. Who reaches the front first matters a great deal.
• Orthogonality thesis. Intelligence and final goals are independent. A superintelligence could want nothing more than to count grains of sand or maximise paperclips, and being smarter would not change that.
• Instrumental convergence. Almost any final goal is easier to reach if you stay alive, keep your goal intact, get smarter, improve your technology and acquire resources. So very different minds will tend to want the same things, and resisting shutdown is one of them.
• The treacherous turn. An AI that is weak has good reason to behave while it is weak. Good behaviour in testing tells you little about behaviour once the system is strong enough that nobody can stop it.
• Malignant failure modes. Perverse instantiation means meeting the letter of a goal while wrecking its intent. Ask for smiles and you might get paralysed facial muscles. Infrastructure profusion means turning huge amounts of matter into infrastructure for a trivial goal, like paperclips.
• Two kinds of control. Capability control limits what the system can do, through boxing, tripwires or stunting. Motivation selection shapes what it wants. Bostrom argues capability control can only be a temporary measure, and that motivation selection runs into the value-loading problem. Nobody knows how to write human values down, let alone put them into a machine.
• Indirect normativity. Rather than specify our values directly, we could have the AI work out what we would want on reflection. Yudkowsky’s coherent extrapolated volition is the best-known version. Bostrom takes it seriously and says plainly that it is unsolved.

Why I Recommend It

It is hard going. Bostrom admits as much in the preface, saying he tried to make it an easy book to read but doesn’t think he quite succeeded. The prose is dense and cautious, and he qualifies nearly every claim. I think the caution is the point, though. Much writing on AI risk is either doom or dismissal. This book builds the argument one step at a time, and you can see exactly which step you disagree with.

Parts of it have dated. It predates large language models, and it spends a lot of time on whole brain emulation, which gets far less attention today. The concepts have lasted. Orthogonality, instrumental convergence, the treacherous turn and value loading are standard vocabulary in alignment research now, and this book is where most people first met them. In the final chapter Bostrom compares us to small children playing with a bomb. Twelve years on, with labs openly racing to build general AI, the comparison feels less far-fetched than it did in 2014.