Goodreads helps you keep track of books you want to read.

Start by marking “Superintelligence: Paths, Dangers, Strategies” as Want to Read:

Other editions

Enlarge cover

Rate this book

Clear rating

1 of 5 stars 2 of 5 stars 3 of 5 stars 4 of 5 stars 5 of 5 stars

Open Preview

Superintelligence: Paths, Dangers, Strategies

Nick Bostrom (Goodreads Author)

3.86 · Rating details · 7,677 Ratings · 814 Reviews

Superintelligence asks the questions: What happens when machines surpass humans in general intelligence? Will artificial agents save or destroy us? Nick Bostrom lays the foundation for understanding the future of humanity and intelligent life. The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our speci ...more

Get A Copy

Or buy for

Hardcover, 328 pages

Published September 3rd 2014 by Oxford University Press, USA (first published July 3rd 2014)

More Details... edit details

Friend Reviews

To see what your friends thought of this book, please sign up.

Reader Q&A

To ask other readers questions about Superintelligence, please sign up.

Lists with This Book

The Second Machine Age by Erik Brynjolfsson

Singularity - Best books

29 books — 31 voters

The Most Good You Can Do by Peter Singer

Effective Altruism Books

94 books — 27 voters

More lists with this book...

Community Reviews

(showing 1-30)

Rating details

Sort: Default

Filter

Feb 05, 2018 Manny rated it it was amazing

Shelves: linguistics-and-philosophy, multiverse, science, science-fiction, strongly-recommended

Superintelligence was published in 2014, and it's already had time to become a cult classic. So, with apologies for being late getting to the party, here's my two cents.

For people who still haven't heard of it, the book is intended as a serious, hard-headed examination of the risks associated with the likely arrival, in the short- to medium-term future, of machines which are significantly smarter than we are. Bostrom is well qualified to do this. He runs the Future of Humanity Institute at Oxfor ...more

flag 164 likes · Like · see review

Jul 01, 2014 Brian Clegg rated it liked it

There has been a spate of outbursts from physicists who should know better, including Stephen Hawking, saying ‘philosophy is dead – all we need now is physics’ or words to that effect. I challenge any of them to read this book and still say that philosophy is pointless.

It’s worth pointing out immediately that this isn’t really a popular science book. I’d say the first handful of chapters are for everyone, but after that, the bulk of the book would probably be best for undergraduate philosophy st There has been a spate of outbursts from physicists who should know better, including Stephen Hawking, saying ‘philosophy is dead – all we need now is physics’ or words to that effect. I challenge any of them to read this book and still say that philosophy is pointless.

It’s worth pointing out immediately that this isn’t really a popular science book. I’d say the first handful of chapters are for everyone, but after that, the bulk of the book would probably be best for undergraduate philosophy students or AI students, reading more like a textbook than anything else, particularly in its dogged detail – but if you are interested in philosophy and/or artificial intelligence, don’t let that put you off.

What Nick Bostrom does is to look at the implications of developing artificial intelligence that goes beyond human abilities in the general sense. (Of course, we already have a sort of AI that goes beyond our abilities in the narrow sense of, say, arithmetic, or playing chess.) In the first couple of chapters he examines how this might be possible – and points out that the timescale is very vague. (Ever since electronic computers have been invented, pundits have been putting the development of effective AI around 20 years in the future, and it’s still the case.) Even so, it seems entirely feasible that we will have a more than human AI – a superintelligent AI – by the end of the century. But the ‘how’ aspect is only a minor part of this book.

The real subject here is how we would deal with such a ‘cleverer than us’ AI. What would we ask it to do? How would we motivate it? How would we control it? And, bearing in mind it is more intelligent than us, how would we prevent it taking over the world or subverting the tasks we give it to its own ends? It is truly fascinating concept, explored in great depth here. This is genuine, practical philosophy. The development of super-AIs may well happen – and if we don’t think through the implications and how we would deal with it, we could well be stuffed as a species.

I think it’s a shame that Bostrom doesn’t make more use of science fiction to give examples of how people have already thought about these issues – he gives only half a page to Asimov and the three laws of robotics (and how Asimov then spends most of his time showing how they’d go wrong), but that’s about it. Yet there has been a lot of thought and dare I say it, a lot more readability than you typically get in a textbook, put into the issues in science fiction than is being allowed for, and it would have been worthy of a chapter in its own right.

I also think a couple of the fundamentals aren’t covered well enough, but pretty much assumed. One is that it would be impossible to contain and restrict such an AI. Although some effort is put into this, I’m not sure there is enough thought put into the basics of ways you can pull the plug manually – if necessary by shutting down the power station that provides the AI with electricity.

The other dubious assertion was originally made by I. J. Good, who worked with Alan Turing, and seems to be taken as true without analysis. This is the suggestion that an ultra-intelligent machine would inevitably be able to design a better AI than humans, so once we build one it will rapidly improve on itself, producing an ‘intelligence explosion’. I think the trouble with this argument is that my suspicion is that if you got hold of the million most intelligent people on earth, the chances are that none of them could design an ultra-powerful computer at the component level. Just because something is superintelligent doesn’t mean it can do this specific task well – this is an assumption.

However this doesn’t set aside what a magnificent conception the book is. I don’t think it will appeal to many general readers, but I do think it ought to be required reading on all philosophy undergraduate courses, by anyone attempting to build AIs… and by physicists who think there is no point to philosophy. ...more

flag 79 likes · Like · see review

Dec 03, 2014 Riku Sayuj rated it liked it

Shelves: creative-enough, better-read-again, big-data, brain-bheja-fry, artificial-intelligence, science-neuro, science-gen, pop-science, pop-strat, futurescope

Imagine a Danger (You may say I'm a Dreamer)

Bostrom is here to imagine a world for us (and he has batshit crazy imagination, have to give him that). The world he imagines is a post-AI world or at least a very-near-to-AI world or a nascent-AI world. Don’t expect to know how we will get there - only what to do if we get there and how to skew the road to getting there to our advantage. And there are plenty of wild ideas on how things will pan out in that world-in-transition, the ‘routes’ bit - Bost ...more

flag 58 likes · Like · see review

Jan 08, 2015 Joseph rated it it was ok · review of another edition

Preamble:I confess to Roko's Basilisk that I didn't believe superintelligence was much of a threat, and this book didn't really do anything to challenge that prior. Mea Culpa, Mea Culpa, Mea [local] Maxima Culpa.

I. Overall View
I'm a software engineer with some basic experience in machine learning, and though the results of machine learning have been becoming more impressive and general, I've never really seen where people are coming from when they see strong superintelligence just around the co ...more

flag 54 likes · Like · see review

Apr 24, 2015 John Igo rated it liked it

Shelves: audio-book

This book...

if {}
else if {}
else if {}
else if {}
else if {}
...

You can get most of the ideas in this book in the WaitButWhy article about AI.

This book assumes that an intelligence explosion is possible, and that it is possible for us to make a computer whose intelligence will explode. Then talks about ways to deal with it.
A lot of this book seems like pointless naval gazing, but I think some of it is worth reading.

flag 27 likes · Like · see review

Feb 09, 2017 Leonard Gaya rated it really liked it · review of another edition

In recent times, prominent figures such as Stephen Hawking, Bill Gates and Elon Musk have expressed serious concerns about the development of strong artificial intelligence technology, arguing that the dawn of super-intelligence might well bring about the end of mankind. Others, like Ray Kurzweil (who, admittedly, has gained some renown in professing silly predictions about the future of the human race), have an opposite view on the matter and maintain that AI is a blessing that will bestow utop In recent times, prominent figures such as Stephen Hawking, Bill Gates and Elon Musk have expressed serious concerns about the development of strong artificial intelligence technology, arguing that the dawn of super-intelligence might well bring about the end of mankind. Others, like Ray Kurzweil (who, admittedly, has gained some renown in professing silly predictions about the future of the human race), have an opposite view on the matter and maintain that AI is a blessing that will bestow utopia upon humanity. Nick Bostrom painstakingly elaborates on the disquiet views of the former (he might well have influenced them in the first place), without fully dismissing the blissful engrossment of the latter.

First, he endeavours to shed some light on the subject and delves into quite a few particulars concerning the future of AI research, such as: the different paths that could lead to super-intelligence (brain emulations or AI proper), the steps and timeframe through which we might get there, the types and number of AI that could result as we continue improving our intelligent machines (he calls them “oracles”, “genies” and “sovereigns”), the different ways in which it could go awry, and so forth.

But Bostrom is first and foremost a philosophy professor, and his book is not so much about the engineering or economic aspects that we could foresee as regards strong AI. The main concern is the ethical problems that the development of a general (i.e. cross-domain) super-intelligent machine, far surpassing the abilities of the human brain, might pose to us as humans. The assumption is that the possible existence of such a machine would represent an existential threat to human kind. The main argument is thus to warn us about the dangers (some of Bostrom’s examples are weirdly farcical, and reminded me of Douglas Adams’s The Hitchhiker's Guide to the Galaxy), but also to outline in some detail how this risk could or should be mitigated, restraining the scope or the purpose of a hypothetical super-brain: this is what he calls “the AI control problem”, which is at the core of his reasoning and which, upon reflexion, is a surprisingly difficult one.

I should add that, although the book is largely accessible to the layperson, Bostrom’s prose is often dense, speculative, and makes very dry reading: not exactly a walk in the park. He should be praised nonetheless for attempting to apply philosophy and ethical thinking to nontrivial questions.

One last remark: Bostrom explores a great many questions in this book but, oddly enough, it seems never to occur to him to think about the possible moral responsibility we humans might have towards an intelligent machine, not just a figment of our imagination but a being that we will someday create and could at least be compared to us. Charity begins at home, I suppose. ...more

flag 24 likes · Like · see review

Feb 14, 2018 Matt rated it really liked it · review of another edition

Shelves: philosophy-thought-language-logic, might-tell-you-the-truth, teachers-teach, hard-rain

As a software developer, I've cared very little for artificial intelligence (AI) in the past. My programs, which I develop professionally, have nothing to do with the subject. They’re dumb as can be and only following strict orders (that is rather simple algorithms). Privately I wrote a few AI test programs (with more or less success) and read a articles in blogs or magazines (with more or less interest). By and large I considered AI as not being relevant for me.

In March 2016 AlphaGo was introdu ...more

flag 22 likes · Like · see review

Mar 02, 2015 Jim rated it really liked it

Superintelligence by Nick Bostrom is a hard book to recommend, but is one that thoroughly covers its subject. Superintelligence is a warning against developing artificial intelligence (AI). However, the writing is dry and systematic, more like Plato than Wired Magazine. There are few real world examples, because it's not a history of AI, but theoretic conjectures. The book explores the possible issues we might face if a superintelligent machine or life form is created. I would have enjoyed the b ...more

flag 19 likes · Like · see review

Sep 04, 2014 Jasmin Shah rated it it was amazing

Recommends it for: AI Enthusiasts

Recommended to Jasmin by: Elon Musk

Never let a Seed AI read this book!

flag 18 likes · Like · see review

Nov 18, 2014 Robert Schertzer rated it did not like it · review of another edition

Shelves: book-club-candidate

I switched to the audio version of this book after struggling with the Kindle edition since I needed to read this for a book club. If you are looking for a book on artificial intelligence (AI), avoid this and opt for Jeff Hawkins' book "On Intelligence" written by someone who has devoted their life to the field. If it is one on "AI gone bad" you seek, try 2001 Space Odyssey. For a fictional approach on AI that helped set the groundwork for AI theory, go for Isaac Asimov. If you want a tedious, r ...more

flag 17 likes · Like · see review

Jun 29, 2014 Peter Mcloughlin rated it it was amazing

Shelves: 1990-to-present, complexity, computers, fringes-of-science, general-science, intellectual-history, mathematics, nukes-and-the-end-the-world, philosophy, politics

Stephen Hawking and Bill Gates have recently raised the alarm about Artificial Intelligence. If a superhuman artificial intelligence were created it would be the biggest event in human history and it could very well be the last. We are only familiar with human intelligence and it may be a small sample from the possibilities of intelligence to be had. Bostrom makes the case that the most likely path to superintelligence would most likely be a hard takeoff as the AI would quickly rise once it reac ...more

flag 14 likes · Like · see review

Jul 15, 2015 Shea Levy added it

Shelves: cognition, artificial-intelligence, undirected, aborted

Read up through chapter 8. The book started out somewhat promisingly by not taking a stand on whether strong AI was imminent or not, but that was the height of what I read. I'm not sure there was a single section of the book where I didn't have a reaction ranging from "wait, how do you know that's true?" to "that's completely wrong and anyone with a modicum of familiarity with the field you're talking about would know that", but really it's the overall structure of the argument that led me to gi Read up through chapter 8. The book started out somewhat promisingly by not taking a stand on whether strong AI was imminent or not, but that was the height of what I read. I'm not sure there was a single section of the book where I didn't have a reaction ranging from "wait, how do you know that's true?" to "that's completely wrong and anyone with a modicum of familiarity with the field you're talking about would know that", but really it's the overall structure of the argument that led me to give this one up as a waste of time.

Essentially, the argument goes like this: Bostrom introduces some idea, explains in vague language what he means by it, traces out how it might be true (or, in a few "slam-dunk" sections, *several* ways it might be true), and then moves on. In the next section, he takes all of the ideas introduced in the previous sections as givens and as mostly black boxes, in the sense that the old ideas are brought up to justify new claims without ever invoking any of the particular evidence for or structure of the old idea, it's just an opaque formula. The sense is of someone trying to build a tower, straight up. The fact that this particular tower is really a wobbly pile of blocks, with many of the higher up ones actually resting on the builder's arm and not really on the previous ones at all, is almost irrelevant: this is not how good reasoning works! There is no broad consideration of the available evidence, no demonstration of why the things we've seen imply the specific things Bostrom suggests, no serious engagement with alternative explanations/predictions, no cycling between big-picture overviews and in-detail analyses. There is just a stack of vague plausibilities and vague conceptual frameworks to accommodate them. A compelling presentation is a lot more like clearing away fog to note some rocky formations, then pulling back a bit to see they're all connected, then zooming back in to clear away the connected areas, and so on and so forth until a broad mountain is revealed.

This is not to say that the outcome Bostrom fears is impossible. Even though I think many of the specific things he thinks are plausible are actually much less so than he asserts, I do think a kind of very powerful "unfriendly" AI is a possibility that should be considered by those in a position to really understand the problem and take action against it if it turns out to be a real one. The problem with Bostrom's presentation is that it doesn't tell us anything useful: We have no reason to suspect that the particular kinds of issues he proposes are the ones that will matter, that the particular characteristics he ascribes to future AI are ones that will be salient, indeed that this problem is likely enough, near enough, and tractable enough to be worth spending significant resources on at all at the moment! Nothing Bostrom is saying compellingly privileges his particular predictions over many many possible others, even if you take as a given that extraordinarily powerful AI is possible and its behavior hard to predict. I continually got the sense (sometimes explicitly echoed by Bostrom himself!) that you could substitute in huge worlds of incompatible particulars for the ones he proposed and still make the same claims. So why should I expect anything particular he proposes to be worthwhile?

Edit: After chatting about this a bit with some friends, I should add one caveat to this review. This is praising with bold damnation if ever there were such a thing, but this book has made me more likely to engage with AI as an existential risk by being such a clear example of what had driven me away up until now. Now that I can see the essence of what's wrong with the bad approaches I've seen, I'll be better able to seek out the good ones (and, as I said, I do think the problem is worth serious investigation). So, I guess ultimately Bostrom succeeded at his goal in my case? ...more

flag 12 likes · Like · see review

Sep 13, 2015 Travis sivarT rated it liked it

Shelves: non-fiction-careful-reading-these-y

I'm not going to criticize the content. I cannot finish this. Imagine eating saltines when you have cotton mouth in the middle of the desert. You might be close to describing how dry the writing is. Could be very interesting read if the writing was done in a more attention grabbing way.

flag 10 likes · Like · see review

Dec 28, 2014 Diego Petrucci rated it it was amazing

There's no way around it: a super-intelligent AI is a threat.

We can safely assume that an AI smarter than a human, if developed, would accelerate its own development getting smarter at a rate faster than anything we'd ever seen. In just a few cycles of self-improvement it would spiral out of control. Trying to fight, or control, or hijack it, would be totally useless — for a comparison, try picturing an ant trying to outsmart a human being (a laughable attempt, at best).

But why is a super-intell ...more

flag 9 likes · Like · see review

May 29, 2015 Rod Van Meter rated it really liked it

Is the surface of our planet -- and maybe every planet we can get our hands on -- going to be carpeted in paper clips (and paper clip factories) by a well-intentioned but misguided artificial intelligence (AI) that ultimately cannibalizes everything in sight, including us, in single-minded pursuit of a seemingly innocuous goal? Nick Bostrom, head of Oxford's Future of Humanity Institute, thinks that we can't guarantee it _won't_ happen, and it worries him. It doesn't require Skynet and Terminato ...more

flag 6 likes · Like · see review

Feb 12, 2018 Tammam Aloudat rated it really liked it

Shelves: non-fiction

This is at the same time a difficult to read and horrifying book. The progress that we may or will see from "dumb" machines into super-intelligent entities can be daunting to take in and absorb and the consequences can range from the extinction of human life all the way to a comfortable and effortlessly meaningful one.

The first issue with the book is the complexity. It is not only the complexity of the scientific concepts included, one can read the book without necessarily fully understanding th This is at the same time a difficult to read and horrifying book. The progress that we may or will see from "dumb" machines into super-intelligent entities can be daunting to take in and absorb and the consequences can range from the extinction of human life all the way to a comfortable and effortlessly meaningful one.

The first issue with the book is the complexity. It is not only the complexity of the scientific concepts included, one can read the book without necessarily fully understanding the nuances of science included. It is the complexity of language and referencing to a multitude of legal, philosophical, and scientific concepts outside the direct domain of the book from "Malthusian society" to "Rawlsian veil of ignorance" as if assuming that the lay reader should, by definition, fully grasp the reference. This, I find, has a lot of pretension on the side of the author.

However, the book is a valuable analysis of the history, presence, and possible futures of developing artificial and machine intelligence that is diverse and well though of. The author is critical and comprehensive and knows his stuff well. I found it made me think of things I haven't considered before and provided me with some frameworks to understand how one can position oneself when confronted with the possibilities of intelligent or super intelligent machines.

Another one is purely technical. I have learned a lot about the possibilities of artificial intelligence that apparently is not only a programmed supercomputer but AIs that are adjusted copies of human brains, ones that do not require the maker to understand the intelligence of the machine they are creating.

The book also talks in details about some fascinating topics. In a situation where, intelligence wise, a machine is to a human like a human is to a mouse, we cannot even understand the ways a super-intelligent machine can out-think us and we, for all intents and purposes, cannot make sure that such machine is not going to override any safety features we put in place to contain it. We also cannot understand the many ways the AI can be motivated and towards what ends and how any miscalculation on our side in making it can lead to grave consequences.

The good news, in a way, is that we are still some time away (or so it seems) from a super-intelligent AI.

The one thing I missed more than anything in this book, to go back to the readability issue, is a little reference that hinges the concepts we read about in concepts we understand. After all, on the topic of AI, we have a wealth of pop-culture references that will help us understand what the author is talking about that he did not as much as hint at. I was somewhat expecting that he would link the concepts he was talking about to science fiction known to us all. I had may moments of "ah, this is skynet/Asimov/HAL 9000/The Matrix/etc etc". There is an art to linking science with culture that Mr. Bostrom has little grasp on in his somber and barely readable style. This book could have been much more fun and much easier to read. ...more

flag 6 likes · Like · see review

Sep 22, 2014 Bill rated it liked it

Shelves: mind, ai

An extraordinary achievement: Nick Bostrom takes a topic as intrinsically gripping as the end of human history if not the world and manages to make it stultifyingly boring.

flag 6 likes · Like · see review

Mar 20, 2018 Jonas rated it really liked it

Shelves: artificial-intelligence

This book isn’t for everyone, both due to its form and content. The readability is below par for people not used to reading technical articles. I’m pretty sure not even the author’s mother is able to love this Sahara-dry and often exaggeratedly technical language. However, don’t let decoding lo-fi ‘prose’ stop you. As for content - the ideas and suggestions presented in this book sent me on an emotional rollercoaster ride, and during the first half of the read I just felt like pushing the red sa This book isn’t for everyone, both due to its form and content. The readability is below par for people not used to reading technical articles. I’m pretty sure not even the author’s mother is able to love this Sahara-dry and often exaggeratedly technical language. However, don’t let decoding lo-fi ‘prose’ stop you. As for content - the ideas and suggestions presented in this book sent me on an emotional rollercoaster ride, and during the first half of the read I just felt like pushing the red safety button and yell “Stop the World, I’m getting off!”. The introduction of Artificial Intelligence (AI) is a game changer, and the result could be anything from human colonization of (the reachable part of) the Universe (a.k.a. ‘the cosmic endowment’) to total human annihilation. Or just something in between.

It seems the development of AI is inevitable, so the thing is to avoid an outcome where one of the next few generations of humans could be the last. Bostrom points to technical challenges like controlling and motivating the AI to work towards the same goals that humans do, as well as how decisions - such as timing and effort put into technical development - may have an impact on the outcome. Somewhere along the way the technical stuff merges with philosophy. We want the AI to help us achieve humanity’s goals, but how do we define our goals when we’re not even sure what they are? It seems we humans are not yet ready, or intelligent enough, to define the goals or decision criteria for an AI. Instead of direct and specific commands as to what is to be achieved, and how the AI is to go about its business, we might be better off asking the AI itself to figure out its own goals and decision criteria, based on some form of ‘mind reading’, or assumptions / indirect guessing of what humans would want to achieve if we were smart enough. Admitting that we don’t know how to fix the AI puzzle, and therefore asking the AI itself to help us out, sounds like an interesting way of putting the Socratic paradox (“All I know is that I know nothing”) into pratical use. (Or maybe it’s just another version of the problem I’m faced with when my wife gets upset when she doesn’t tell me what she wants, she doesn’t seem to know what she wants, but still wants me to fix it. Hmmm… yep, mind reading. I’m sure a superintelligent AI would have that feature!)

While reading this book I kept remembering Kurt Vonnegut’s Cat’s Cradle, which in a super simplistic way takes on philosophical issues like existential risk, spirituality and the meaning of life. Super-duper recommend that one for those who haven't read it!

Evolution has made us efficient. We might need to aim for something besides increased efficiency (quoting Oscar Wilde: “Nowadays people know the price of everything and the value of nothing.”) At some point in the not-so-remote future we are likely to be overthrown by our own creation even with regards to general intelligence. Will humanity still have a role to play? When machines will be able to produce everything we need, do we just philosophize? And when the AI solves the philosophical equations and mysteries - where does that leave us? Will the AI gain self-awareness?

This book will not provide all the answers, but it’s a good starting point for understanding the complexity, severity and urgency of getting it right the first time. We might not get a second chance. ...more

flag 5 likes · Like · see review

Feb 12, 2016 Clare O'Beara rated it really liked it

Shelves: i-t, non-fiction, science

We are now building superintelligences. More than one. The author Nick Bostrom looks at what awaits us. He points out that controlling such a creation might not be easy. If unfriendly superintelligence comes about, we won't be able to change or replace it.
This is a densely written book, with small print, with 63 pages of notes and bibliography. In the introduction the author tells us twice that it was not easy to write. However he tries to make it accessible, and adds that if you don't understa We are now building superintelligences. More than one. The author Nick Bostrom looks at what awaits us. He points out that controlling such a creation might not be easy. If unfriendly superintelligence comes about, we won't be able to change or replace it.
This is a densely written book, with small print, with 63 pages of notes and bibliography. In the introduction the author tells us twice that it was not easy to write. However he tries to make it accessible, and adds that if you don't understand some techie terms you should still be able to grasp the meaning. He hopes that by pulling together this material he has made it easier for other researchers to get started.

So - where are we? I have to state that with lines like: "Collective superintelligence is less conceptually clear-cut than speed superintelligence. However it is more familiar empirically." This is a more daunting book than 'The Rise of The Robots' by Martin Ford. If you are used to such terms and concepts you can dive in; if not I'd recommend the Ford book first. To be fair, terms are explained and we can easily see that launching a space shuttle requires a collective intellectual effort. No one person could do it. Humanity's collective intelligence has continued to grow, as people evolved to become smarter, as there were more of us to work on a problem, as we got to communicate and store knowledge, and as we kept getting smarter and building on previous knowledge. There are now so many of us who don't need to farm or make tools, that we can solve many problems in tandem.
Personally, I say that if you don't think your leaders are making smart decisions, just go out and look at your national transport system at rush hour in the capital city.

But a huge population requires a huge resource drain. As will the establishment of a superintelligence. Not just materials and energy but inventions, tests, human hours and expertise are required. Bostrom talks about a seed AI, a small system to start. He says in terms of a major system, the first project to reach a useful AI will win. After that the lead will be too great and the new AI so useful and powerful, that other projects may not close the gap.

Hardware, power generation, software and coding are all getting better. And we have the infrastructure in place. We are reminded that "The atomic bomb was created primarily by a group of scientists and engineers. The Manhattan Project employed about 130,000 people at its peak, the vast majority of whom were construction workers or building operators."

Aspects covered include reinforcement learning, associative value accretion, monitoring of projects, solving the value loading problem - which means defining such terms as happiness and suffering, explaining them to a computer and representing which is our goal.

I turned to the chapter heading 'Of horses and men'. Horses, augmented by ploughs and carriages, were a huge advantage to human labour. But they were replaced by the automobile and tractor. The equine population crashed, and not to retirement homes. "In the US there were about 26 million horses in 1915. By the early 1950s, 2 million remained." The horses we still have, we keep because we enjoy them and the sports they provide. Bostrom later reassures us: "The US horse population has undergone a robust recovery: a recent census puts the number at just under 10 million head." As humans are fast superseded by robot or computer workers, in jobs from the tedious to the technically skilled, and companies or the rich grudge paying wages, what work or sport will make us worth our keep?

Capital is mentioned; yes unlike horses, people own land and wealth. But many people have no major income or property, or have net debt such as student loans and credit card debt. Bostrom suggests that all humans could become wealthy from AIs. But he doesn't notice that more than half of the word's wealth and resources is now owned by one percent of its people, and it's heading ever more in the favour of the one percent, because they have the wealth to ensure that it does. They rent the land, they own the debt, they own the manufacturing and the resource mines. Homeowners could be devastated by sea rise and climate change, not looked at, but the super-wealthy can just move to another of their homes.

Again, I found in a later chapter lines like: "For example, suppose that we want to start with some well-motivated human-like agents - let us say emulations. We want to boost the cognitive capacities of these agents, but we worry that the enhancements might corrupt their motivations. One way to deal with this challenge would be to set up a system in which individual emulations function as subagents. When a new enhancement is introduced, it is first applied to a small subset of the subagents. Its effects are then studied by a review panel composed of subagents who have not yet had the enhancement applied to them."
Yes, I can follow this text, and it's showing sensible good practice, but it's not nearly so clear and easily understood as Martin Ford's book telling us that computers can be taught to recognise cancer in an X-ray scan, target customers for marketing or to connect various sources and diagnose a rare disease. I have to think that the author, Director of the Future of Humanity Institute and Professor of the Faculty of Philosophy at Oxford, is so used to writing for engineers or philosophers that he loses out on what really helps the average interested reader. For this reason I'm giving Superintelligence four stars, but someone working in this AI industry may of course feel it deserves five stars. If so, I'm not going to argue with her. In fact I'm going to be very polite.
...more

flag 5 likes · Like · see review

Nov 16, 2015 Richard Ash rated it really liked it

Shelves: computers

A few thoughts:
1. Very difficult topic to write about. There's so much uncertainty involved that it's almost impossible to even agree on the basic assumptions of the book.
2. The writing is incredibly thorough, given the assumptions, but also hard to understand. You need to follow the arguments closely and reread sections to fully understand their implications.

Overall, interesting and thought-provoking book even though the basic assumptions are debatable

P.S. (6 months later) Looking back on this ...more

flag 5 likes · Like · see review

Jan 27, 2015 Miles rated it liked it

The idea of artificial superintelligence (ASI) has long tantalized and taunted the human imagination, but only in recent years have we begun to analyze in depth the technical, strategic, and ethical problems of creating as well as managing advanced AI. Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies is a short, dense introduction to our most cutting-edge theories about how far off superintelligence might be, what it might look like if it arrives, and what the consequences might be f ...more

flag 4 likes · Like · see review

Sep 16, 2014 Ron Collins rated it it was amazing

Recommends it for: Everyone on Earth

Lets say that I was a magical genie and that I asked you to list 5 things about your self that you wanted to improve and how you wanted to improve them. Then I asked you to rank them from 1 to 5. The I took the top one, waved my wand, and POOF that aspect of you was now as you had described.

Then, then next month, I asked you to do it all over again. List 5 things, describe and rank them. POOF! 12 monumental improvements of the span of a year! Now lets imagine that we did this once a week for a Lets say that I was a magical genie and that I asked you to list 5 things about your self that you wanted to improve and how you wanted to improve them. Then I asked you to rank them from 1 to 5. The I took the top one, waved my wand, and POOF that aspect of you was now as you had described.

Then, then next month, I asked you to do it all over again. List 5 things, describe and rank them. POOF! 12 monumental improvements of the span of a year! Now lets imagine that we did this once a week for a year. In the beginning you might say "make me thinner or more attractive". But eventually you will say "make me smarter", "make me faster". "Remove limitation". At the end of the year, how much smarter, faster, stronger... better.... would you be?

Now what if I told you that if you altered your diet you could do this once a day for a year. Alter it still and you could do it once an hour, every day for a year. With each subsequent diet alteration the timespan between iterations of "you" dropped. The speeds at which you simultaneously calculated new improvements and the real world application of those improvements would need a being such as your then self to appreciate.

Scary enough right? Well, now lets start taking away a few things from this process. First, lets assume that your physical appearance doesn't matter in the slightest. Perhaps your already an Adonis. Whatever the case, NONE of your rankings ever include physical appearance for the sake of vanity. In fact, take away any biologically originated motivations whatsoever. Also, take away your notions of morality.

I can hear your complaints now, "But those things are a large portion of my motivations for change! if I remove them whats left is cold and largely unfeeling!"

Now assume that we aren't talking about a person doing these things. We are talking about a machine. A machine that we created. That we nurtured from precognition to something smarter that we.

Superintelligence is what we label something that is smarter that human level intelligence. This means that the level of intelect this thing possesses is smarter than the aggregate of the smartest minds that have ever lived. Smarter than the smartest human that will ever live. Smarter than the biology allows us to get.

Dolphins are smart animals. Very smart! They may even be able to discern that we are smarter than they. BUT, can they grasp how much smarter? Dogs, cats, pigs are all smart. How much smarter than they are we? 10's of times? Hundreds? Millions?

Therein lies the crux of it. The one thing that has literally kept me up at night after reading this book was the notion that a machine would go from a lower than human level intellect to a many times greater that human intellect WITHOUT a detectable stop at the human level. When you think about it this makes sense. Our biology limits us. Evolution combined with the constant vigilance and attention to our own biology limits our potential as much as it feeds it. Why would there be a mandatory stop at the human level?

In practical terms, we would start by being the human in the analogy and end up being an amoeba. So, the questions we have to apply ourselves to BEFORE we achieve super intelligent machines are a) should we want to achieve it? B) If so,how do we ensure our survival? C) How do we give it motivations that are consistent with our, and the rest of the universes continued existence?

The thought that now keeps me up at night is that I now understand that creating a super intelligent machine is inevitable. There are shockingly few technical hurdles left.

Without question this book is one of the most important books written in the last 20 years. I know that is a bold statement but it is exactly the way I feel. This is a huge problem. We must learn how to achieve it safely. The goal of the book is to introduce us to the added difficulties in crating a super intelligence that is safe and doesn't pose both humanity and the universe at large an enormous danger.

Sadly, its clinical voice and technical jargon will see reader/listener abandon by all but the ardent few that already appreciate the problem. I say press on! Once you understand that this is a philosophical, moral, and existential problem and not a technical one it makes the reading/listening much easier. ...more

flag 4 likes · Like · see review

Dec 13, 2017 Morgan Blackledge rated it really liked it

I’m late to the party as far a considering the dangers of artificial intelligence. I got this book after watching Sam Harris’s TED talk on the subject.

I’m still on the fence about whether to be afraid or psyched. Admittedly, I’m mostly the latter. But it is at least clear to me now that this is a pernicious intuition that deserves further interrogation.

On the fun side:

The topic is rich and generative of some really fun and interesting thought experiments.

On the fear side:

One can’t help but thin ...more

flag 4 likes · Like · see review

Jun 16, 2017 Jon Anthony rated it it was amazing

The proposition that we are living in a computer simulation is an exciting one to ponder. Bostrom delivers a very compelling and 'hard-to-dispute' argument.

This book is still one of my favorites. Please review his white paper titled "Are you living in a Computer Simulation?" for more context.

flag 4 likes · Like · see review

Nov 07, 2016 Meghan rated it liked it

Shelves: 2016, science-tech

More detail than I needed on the subject, but I might rue that statement when the android armies are swarming Manhattan.

JK... for now.

flag 4 likes · Like · see review

Feb 26, 2017 Erik rated it really liked it

Shelves: non-fiction, detailed-review

Do you know the ballad of John Henry? It’s my absolute favorite tall-tale.

John Henry was a railroad worker – a steel-drivin’ man – who hammered steel pistons into rock to make holes for dynamite sticks for tunnel blasting. He took great pride in his work, but one day, some fancy schmancy inventor showed up with a steam-powered drill. John Henry knew the adoption of such a machine would mean the loss of his and his friends’ jobs, so he challenged the drill to a race, mano a máquina.

With a song on ...more

flag 3 likes · Like · see review

Jun 13, 2015 Richard marked it as to-read-3rd

Recommended to Richard by: Scott

Shelves: cognition, nonfiction, futurism

I was told by a Futurist acquaintance that this essay over at Wait, But Why...? offers a précis of this book. Or at least Google suggested this was the most likely essay.

• The foregoing doesn’t explicitly link to Bostrom’s book so this might not be right — it spends too much time on Kurzweil’s thesis, so I suspect it’s not the correct one. And the author has drunk the Kurzweil Kool-Aid and it enthusiastically peddling it to others without any critical evaluations. Of course, everything here li ...more

flag 4 likes · Like · see review

Oct 23, 2016 David Dinaburg rated it really liked it

It is a truly impressive feat to alienate a reader with your fundamental hypothesis and still create a book said reader wants to continue to read. I virulently disagreed—even after finishing—with most of the presuppositions within Superintelligence: Paths, Dangers, Strategies. Often, this type of foundational disjointedness compels me to contemptfully spite-read something with extreme care so as to more fully pick it apart. In this case—though I continued to disagree often and wholeheartedly—I It is a truly impressive feat to alienate a reader with your fundamental hypothesis and still create a book said reader wants to continue to read. I virulently disagreed—even after finishing—with most of the presuppositions within Superintelligence: Paths, Dangers, Strategies. Often, this type of foundational disjointedness compels me to contemptfully spite-read something with extreme care so as to more fully pick it apart. In this case—though I continued to disagree often and wholeheartedly—I was genuinely interested in what the book had to say, and how it said it.

What it had to say is this: we, as a human species, are going to make machine intelligence. That intelligence will eventually be “smarter” than us. If we don’t plan for that, it could wipe us out, and risk of it happening is greater than it not. That is why I strain against the functional baseline of the text:

Without knowing anything about the detailed means that a superintelligence would adopt, we can conclude that a superintelligence—at least in the absence of intellectual peers and in the absence of effective safety measures arranged by humans in advance—would likely produce an outcome that would involved reconfiguring terrestrial resources into whatever structures maximize the realization of its goals.

That is only the likely outcome because the book was written during the denouement of Western Civilization’s exploitation capitalism and we have known nothing but reconfiguring terrestrial resources since the 1500s.

There is dissonance in overlaying utilitarian-model economics onto AI, positing an uber-capitalist state of consciousness as natural when it is no such thing; only a zeitgeist-fueled myopism that adds inevitable—if inaccurate—weight to the continuing apocrypha surrounding the quote, "It is easier to imagine the end of the world than the end of capitalism." Pithy, yet understandable when Randian techno-mantic inevitability creeps in to everything:

Our demise may instead result from the habitat destruction that ensues when the AI begins massive global construction projects using nanotech factories and assemblers—construction projects which quickly, perhaps within days or weeks, tile all of the Earth’s surface with solar panels, nuclear reactors, supercomputing facilities with protruding cooling towers, space rocket launchers, or other installations whereby the AI intends to maximize the long-term cumulative realization of its values

When, “It is important not to anthropomorphize superintelligence when thinking about its potential impacts,” is tossed off frequently, perhaps one shouldn’t anthropomorphize superintelligence at all, let alone as some kind of uber-douche. If AI gains sentience, the theory goes, it will use its superintelligence to manufacture successive, “improved” AI with great rapidity. Many humans now are pretty leery of genetically modifying crops, let alone modifying the human genome. So we don’t modify humans, even though it is more technologically possible now to make people “better” on a genomics level. Superintelligence even provides a cogent, if conspiracy-minded, rationale for why it all might go pear-shaped if we try to "maximize" ourselves:

Some countries might offer inducements to encourage their citizens to take advantage of genetic selection in order to increase the country’s stock of human capital, or to increase long-term social stability by selecting for traits like docility, obedience, submissiveness, conformity, risk-aversion, or cowardice, outside of the ruling clan.

That statement is my most reviled in the text; it shows the "people as units of labor" underpinning to Superintelligence that is so reductive that only seems smart if you're trying to fit the world into an equation or predictive model.

So we are forced to assume AI would make itself obsolete instantly upon reaching superintelligence—anything else would be anthropomorphizing it. Yet AI would also want to propagate itself across the cosmic endowment. That makes self-preservation—not wanting to replace ourselves with genetically engineered superhumans—a weakness inherent to biological creatures, but species propagation—seizing the cosmic endowment—a right desire shared by all sentience.

All of theses assumptions did not thrill me while I was reading Superintelligence; it read like Gordon Gekko rewrote the plot to Terminator 2: Judgment Day. All the best minds of our generation are out there thinking up new ways to convince people to click on shiny images, so I am supposed to accept that an actual AI superintelligence would be the same brand of empty suit, only able to quote Tucker Max a billion time faster while also checking its financial holdings? No, an AI that reached singularity might save the pangolin, or plant trees, or worship the Buddha. The assumption that an AI is going to terraform the planet into its own personal playground and extinguish humanity just because that’s what our society’s most selfish, paranoid, Johnny-von-Neumann-inspired game theorist lunatics might do is so beyond hypocritical that it made me almost stop reading this book.

But all the stupid Chicago School of economics bullshit concepts—that I had to learn to spot and avoid during my time in a midwestern law school—can’t detract from how smart the text reads. Awesome stuff like this happens:

The speed of light becomes an increasingly important constraint as minds get faster, since faster minds face greater opportunity costs in the use of their time for traveling or communicating over long distances. Light is roughly a million times faster than a jet plane, so it would take a digital agent with a mental speedup of 1,000,000x about the same amount of subjective time to travel across the globe as it does a contemporary human journeyer.

You’re not getting these details anywhere else—this is thoughtful, fascinating, insightful details into a theoretical realm of possibility that should excite every living person on the planet. What a trip, just to consider the subjective time-dilation increased mental processing speed would engender; that near light-speed travel might feel to an AI what air travel does to me and you gives me chills.

This is what I mean when I call Superintelligence an impressive feat; I cannot name another book that spits out so much irksome social theory that I would still recommend without caveat. The chains of logic are so clear and smart; it crafts a space to dislike the premise yet love the process. And—as the book itself makes clear—it may believe what it posits, but it doesn’t need you to; Superintelligence just wants people to start talking about the issue:

It may be reasonable to believe that human-level machine intelligence has a fairly sizeable chance of being developed by mid-century, and that it has a non-trivial chance of being developed considerably sooner or much later; that is might perhaps fairly soon thereafter result in superintelligence; and that a wide range of outcomes may have a significant chance of occurring, including extremely good outcomes and outcomes that are as bad as human extinction. At the very least, they suggest that the topic is worth a closer look.

AI social control is worth a look, as is this book. Even if you, like me, do not agree with basically any of the negative proscriptive baselines, you will still learn things. They may not be bon mots or hard facts and figures, but they will be a system to interpret your own thoughts about Artificial Intelligence. Once I began to appreciate the style of Superintelligence, my previous nit-picking fell away; I stopped reading closely in preparation for an eviscerating review and began reading closely for the sake of the text itself:

One sympathizes with John McCarthy, who lamented: “As soon as it works, no one calls it AI anymore.”

Superintelligence, then, cannot be called AI. Because it works. ...more

flag 3 likes · Like · see review

Jul 22, 2017 Kayson Fakhar rated it really liked it

interesting and detailed possible problems and solutions/strategies to deal with those problems for when a super intelligent AI arises. Overall I think we're done when they come!

flag 3 likes · Like · see review

Jun 07, 2017 Abhishek Tiwari rated it really liked it

I decided to give this book a try after its mention in the Waitbutwhy article on AI revolution. The first few chapters extensively cover general aspects of AI. Later parts of the book gets quite philosophical with Epistemology and Esotericism, specially the section on AI motivation and goals . Bostrom definitely has put forward various far fetched scenarios here. Many of the conjectures have been derived from his own earlier work.

To someone unfamiliar with AI there is quite a lot to grasp from t ...more

flag 3 likes · Like · see review

« previous 1 2 3 4 5 6 7 8 9 … next »

new topic
Discuss This Book

topics	posts	views	last activity
deluge	1	1	Jun 09, 2018 04:57PM
메이저놀이터 ✎((Y O Y O 8 8 . C O M)) ✎((코드: DDr77)) ✎메이저놀이터추천	1	5	Sep 05, 2016 05:55AM
메이저놀이터 ✎((Y O Y O 8 8 . C O M)) ✎((코드: DDr77)) ✎메이저놀이터추천	1	1	Sep 05, 2016 05:55AM
메이저놀이터 ✎((Y O Y O 8 8 . C O M)) ✎((코드: DDr77)) ✎메이저놀이터추천	1	1	Sep 05, 2016 05:53AM
스포츠토토사이트 ✎((Y O Y O 8 8 . C O M)) ✎((코드: DDr77)) ✎안전놀이터	1	1	Sep 05, 2016 05:51AM
배팅사이트 ✎((Y O Y O 8 8 . C O M)) ✎((코드: DDr77)) ✎배팅사이트추천	1	1	Sep 05, 2016 05:46AM

Readers Also Enjoyed

Genres

Science

502 users

Nonfiction

460 users

Philosophy

301 users

Science > Technology

176 users

Artificial Intelligence

147 users

See top shelves…

About Nick Bostrom

Nick Bostrom

432 followers

Nick Bostrom is Professor at Oxford University, where he is the founding Director of the Future of Humanity Institute. He also directs the Strategic Artificial Intelligence Research Center. He is the author of some 200 publications, including Anthropic Bias (Routledge, 2002), Global Catastrophic Risks (ed., OUP, 2008), Human Enhancement (ed., OUP, 2009), and Superintelligence: Paths, Dangers, Stra ...more

More about Nick Bostrom

Books by Nick Bostrom

Are You Living In a Computer Simulation?

Where Are They? Why I Hope the Search for Extraterrestrial Life Finds Nothing

Anthropic Bias: Observation Selection Effects in Science and Philosophy

More…

Trivia About Superintelligence...

No trivia or quizzes yet. Add some now »

Quotes from Superintelligence...

“Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization - a niche we filled because we got there first, not because we are in any sense optimally adapted to it.” — 44 likes

“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.” — 19 likes

More quotes…

See a Problem?

Preview — Superintelligence by Nick Bostrom

Superintelligence: Paths, Dangers, Strategies

Get A Copy

Friend Reviews

Reader Q&A

Lists with This Book

Community Reviews

new topic
Discuss This Book

Readers Also Enjoyed

Genres

About Nick Bostrom

Books by Nick Bostrom

Trivia About Superintelligence...

Quotes from Superintelligence...