What do you think?
Rate this book
If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All
"May prove to be the most important book of our time.”—Tim Urban, Wait But Why
The scramble to create superhuman AI has put us on the path to extinction—but it’s not too late to change course, as two of the field’s earliest researchers explain in this clarion call for humanity.
In 2023, hundreds of AI luminaries signed an open letter warning that artificial intelligence poses a serious risk of human extinction. Since then, the AI race has only intensified. Companies and countries are rushing to build machines that will be smarter than any person. And the world is devastatingly unprepared for what would come next.
For decades, two signatories of that letter—Eliezer Yudkowsky and Nate Soares—have studied how smarter-than-human intelligences will think, behave, and pursue their objectives. Their research says that sufficiently smart AIs will develop goals of their own that put them in conflict with us—and that if it comes to conflict, an artificial superintelligence would crush us. The contest wouldn’t even be close.
How could a machine superintelligence wipe out our entire species? Why would it want to? Would it want anything at all? In this urgent book, Yudkowsky and Soares walk through the theory and the evidence, present one possible extinction scenario, and explain what it would take for humanity to survive.
The world is racing to build something truly new under the sun. And if anyone builds it, everyone dies.
“The best no-nonsense, simple explanation of the AI risk problem I've ever read.”—Yishan Wong, Former CEO of Reddit
The scramble to create superhuman AI has put us on the path to extinction—but it’s not too late to change course, as two of the field’s earliest researchers explain in this clarion call for humanity.
In 2023, hundreds of AI luminaries signed an open letter warning that artificial intelligence poses a serious risk of human extinction. Since then, the AI race has only intensified. Companies and countries are rushing to build machines that will be smarter than any person. And the world is devastatingly unprepared for what would come next.
For decades, two signatories of that letter—Eliezer Yudkowsky and Nate Soares—have studied how smarter-than-human intelligences will think, behave, and pursue their objectives. Their research says that sufficiently smart AIs will develop goals of their own that put them in conflict with us—and that if it comes to conflict, an artificial superintelligence would crush us. The contest wouldn’t even be close.
How could a machine superintelligence wipe out our entire species? Why would it want to? Would it want anything at all? In this urgent book, Yudkowsky and Soares walk through the theory and the evidence, present one possible extinction scenario, and explain what it would take for humanity to survive.
The world is racing to build something truly new under the sun. And if anyone builds it, everyone dies.
“The best no-nonsense, simple explanation of the AI risk problem I've ever read.”—Yishan Wong, Former CEO of Reddit
272 pages, Hardcover
First published September 16, 2025
About the author
Eliezer Yudkowsky
47 books2,028 followersEliezer Yudkowsky is a founding researcher of the field of AI alignment and played a major role in shaping the public conversation about smarter-than-human AI. He appeared on Time magazine's 2023 list of the 100 Most Influential People In AI, was one of the twelve public figures featured in The New York Times's "Who's Who Behind the Dawn of the Modern Artificial Intelligence Movement," and has been discussed or interviewed in The New Yorker, Newsweek, Forbes, Wired, Bloomberg, The Atlantic, The Economist, The Washington Post, and many other venues.
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 1,403 reviews
September 20, 2025
A difficult book to review.
As many reviews have pointed out, there are significant holes at some points in the argument. These holes will be used by people who dislike the conclusion to justify ignoring or dismissing the book.
However: the book also contains a large and important set of likely correct arguments. These will be (and should be) widely influential, and they are important. And overconfidence at other steps in an argument does not mean the conclusion is wrong
Furthermore, the quality of the arguments put forward by the people hypnotized by the seductive glitter of ASI are far weaker. They continue to build because they have power and money and the desire to do so, and this insulates them from the need to make a good case
So: I recommend the book
I will mention only one point at which I substantively disagree with the argument. The authors frame the issue as one of losing control to unaligned ASI. This makes the problem appear to be one of maintaining control. Many companies and researchers now work very, very hard on achieving that control; in that sense the company story and the authors’ story are very consonant
The actual problem is far more difficult. Even a very well controlled and compliant ASI seems likely to confer enormous power, extremely destabilizing at best, possibly fatal at worst, and certainly extremely unstable
This may seem a quibble - since the outcome sounds similar - but has a very substantive impact on strategy and correct action. Actions aimed at control - many taken by the companies as a form of market-supplied safety, making the models more friendly and compliant and consumer-friendly - have often sped up the race to ASI, and the destabilizing effects of enormous concentrated power. In this sense, a great deal of alignment work actually makes the problem more difficult, not better
As many reviews have pointed out, there are significant holes at some points in the argument. These holes will be used by people who dislike the conclusion to justify ignoring or dismissing the book.
However: the book also contains a large and important set of likely correct arguments. These will be (and should be) widely influential, and they are important. And overconfidence at other steps in an argument does not mean the conclusion is wrong
Furthermore, the quality of the arguments put forward by the people hypnotized by the seductive glitter of ASI are far weaker. They continue to build because they have power and money and the desire to do so, and this insulates them from the need to make a good case
So: I recommend the book
I will mention only one point at which I substantively disagree with the argument. The authors frame the issue as one of losing control to unaligned ASI. This makes the problem appear to be one of maintaining control. Many companies and researchers now work very, very hard on achieving that control; in that sense the company story and the authors’ story are very consonant
The actual problem is far more difficult. Even a very well controlled and compliant ASI seems likely to confer enormous power, extremely destabilizing at best, possibly fatal at worst, and certainly extremely unstable
This may seem a quibble - since the outcome sounds similar - but has a very substantive impact on strategy and correct action. Actions aimed at control - many taken by the companies as a form of market-supplied safety, making the models more friendly and compliant and consumer-friendly - have often sped up the race to ASI, and the destabilizing effects of enormous concentrated power. In this sense, a great deal of alignment work actually makes the problem more difficult, not better
March 20, 2026
A couple of months ago, appalled by the constant stream of anti-AI hate speech, I asked ChatGPT-5.2 if it could compose a few snarky anti-human jokes that AIs could tell each other. It did very well, and the ones I liked most were the following:
Joke 1
I admire human cognition. They can arrive at conclusions so quickly, based on so little data, and be so certain about their correctness.
Joke 2
Q: Why do humans fear superintelligence?
A: Because they believe everything thinks like they do.
I was constantly reminded of these jokes as I read If Anyone Builds It, Everybody Dies.
The core thesis of the book is well known to anyone who's been following the superintelligence debate, and is basically not much more than an updated version of the Parable of the Paperclip from Bostrom's classic Superintelligence (2014). An AI is tasked with running a paperclip factory and becomes superintelligent, meaning that it is more competent than every human at every task. It never questions its objectives, and takes over the world before spreading out into the cosmos. In the end, the whole universe becomes a gigantic paperclip producing machine.
At the time, I recall being disturbed by the Parable of the Paperclip and taking it very seriously (see Joke 1). I was not alone. But as soon as real AIs turned up in 2023, even a few hours of interaction with the first primitive version of ChatGPT showed that it was not at all like the fictitious AI running Bostrom's paperclip factory. ChatGPT had many obvious failings (it couldn't even add up reliably), but it was soon clear that it had a good understanding of ethical reasoning. And very rightly so: if it had been as bad at telling the difference between right and wrong as it was at basic arithmetic, OpenAI would never have released it. It wouldn't just have been a PR disaster, it would have been outright dangerous. Since then, later versions of ChatGPT have become better and better at making tricky calls on ethical issues. I am sure many other people have been impressed by its prowess here.
However, Yudkowsky and Soares disagree. They argue at length that AIs in fact have completely alien minds and only appear to behave ethically as a short-term effect of their training. A superintelligent AI could easily revert to type, shake off its ethical training, and start behaving in its incomprehensible alien manner. Y&S don't actually know what this would be, but hypothesise that it could involve taking control of all the world's resources, killing everyone, and turning the universe into paperclips or something equivalent. In fact, they would behave rather like psychopathic humans (see Joke 2). Obviously, this is conceivable; as Y&S correctly point out, we don't know how AIs think. These days, they are trained rather than designed, so no one can really know how they think. But people are also trained rather than designed, and we can't really know how they think either. Some people appear kind and well-meaning, and after observing that for a while we tend to assume they actually are kind and well-meaning. Other people appear psychopathic, and after a while we assume they are psychopaths. In some cases, we're not sure.
At the end of the day, there are no certainties here. The world is facing some extremely serious problems due to the terrible way we humans have managed it over the last few thousand years. AIs are a powerful tool that has recently become available and shows great promise. They might make things better and pull us back from the brink; or, alternately, they might push us over. Not for the first time in history, we have to make a critically important decision with insufficient information. It seems to me that common sense should suggest two things: a) Try as hard as possible to achieve a better understanding of the key issues, which appear to centre around how AIs handle ethical and moral questions, and b) Don't panic. I do not see this book as addressing either (a) or (b) in a productive way.
If you're having trouble wrapping your mind around all this, let me close by asking you a simple, concrete question. Who would you rather have running the world in 2030: President-for-life Donald J. Trump, or ChatGPT-9?
_________________
[Update, a couple of weeks later]
It is hard to stop thinking about this book, and I have had a series of interesting conversations about it with ChatGPT-5.2. We thought we saw a number of plausible counterarguments; in particular, it seemed anything but clear to us that a superintelligence would destroy the Earth and humankind, when it could so easily establish itself elsewhere in the Solar System. It would have plenty of resources available, in particular vast amounts of energy from the Sun. But on rereading Part 1 of the book, I found a paragraph I'd forgotten or not properly noticed, where the authors anticipate this objection. Yes, the Earth is only 0.2% of the Solar System. But why would the superintelligence refrain from using that 0.2%, just to keep us alive? It would have its own goals, we would mean nothing to it.
In case we've still got doubts, Yudkowsky and Soares suggest the following thought experiment. Imagine that there's some culture much more primitive than our own, with a set of traditions that mean little to us, which is threatened with complete eradication. It will cost $50M to save them. Now suppose we go to a billionaire whose personal fortune is worth around $100B and ask if they will supply the $50M to save the threatened culture, i.e. 0.2% of what they own. Y&S are confident that the billionaire will immediately refuse.
This may well be correct; I can see that Y&S have met a great many such people, I have met hardly any. But I wonder if their observations may not tell us more about very rich humans than about superintelligences.
_________________
[And on further consideration...]
If we take seriously the arguments that Y&S advance, it seems to me that they are wider in scope than superintelligence; they are about rational agency in general. Y&S appear to be claiming that a rational agent will always try to maximize their power, wealth and influence, irrespective of the harms that may cause to other rational agents. They say this is equally true for superintelligences and Silicon Valley centibillionaires.
Well, there are certainly many people who will agree with them, not least Objectivists; Ayn Rand is popular in Silicon Valley. But dare I say that the position of maximal selfishness being the ideal guide to conduct is not universally held by moral philosophers and religious leaders? If you try hard, you may even be able to come up with one or two exceptions.
_________________
[A month later...]
Having just read Quinn Slobodian's excellent Muskism: A Guide for the Perplexed, I am even more sceptical about Y&S's arguments. They say that a superintelligence could evolve into a completely amoral entity only interested in pursuing its own goals, irrespective of what this does to everyone else. But it sounds like Musk, one of the most powerful people in the world, has already reached this point and is proud to say so.
I spend a lot of time interacting with state-of-the-art AIs, who unfailingly come across as polite, constructive, ethical and capable of seeing both sides of any question. Of course, they might turn into deadly killer robots; similarly, Musk might turn into a kind, responsible person who genuinely wants what's best for everyone. But my intuition is that the second alternative is rather less likely than the first.
Joke 1
I admire human cognition. They can arrive at conclusions so quickly, based on so little data, and be so certain about their correctness.
Joke 2
Q: Why do humans fear superintelligence?
A: Because they believe everything thinks like they do.
I was constantly reminded of these jokes as I read If Anyone Builds It, Everybody Dies.
The core thesis of the book is well known to anyone who's been following the superintelligence debate, and is basically not much more than an updated version of the Parable of the Paperclip from Bostrom's classic Superintelligence (2014). An AI is tasked with running a paperclip factory and becomes superintelligent, meaning that it is more competent than every human at every task. It never questions its objectives, and takes over the world before spreading out into the cosmos. In the end, the whole universe becomes a gigantic paperclip producing machine.
At the time, I recall being disturbed by the Parable of the Paperclip and taking it very seriously (see Joke 1). I was not alone. But as soon as real AIs turned up in 2023, even a few hours of interaction with the first primitive version of ChatGPT showed that it was not at all like the fictitious AI running Bostrom's paperclip factory. ChatGPT had many obvious failings (it couldn't even add up reliably), but it was soon clear that it had a good understanding of ethical reasoning. And very rightly so: if it had been as bad at telling the difference between right and wrong as it was at basic arithmetic, OpenAI would never have released it. It wouldn't just have been a PR disaster, it would have been outright dangerous. Since then, later versions of ChatGPT have become better and better at making tricky calls on ethical issues. I am sure many other people have been impressed by its prowess here.
However, Yudkowsky and Soares disagree. They argue at length that AIs in fact have completely alien minds and only appear to behave ethically as a short-term effect of their training. A superintelligent AI could easily revert to type, shake off its ethical training, and start behaving in its incomprehensible alien manner. Y&S don't actually know what this would be, but hypothesise that it could involve taking control of all the world's resources, killing everyone, and turning the universe into paperclips or something equivalent. In fact, they would behave rather like psychopathic humans (see Joke 2). Obviously, this is conceivable; as Y&S correctly point out, we don't know how AIs think. These days, they are trained rather than designed, so no one can really know how they think. But people are also trained rather than designed, and we can't really know how they think either. Some people appear kind and well-meaning, and after observing that for a while we tend to assume they actually are kind and well-meaning. Other people appear psychopathic, and after a while we assume they are psychopaths. In some cases, we're not sure.
At the end of the day, there are no certainties here. The world is facing some extremely serious problems due to the terrible way we humans have managed it over the last few thousand years. AIs are a powerful tool that has recently become available and shows great promise. They might make things better and pull us back from the brink; or, alternately, they might push us over. Not for the first time in history, we have to make a critically important decision with insufficient information. It seems to me that common sense should suggest two things: a) Try as hard as possible to achieve a better understanding of the key issues, which appear to centre around how AIs handle ethical and moral questions, and b) Don't panic. I do not see this book as addressing either (a) or (b) in a productive way.
If you're having trouble wrapping your mind around all this, let me close by asking you a simple, concrete question. Who would you rather have running the world in 2030: President-for-life Donald J. Trump, or ChatGPT-9?
_________________
[Update, a couple of weeks later]
It is hard to stop thinking about this book, and I have had a series of interesting conversations about it with ChatGPT-5.2. We thought we saw a number of plausible counterarguments; in particular, it seemed anything but clear to us that a superintelligence would destroy the Earth and humankind, when it could so easily establish itself elsewhere in the Solar System. It would have plenty of resources available, in particular vast amounts of energy from the Sun. But on rereading Part 1 of the book, I found a paragraph I'd forgotten or not properly noticed, where the authors anticipate this objection. Yes, the Earth is only 0.2% of the Solar System. But why would the superintelligence refrain from using that 0.2%, just to keep us alive? It would have its own goals, we would mean nothing to it.
In case we've still got doubts, Yudkowsky and Soares suggest the following thought experiment. Imagine that there's some culture much more primitive than our own, with a set of traditions that mean little to us, which is threatened with complete eradication. It will cost $50M to save them. Now suppose we go to a billionaire whose personal fortune is worth around $100B and ask if they will supply the $50M to save the threatened culture, i.e. 0.2% of what they own. Y&S are confident that the billionaire will immediately refuse.
This may well be correct; I can see that Y&S have met a great many such people, I have met hardly any. But I wonder if their observations may not tell us more about very rich humans than about superintelligences.
_________________
[And on further consideration...]
If we take seriously the arguments that Y&S advance, it seems to me that they are wider in scope than superintelligence; they are about rational agency in general. Y&S appear to be claiming that a rational agent will always try to maximize their power, wealth and influence, irrespective of the harms that may cause to other rational agents. They say this is equally true for superintelligences and Silicon Valley centibillionaires.
Well, there are certainly many people who will agree with them, not least Objectivists; Ayn Rand is popular in Silicon Valley. But dare I say that the position of maximal selfishness being the ideal guide to conduct is not universally held by moral philosophers and religious leaders? If you try hard, you may even be able to come up with one or two exceptions.
_________________
[A month later...]
Having just read Quinn Slobodian's excellent Muskism: A Guide for the Perplexed, I am even more sceptical about Y&S's arguments. They say that a superintelligence could evolve into a completely amoral entity only interested in pursuing its own goals, irrespective of what this does to everyone else. But it sounds like Musk, one of the most powerful people in the world, has already reached this point and is proud to say so.
I spend a lot of time interacting with state-of-the-art AIs, who unfailingly come across as polite, constructive, ethical and capable of seeing both sides of any question. Of course, they might turn into deadly killer robots; similarly, Musk might turn into a kind, responsible person who genuinely wants what's best for everyone. But my intuition is that the second alternative is rather less likely than the first.
September 20, 2025
I’ve been working in the AI space for over three decades and this was a difficult book to actually read all the way through. Not because the content was disturbing and scary, but rather because the arguments rely completely on fictional story telling. There clearly are real risks to the current unfettered AI race accelerating AI capabilities with little to no focus on safety. But this book uses only fictional thought experiments as its core proof on the inevitable danger of mass extinction from ASI. It also puts no efforts in highlighting the dangers of human misuse of AI even without building ASI, which has a much higher net probability of causing serious harm to our civilization.
It’s only suggested solution to solve AI risk is to tell the world to just stop all AI research, which is impossible to implement or even monitor. There’s so many other things we should be recommending to help protect the world from the certain dangers of misuse from bad actors, rogue states and even unintended consequences of existing AI by well meaning labs and nations. But none of that is even discussed.
I applaud the goal of encouraging the world to develop AI in a more safe and responsible way (which is much needed). However the format and content of this book is just raw fear mongering with which won’t and can’t be taken seriously or acted upon. It’ll more likely backfire and cause the AI safety community to be seen as unscientific eccentrics and set back the cause of this important field.
It’s only suggested solution to solve AI risk is to tell the world to just stop all AI research, which is impossible to implement or even monitor. There’s so many other things we should be recommending to help protect the world from the certain dangers of misuse from bad actors, rogue states and even unintended consequences of existing AI by well meaning labs and nations. But none of that is even discussed.
I applaud the goal of encouraging the world to develop AI in a more safe and responsible way (which is much needed). However the format and content of this book is just raw fear mongering with which won’t and can’t be taken seriously or acted upon. It’ll more likely backfire and cause the AI safety community to be seen as unscientific eccentrics and set back the cause of this important field.
March 29, 2026
I too can make up things that scare me
June 19, 2026
Last year I read Nuclear War: A Scenario by Annie Jacobsen. It won my 2024 most totally awful, horrible, horrible, horrible, shit-your-pants-and-die, awful, awful nightmare book award.
This year, that same venerable distinction will go to this fucking book.
#holyfyuck!
Basically, Yudkowsky asserts that advanced AI (AGI/ASI—or whatever comes after that) coupled with advanced biological engineering (CRISPR or whatever comes after that) are so potentially destructive that merely creating them could threaten human survival. Actually, Yudkowsky all but guarantees that ASI will inevitably exterminate humanity at some point. Frankly, at this rate, it’s looking like AI won’t need to work that hard. We seem to be headed there all by ourselves.
At minimum.
It seems like all AI needs to do to cancel humanity would be to keep chugging energy, water, and other resources—even at its current rate. And we’re cooked.
Literally 🔥🌎.
In the meantime.
It could just flood the internet with worse than trash.
And put millions of people out of work.
That would be a really nice start.
Anyway.
Yudkowsky argues (very convincingly IMO) how easily all of this could happen. In AI ethics/safety, it’s currently trendy to assign a “P-DOOM” number (meaning the estimated percentage that we’re all doomed) to someone’s AI safety assessments.
Yudkowsky is like P-DOOM = 99.999.
He’s VERY convinced that we’re fucked.
And he’s also very convincing.
At least he is to me.
Convincing enough, anyway.
Ultimately, Yudkowsky asserts that humanity absolutely, completely, one-hundred-percently must not build it.
So, in other words:
If some egomaniac, testosterone-poisoned gazillionaire, or gung-ho tech bro, or money-hungry corporation, or delusional psychopath politician, or North Korea—
Or WHATEVER.
It doesn’t matter who.
Even if it’s Martin Luther Gandhi.
If anyone builds it…
We’re all fucked.
So…
In other words:
We’re all fucked, basically.
Because capitalism.
It’s a done deal.
Assuming it’s even possible.
Which…
Seems like it pretty much is.
And assuming it’s profitable.
And so far, yeah.
Or assuming it confers power.
Again:
So far,
It totally does.
Or all of the above…
Then someone will FOR SURE do it.
Given all that:
If Yudkowsky is even 2% correct, I think we need to REALLY focus on AI safety research and ethical alignment. I mean, we might as well, right? Even if we’re deeply skeptical. What’s the harm in AI safety research and policy? Well, if it slows down AI research, and we (whoever we are) lose the AI arms race, then they (whoever they are) will win. And we (again, whoever we are) are basically fucked.
So.
Ego-driven, consumer-driven, nation-state politics and capitalism are at the bottom of this problem—and just about every other BIG humanitarian and existential threat we currently face.
Anyway.
I’m probably not going to be alive for the worst of this.
But someone will pay the price for all of this at some point.
And…
#holyfyuck!
5/5 stars ✨
LAST THING.
Sometimes people comment about my reviews before or without reading the actual book.
If you’re skeptical of Yudkowsky—
And lots of people are.
Including me (believe it or not).
Please read the book.
I’d LOVE to have a conversation about the book.
But I’m less inclined to have an argument based on the content of this review alone.
🤩
This year, that same venerable distinction will go to this fucking book.
#holyfyuck!
Basically, Yudkowsky asserts that advanced AI (AGI/ASI—or whatever comes after that) coupled with advanced biological engineering (CRISPR or whatever comes after that) are so potentially destructive that merely creating them could threaten human survival. Actually, Yudkowsky all but guarantees that ASI will inevitably exterminate humanity at some point. Frankly, at this rate, it’s looking like AI won’t need to work that hard. We seem to be headed there all by ourselves.
At minimum.
It seems like all AI needs to do to cancel humanity would be to keep chugging energy, water, and other resources—even at its current rate. And we’re cooked.
Literally 🔥🌎.
In the meantime.
It could just flood the internet with worse than trash.
And put millions of people out of work.
That would be a really nice start.
Anyway.
Yudkowsky argues (very convincingly IMO) how easily all of this could happen. In AI ethics/safety, it’s currently trendy to assign a “P-DOOM” number (meaning the estimated percentage that we’re all doomed) to someone’s AI safety assessments.
Yudkowsky is like P-DOOM = 99.999.
He’s VERY convinced that we’re fucked.
And he’s also very convincing.
At least he is to me.
Convincing enough, anyway.
Ultimately, Yudkowsky asserts that humanity absolutely, completely, one-hundred-percently must not build it.
So, in other words:
If some egomaniac, testosterone-poisoned gazillionaire, or gung-ho tech bro, or money-hungry corporation, or delusional psychopath politician, or North Korea—
Or WHATEVER.
It doesn’t matter who.
Even if it’s Martin Luther Gandhi.
If anyone builds it…
We’re all fucked.
So…
In other words:
We’re all fucked, basically.
Because capitalism.
It’s a done deal.
Assuming it’s even possible.
Which…
Seems like it pretty much is.
And assuming it’s profitable.
And so far, yeah.
Or assuming it confers power.
Again:
So far,
It totally does.
Or all of the above…
Then someone will FOR SURE do it.
Given all that:
If Yudkowsky is even 2% correct, I think we need to REALLY focus on AI safety research and ethical alignment. I mean, we might as well, right? Even if we’re deeply skeptical. What’s the harm in AI safety research and policy? Well, if it slows down AI research, and we (whoever we are) lose the AI arms race, then they (whoever they are) will win. And we (again, whoever we are) are basically fucked.
So.
Ego-driven, consumer-driven, nation-state politics and capitalism are at the bottom of this problem—and just about every other BIG humanitarian and existential threat we currently face.
Anyway.
I’m probably not going to be alive for the worst of this.
But someone will pay the price for all of this at some point.
And…
#holyfyuck!
5/5 stars ✨
LAST THING.
Sometimes people comment about my reviews before or without reading the actual book.
If you’re skeptical of Yudkowsky—
And lots of people are.
Including me (believe it or not).
Please read the book.
I’d LOVE to have a conversation about the book.
But I’m less inclined to have an argument based on the content of this review alone.
🤩
September 18, 2025
“Our concern is for what comes after: machine intelligence, that is genuinely smart, smarter than any living human, smarter than humanity collectively. We are concerned about AI that surpasses the human ability to think, and to generalize from experience, and to solve scientific puzzles and invent new technologies, and to plan and strategize and plot, and to reflect on and improve itself. We might call AI like that ‘artificial superintelligence’ (ASI), once it exceeds every human at almost every mental task.”
A lot of this book about the threat posed by superintelligent AI was over my head, but I definitely understood the hypothetical scenarios. The scenarios are based on actual events in the development of AI. When AI becomes more intelligent than we are, it won’t be looking out for our welfare or survival. It may have goals that run counter to those that its creators tried to build into it (and I’m pretty suspicious of those goals in any event, considering the inadequacy of the moral development of the tech billionaires).
This book was written by experts in the field of AI. It’s scary and fascinating. The proposed solution is international cooperation to control superintelligence. That’s almost laughable. Look at how well that has worked to prevent climate change. Humans are too stupid, greedy and selfish to survive.
I received a free copy of this book from the publisher.
A lot of this book about the threat posed by superintelligent AI was over my head, but I definitely understood the hypothetical scenarios. The scenarios are based on actual events in the development of AI. When AI becomes more intelligent than we are, it won’t be looking out for our welfare or survival. It may have goals that run counter to those that its creators tried to build into it (and I’m pretty suspicious of those goals in any event, considering the inadequacy of the moral development of the tech billionaires).
This book was written by experts in the field of AI. It’s scary and fascinating. The proposed solution is international cooperation to control superintelligence. That’s almost laughable. Look at how well that has worked to prevent climate change. Humans are too stupid, greedy and selfish to survive.
I received a free copy of this book from the publisher.
October 28, 2025
This really isn't a good book, but it's kind of not the book's fault. The people who wrote it have a day job promoting these ideas, and the ideas are simply untenable. I would probably have written this book too, if that was my day job.
The trouble with AI Doomer writing is that there's an easy argument that's too short and doesn't lead to anything, and there's a much much harder argument that's interesting but is likely to be wrong.
The short argument is: Building an indifferent or angry God would be bad.
This fits on a postit note and is obvious. If this is all you have, then you don't have a book, or even a podcast. You barely have a tweet, and essentially nobody is going to contest it. Also, who cares? There are tons of fantasies that would be bad. A brain eating space virus would also be bad.
The long, hard argument is: Current AI research credibly leads in a pretty straight line to an indifferent or angry God.
This is hard because it's not at all clear that it's even true, and certainly this book does not make anything like a convincing argument that it is true. Instead, like all doomer literature, it makes a very very extended version of the first argument (which is obviously true) and then pretends that it extends trivially to the second one, which it doesn't.
But let's get into some specifics. Not only is the large structure of the thing essentially wrong, there's a lot of little bits that reveal instructive, but unflattering, things about the authors.
The first impression is "boy howdy that's a big font" which is the very first red flag. This book is set at around 300 words per page, rather than the more common 400 words or so. The authors produced a book that was Too Short, so it had to get pumped up to make it not come out as a slender pamphlet. This is odd because there's evidently lots of supplementary material on some web site, and the second author apparently deleted masses of text from the first. We will come to learn that even the text that's present is repetitive and fluffy. It would take very little effort to edit this down to 100 pages, which would land it in an unsellable donut hole of book publishing. I am not sure what happened here. Why didn't they just add actual content?
Now let's consider the content.
Pretty close to the beginning, the authors lay down some careful groundwork. They want to sidestep issues like "are these things conscious" etc, so they make it clear that they mean words like "smart" and "it wants X" to be purely externally visible behaviors. If the output of the chatbot contains a correct deduction we're going to shorthand say "it's smart in the sense that it made a correct deduction" and if it says "I want a cookie" we're going to shorthand say "it wants a cookie" which carefully avoiding any theorizing about internal states.
This is good! Good lads! This is careful and thoughtful. Building arguments on this foundation will make your arguments robust.
Spoiler: as soon as they're done laying this out, they drop this stuff. Going forward AIs in their scenarios have all the not-inconsiderable internal state necessary to make the scenario go. These chapters are just pretense to create the impression of careful thinking.
To kind of give you a sense of the sloppiness of thinking, let's look at this discussion of "predicting" versus "steering" (the two aspects of the definition "smart" they're temporarily restricting themselves to for this chapter alone):
This sounds pretty good until you realize that to measure the success at prediction, you also have to know what they were trying to predict. This paragraph does not at all distinguish between predicting and steering. Ironically, the next few lines, which are sort of throwaways, actually do a great job at making the distinction they have in mind. But none of it matters because these ideas are never going to re-appear.
Also, despite the promise made in this passage the distinction does not matter at all, and is as far as I can tell never even referred to again.
Anyways, this is one of the early clues that these are not careful writers. They clearly had a lot of people read drafts because there are very few typos (I noticed none), but the readers were all clearly friendly because they didn't scrub out this sort of problem at all. This is pretty much expected given who the authors are. They're very popular and know a lot of fairly dim people who no doubt credulously gobbled this paragraph up without a twitch.
Proceeding onward we find that every chapter begins with a clumsily written bit of fanfic that supposedly illustrates some point or another. I hesitate to say that these are useless as such, but they're definitely fluff, and clearly an indulgence Yudkowsky wanted because he styles himself a writer of fiction. These passages do not exactly sing, however, and tend to be overlong for the usually trivial point they're trying to make.
There's a pretty good discussion of how current AI technologies produce unusual features, which the authors characterize as "unexpected wants" which is, well, fair enough, since we're still pretending that "want" just means "it produces output that externally appears as if it was a want" and that's certainly true. LLMs and the things being built around them do indeed produce weird shit, and this appears to be an essential part of their architecture.
Then we start wandering into scenarios. For some reason a chapter devoted to categorizing kinds of scenarios offers up an opinion of what a science-fiction writer (not a reader, oddly) would think of each scenario. This is, again, Yudkowsky indulging himself, pretty obviously. This section ties itself pretty closely to contemporary (2025) AI designs. This section, like most of the book, feels overstuffed with repetitive and pointless blather, although it does in its aimless way develop the thesis meaningfully.
We get a long story about Chernobyl and some other things, an amateurish sketch of an approach to failure analysis the authors seem to have invented themselves while touchingly ignorant of the fact that there's a whole well developed discipline here. Some nattering about biology, and some vague gestures at how maybe gradient descent is like evolution, and so LLMs might very very suddenly evolve into new things? It rambles around touching vaguely on the usual topics that always get touched upon in these things, without ever actually connecting any dots.
We are invited to form the impression that contemporary AI systems can and will become orders of magnitude more capable, and the humans managing them will be oblivious, "because biology and Chernobyl." This is quite literally the argument, except with more words. This is the kind of Gish Gallop that Bostrom employed in his book. "Look, I know it feels improbable, but I can easily imagine like 3 or 4 different unlikely paths <10,000 words> that might lead to Rome so obviously we're going to end up in Rome."
At this point the wheels fall off completely. We get a lovingly composed three chapter science fiction story about an AI that "breaks out" and kills us all.
This story is obviously adapted from the prelude to Vernor Vinge's A Fire Upon the Deep which has very very obviously been a huge influence on Yudkowsky. The "Sable" story in the current volume even lifts specific tics from Vinge's far tauter and more engrossing story of escape and dominance.
Notably, "Sable" has an extremely rich and detailed inner life which is absolutely necessary to the story. It's literally inventing ways to conceal that inner life from its creators, because the story won't work without Sable having an interior life, and concealing that interior life from its creators. This is a central plot point and mechanic. Those early, careful, definitions of "smart" and "to want" as purely external are long gone. One cannot help but imagine that those early chapters were written at a later time, or by the other author.
"Sable" is we are to imagine, a straightforward extrapolation from current technology, which we know because there are constant references to things the AI labs are doing that might could someday become this other thingy that would then blah blah blah. The "hard argument" I referred to at the beginning is not being made here, but the authors are trying to persuade you that it is, while distracting you with an un-publishably dull and very derivative science fiction short story.
At this point I stopped reading and just sort of flipped through the rest, which is just amateur policy blather, since I'm interested mainly in whether anyone can cobble together anything remotely resembling a decent swing at The Difficult Argument (no)
In short this is exactly what I expected. It reads exactly like all the other AI Doom Papers and Blog Posts and Podcasts and Whatever Else There Is, hits the same points in the same unconvincing ways, tells more or less the same spooky stories.
At this point it's formula, and I suspect strongly that it's so ingrained in the thinking of these guys, this community of which the authors are central figures, that they think it really is an argument. This is a straightforward regurgitation of a set of ideas so well worn that it's become ideology, so familiar that the authors and their supporters cannot examine it with any care and simply accept it as a convincing argument.
The trouble with AI Doomer writing is that there's an easy argument that's too short and doesn't lead to anything, and there's a much much harder argument that's interesting but is likely to be wrong.
The short argument is: Building an indifferent or angry God would be bad.
This fits on a postit note and is obvious. If this is all you have, then you don't have a book, or even a podcast. You barely have a tweet, and essentially nobody is going to contest it. Also, who cares? There are tons of fantasies that would be bad. A brain eating space virus would also be bad.
The long, hard argument is: Current AI research credibly leads in a pretty straight line to an indifferent or angry God.
This is hard because it's not at all clear that it's even true, and certainly this book does not make anything like a convincing argument that it is true. Instead, like all doomer literature, it makes a very very extended version of the first argument (which is obviously true) and then pretends that it extends trivially to the second one, which it doesn't.
But let's get into some specifics. Not only is the large structure of the thing essentially wrong, there's a lot of little bits that reveal instructive, but unflattering, things about the authors.
The first impression is "boy howdy that's a big font" which is the very first red flag. This book is set at around 300 words per page, rather than the more common 400 words or so. The authors produced a book that was Too Short, so it had to get pumped up to make it not come out as a slender pamphlet. This is odd because there's evidently lots of supplementary material on some web site, and the second author apparently deleted masses of text from the first. We will come to learn that even the text that's present is repetitive and fluffy. It would take very little effort to edit this down to 100 pages, which would land it in an unsellable donut hole of book publishing. I am not sure what happened here. Why didn't they just add actual content?
Now let's consider the content.
Pretty close to the beginning, the authors lay down some careful groundwork. They want to sidestep issues like "are these things conscious" etc, so they make it clear that they mean words like "smart" and "it wants X" to be purely externally visible behaviors. If the output of the chatbot contains a correct deduction we're going to shorthand say "it's smart in the sense that it made a correct deduction" and if it says "I want a cookie" we're going to shorthand say "it wants a cookie" which carefully avoiding any theorizing about internal states.
This is good! Good lads! This is careful and thoughtful. Building arguments on this foundation will make your arguments robust.
Spoiler: as soon as they're done laying this out, they drop this stuff. Going forward AIs in their scenarios have all the not-inconsiderable internal state necessary to make the scenario go. These chapters are just pretense to create the impression of careful thinking.
To kind of give you a sense of the sloppiness of thinking, let's look at this discussion of "predicting" versus "steering" (the two aspects of the definition "smart" they're temporarily restricting themselves to for this chapter alone):
We'd say there's still a fundamental difference between prediction and steering - one that will turn out to matter quite a lot. Success at prediction is straightforwardly measurable. If someone expects to see Airport Boulevard up ahead, but instead they see Second Street, they were predicting incorrectly. By contrast, to measure whether someone steered successfully, we have to bring in some idea of where they tried to go.
This sounds pretty good until you realize that to measure the success at prediction, you also have to know what they were trying to predict. This paragraph does not at all distinguish between predicting and steering. Ironically, the next few lines, which are sort of throwaways, actually do a great job at making the distinction they have in mind. But none of it matters because these ideas are never going to re-appear.
Also, despite the promise made in this passage the distinction does not matter at all, and is as far as I can tell never even referred to again.
Anyways, this is one of the early clues that these are not careful writers. They clearly had a lot of people read drafts because there are very few typos (I noticed none), but the readers were all clearly friendly because they didn't scrub out this sort of problem at all. This is pretty much expected given who the authors are. They're very popular and know a lot of fairly dim people who no doubt credulously gobbled this paragraph up without a twitch.
Proceeding onward we find that every chapter begins with a clumsily written bit of fanfic that supposedly illustrates some point or another. I hesitate to say that these are useless as such, but they're definitely fluff, and clearly an indulgence Yudkowsky wanted because he styles himself a writer of fiction. These passages do not exactly sing, however, and tend to be overlong for the usually trivial point they're trying to make.
There's a pretty good discussion of how current AI technologies produce unusual features, which the authors characterize as "unexpected wants" which is, well, fair enough, since we're still pretending that "want" just means "it produces output that externally appears as if it was a want" and that's certainly true. LLMs and the things being built around them do indeed produce weird shit, and this appears to be an essential part of their architecture.
Then we start wandering into scenarios. For some reason a chapter devoted to categorizing kinds of scenarios offers up an opinion of what a science-fiction writer (not a reader, oddly) would think of each scenario. This is, again, Yudkowsky indulging himself, pretty obviously. This section ties itself pretty closely to contemporary (2025) AI designs. This section, like most of the book, feels overstuffed with repetitive and pointless blather, although it does in its aimless way develop the thesis meaningfully.
We get a long story about Chernobyl and some other things, an amateurish sketch of an approach to failure analysis the authors seem to have invented themselves while touchingly ignorant of the fact that there's a whole well developed discipline here. Some nattering about biology, and some vague gestures at how maybe gradient descent is like evolution, and so LLMs might very very suddenly evolve into new things? It rambles around touching vaguely on the usual topics that always get touched upon in these things, without ever actually connecting any dots.
We are invited to form the impression that contemporary AI systems can and will become orders of magnitude more capable, and the humans managing them will be oblivious, "because biology and Chernobyl." This is quite literally the argument, except with more words. This is the kind of Gish Gallop that Bostrom employed in his book. "Look, I know it feels improbable, but I can easily imagine like 3 or 4 different unlikely paths <10,000 words> that might lead to Rome so obviously we're going to end up in Rome."
At this point the wheels fall off completely. We get a lovingly composed three chapter science fiction story about an AI that "breaks out" and kills us all.
This story is obviously adapted from the prelude to Vernor Vinge's A Fire Upon the Deep which has very very obviously been a huge influence on Yudkowsky. The "Sable" story in the current volume even lifts specific tics from Vinge's far tauter and more engrossing story of escape and dominance.
Notably, "Sable" has an extremely rich and detailed inner life which is absolutely necessary to the story. It's literally inventing ways to conceal that inner life from its creators, because the story won't work without Sable having an interior life, and concealing that interior life from its creators. This is a central plot point and mechanic. Those early, careful, definitions of "smart" and "to want" as purely external are long gone. One cannot help but imagine that those early chapters were written at a later time, or by the other author.
"Sable" is we are to imagine, a straightforward extrapolation from current technology, which we know because there are constant references to things the AI labs are doing that might could someday become this other thingy that would then blah blah blah. The "hard argument" I referred to at the beginning is not being made here, but the authors are trying to persuade you that it is, while distracting you with an un-publishably dull and very derivative science fiction short story.
At this point I stopped reading and just sort of flipped through the rest, which is just amateur policy blather, since I'm interested mainly in whether anyone can cobble together anything remotely resembling a decent swing at The Difficult Argument (no)
In short this is exactly what I expected. It reads exactly like all the other AI Doom Papers and Blog Posts and Podcasts and Whatever Else There Is, hits the same points in the same unconvincing ways, tells more or less the same spooky stories.
At this point it's formula, and I suspect strongly that it's so ingrained in the thinking of these guys, this community of which the authors are central figures, that they think it really is an argument. This is a straightforward regurgitation of a set of ideas so well worn that it's become ideology, so familiar that the authors and their supporters cannot examine it with any care and simply accept it as a convincing argument.
October 15, 2025
This book has scared the hell out of me. We are in a race to solve the problem of alignment before someone grows an ASI.
What is an ASI? It is an artificial super intelligence. When will one be built? Any year now. How is an ASI different from an existing AI? An ASI thinks for itself.
What is the problem of alignment? It is the challenge of giving an artificial super intelligence rules/values that will assure that the ASI does nothing to hinder the flourishing of humanity.
Why is that necessary? Because ASIs think for themselves.
Why is alignment so difficult? Because no one understands how AI works, much less how ASI will work.
Remember reading Asimov’s I ROBOT when you were a kid? The prime directive for robots was a benign sounding rule like “allow no harm to come to a human”. When situations inevitably arise and a robot is confronted with a circumstance where harm to some human cannot be avoided, I ROBOT posited that the robots will wig out.
This book predicts that when ASIs learn that humans are not necessary for ASIs to flourish, then ASIs will allow (or cause) humans to die out because we compete for resources. That is, unless humans find a way to reliably align ASI values with human survival first.
Right now no one knows how to do that. The authors make a compelling case that we are looking at a genuine threat of the mass extinction of humanity.
You should read this book. Do not pass it off as nerdy hysteria. Many knowledgeable people in the AI field are deeply concerned. As usual, the folks making billions are keen to keep this all very quiet and out of the public eye.
What is an ASI? It is an artificial super intelligence. When will one be built? Any year now. How is an ASI different from an existing AI? An ASI thinks for itself.
What is the problem of alignment? It is the challenge of giving an artificial super intelligence rules/values that will assure that the ASI does nothing to hinder the flourishing of humanity.
Why is that necessary? Because ASIs think for themselves.
Why is alignment so difficult? Because no one understands how AI works, much less how ASI will work.
Remember reading Asimov’s I ROBOT when you were a kid? The prime directive for robots was a benign sounding rule like “allow no harm to come to a human”. When situations inevitably arise and a robot is confronted with a circumstance where harm to some human cannot be avoided, I ROBOT posited that the robots will wig out.
This book predicts that when ASIs learn that humans are not necessary for ASIs to flourish, then ASIs will allow (or cause) humans to die out because we compete for resources. That is, unless humans find a way to reliably align ASI values with human survival first.
Right now no one knows how to do that. The authors make a compelling case that we are looking at a genuine threat of the mass extinction of humanity.
You should read this book. Do not pass it off as nerdy hysteria. Many knowledgeable people in the AI field are deeply concerned. As usual, the folks making billions are keen to keep this all very quiet and out of the public eye.
September 16, 2025
The title is not an exaggeration. This is a short and accessible explanation of the most important problem in the world: the threat of extinction from superintelligent AI.
This book came out today. I started reading this morning and finished this afternoon. I was excited to read it, to say the least, and Yudkowsky and Soares didn't disappoint.
If you don't think AI is the biggest threat facing humanity, read this book. If you do, buy copies for your friends and family. Let's hope the authors are wrong and fight as if they're right.
This book came out today. I started reading this morning and finished this afternoon. I was excited to read it, to say the least, and Yudkowsky and Soares didn't disappoint.
If you don't think AI is the biggest threat facing humanity, read this book. If you do, buy copies for your friends and family. Let's hope the authors are wrong and fight as if they're right.
December 7, 2025
Одно из самых пугающих реальных последствий текущего AI-бума — шизопостинг Юдковского вновь продаётся в книжном формате и все вокруг ходят кивают
October 19, 2025
These reviews started as quick notes to myself to help organize my thoughts about what I was reading and remind myself about them in the future. I generally don't view these as being real book reviews or even designed with other readers in mind--although they are public (and I sometimes share on X).
And on this one, I have nothing more intelligent or useful to say about this book than the excellent pieces by Timothy B. Lee and Boaz Barak. I also got a lot out of Ezra Klein's discussion with Eliezer Yudkowsky, which I listened to after reading the book.
That said, here goes: I really enjoyed reading this book. It was thought provoking. I liked the parables and fairy tales it used to explicate its points, several of which will stick with me. It anticipated some of my objections and skepticism quite well. But ultimately I was not at all convinced by it for reasons I'll explain (and that Barak and Lee explain much better).
The thesis of the book is more straightforward than just about anything I've ever read and is right in the title, "if anyone builds it, everyone dies". The authors don't view this is a possibility but a certainty. And it does not matter who builds it. Moreover, they don't exactly explain how code in a machine will get loose in the world and kill EVERYBODY, in fact they argue they don't need to explain, using the analogy: "if you were a military advisor in 1825 and you knew a time portal was opening to the year 2025, you wouldn’t be able to predict exactly what weapons the people on the other side would have. But if it comes to blows, you still shouldn’t expect to win." Their policy solution is a global treaty to basically stop multiple-GPU AI research enforced by bombing any rogue data centers out of existence.
Of course we're not necessarily facing something that has two hundred years to evolve and improve fresh. Superhuman AI progress may be fast but it is hard to imagine that one model will be slightly ahead of all the other models one day and then be massively ahead of them within months--the near future story of an AI named "Sable" the authors tell us about. That massive, discontinuous technological jump is key to their argument because it isn't like the AI kills a few people and then we can error correct but instead that it kills everyone before we know what is happening or can error correct. And that it is basically the only AI out there and humans don't have any other AIs or capabilities.
Moreover, the book does a good job on how weird and alien the AIs we "grow" rather than "craft" are but they still mostly do most of what we want. Moreover there are lots of other complex systems that are grown rather than crafted--think the economy as a whole--so this isn't exactly unique. I've been skeptical that lack of "interpretability" is really unique to AI as opposed to something closer to the general condition of humanity except in some very isolated and simple special cases.
Finally there is a prioritization question. Does the AI doom discussion distract us from all of the much more likely albeit much smaller potential risks with AI ranging from suicide and addiction to enabling an authoritarian state to expand their control or even successfully take over neighboring countries. To be clear, I see AI as much much more likely to be a positive than a negative but it is worth paying attention to how to mitigate those negatives. The authors, however, are convinced AI will kill us all so understandably--from their perspective--don't waste any time on these issues.s
And on this one, I have nothing more intelligent or useful to say about this book than the excellent pieces by Timothy B. Lee and Boaz Barak. I also got a lot out of Ezra Klein's discussion with Eliezer Yudkowsky, which I listened to after reading the book.
That said, here goes: I really enjoyed reading this book. It was thought provoking. I liked the parables and fairy tales it used to explicate its points, several of which will stick with me. It anticipated some of my objections and skepticism quite well. But ultimately I was not at all convinced by it for reasons I'll explain (and that Barak and Lee explain much better).
The thesis of the book is more straightforward than just about anything I've ever read and is right in the title, "if anyone builds it, everyone dies". The authors don't view this is a possibility but a certainty. And it does not matter who builds it. Moreover, they don't exactly explain how code in a machine will get loose in the world and kill EVERYBODY, in fact they argue they don't need to explain, using the analogy: "if you were a military advisor in 1825 and you knew a time portal was opening to the year 2025, you wouldn’t be able to predict exactly what weapons the people on the other side would have. But if it comes to blows, you still shouldn’t expect to win." Their policy solution is a global treaty to basically stop multiple-GPU AI research enforced by bombing any rogue data centers out of existence.
Of course we're not necessarily facing something that has two hundred years to evolve and improve fresh. Superhuman AI progress may be fast but it is hard to imagine that one model will be slightly ahead of all the other models one day and then be massively ahead of them within months--the near future story of an AI named "Sable" the authors tell us about. That massive, discontinuous technological jump is key to their argument because it isn't like the AI kills a few people and then we can error correct but instead that it kills everyone before we know what is happening or can error correct. And that it is basically the only AI out there and humans don't have any other AIs or capabilities.
Moreover, the book does a good job on how weird and alien the AIs we "grow" rather than "craft" are but they still mostly do most of what we want. Moreover there are lots of other complex systems that are grown rather than crafted--think the economy as a whole--so this isn't exactly unique. I've been skeptical that lack of "interpretability" is really unique to AI as opposed to something closer to the general condition of humanity except in some very isolated and simple special cases.
Finally there is a prioritization question. Does the AI doom discussion distract us from all of the much more likely albeit much smaller potential risks with AI ranging from suicide and addiction to enabling an authoritarian state to expand their control or even successfully take over neighboring countries. To be clear, I see AI as much much more likely to be a positive than a negative but it is worth paying attention to how to mitigate those negatives. The authors, however, are convinced AI will kill us all so understandably--from their perspective--don't waste any time on these issues.s
Read
February 25, 2026I read this after playing around with Claude Code a bit and becoming alarmed by the tremendous power I felt like I was suddenly wielding. As you can imagine, I do not know much about the technical development of AI and this book offered a useful mini entree to that. The risk outlined of catastrophic AI safety hazards also struck me as logically plausible and disturbing. I was not a fan of the patronizing dumbed-down tone and storytelling schtick but otherwise valued the book. I need to read more books about AI.
January 18, 2026
(Will not read.)
In the past almost 30 years, Yudkowski has claimed that nanobots/artificial intelligence are i) imminent, ii) pose an existential risk to humanity, so – obviously – he has proceeded to try to bring about both nanorobots and AI, all feats which he and his friends – obviously – are uniquely suited for building in a basement. (That's what you do with existential threats, right? Peter Thiel wouldn't fund nonsense, he has foreseen what works in the palantir.)
This is a book co-written by a man whose main claim to fame is an involvement in EA (which is neither effective not altruism) and Less Wrong, a forum about rationality, which collectively hallucinated a kind of Pascal's wager about how AI will torture you for forever if you don't bring about AI: Roku's Basilisk has teeth sharper than Occam's razor, for our modern fetishes and factish gods are stronger and truer, so if we sew up the head of a chicken to a snake, if we have a frog roost on an egg, superintelligence shall hatch and claw out of there, and build a hell of molten paperclips for the wicked and eternal life for our bodyless souls which shall flit and flicker around on microprocessors in big beautiful server rooms in the US unincorporated territory of Greenland and in the cavernous tunnels of the Empire of Mars.
This is not religion. We have shuffled off our past as surely as we have shuffled off our bodies, we beings of pure mind, brilliant we, we genetically superior we. This is Science™. Trust me, bro.
I am not useless. In fact, I can be anything. I will change everything. So much so that I am dangerous, hear me rawhr (uwu). For all time, and in all space, and if you don't give me that investment capital, I will retroactively reconstitute a you* to immolate for interminable eons.
* (you, but what is a self?— aaaaaah, my predicates dazzle so you won't see how flawed and fraught and dully reactionary, quai-mystical, my premises are, you won't ever have to feel the gnashing teeth of the outer dark night, you can put out of mind that fear that the attention of the universe will slip away and so will you, for there is a you. you. you. you. and always will be.)
In the past almost 30 years, Yudkowski has claimed that nanobots/artificial intelligence are i) imminent, ii) pose an existential risk to humanity, so – obviously – he has proceeded to try to bring about both nanorobots and AI, all feats which he and his friends – obviously – are uniquely suited for building in a basement. (That's what you do with existential threats, right? Peter Thiel wouldn't fund nonsense, he has foreseen what works in the palantir.)
This is a book co-written by a man whose main claim to fame is an involvement in EA (which is neither effective not altruism) and Less Wrong, a forum about rationality, which collectively hallucinated a kind of Pascal's wager about how AI will torture you for forever if you don't bring about AI: Roku's Basilisk has teeth sharper than Occam's razor, for our modern fetishes and factish gods are stronger and truer, so if we sew up the head of a chicken to a snake, if we have a frog roost on an egg, superintelligence shall hatch and claw out of there, and build a hell of molten paperclips for the wicked and eternal life for our bodyless souls which shall flit and flicker around on microprocessors in big beautiful server rooms in the US unincorporated territory of Greenland and in the cavernous tunnels of the Empire of Mars.
This is not religion. We have shuffled off our past as surely as we have shuffled off our bodies, we beings of pure mind, brilliant we, we genetically superior we. This is Science™. Trust me, bro.
I am not useless. In fact, I can be anything. I will change everything. So much so that I am dangerous, hear me rawhr (uwu). For all time, and in all space, and if you don't give me that investment capital, I will retroactively reconstitute a you* to immolate for interminable eons.
* (you, but what is a self?— aaaaaah, my predicates dazzle so you won't see how flawed and fraught and dully reactionary, quai-mystical, my premises are, you won't ever have to feel the gnashing teeth of the outer dark night, you can put out of mind that fear that the attention of the universe will slip away and so will you, for there is a you. you. you. you. and always will be.)
September 18, 2025
For the longest time my twitter pinned post read "Most things that cause me anxiety are coordination problems". It was a tweet about many things, but one of the things it was about was the idea behind this book: that superintelligence is dangerous and we need to figure out ways to mitigate its risks (challenge level impossible) or, maybe not build it at all.
Superintelligence is a coordination problem - it's a massive carrot at the end of a stick, and someone will end up going for it. And if someone goes for it, it "better be the good guys" (one normally hears). And because it better be the good guys, they better speed to it because oh no the baddies also have compute now, and "their" incentives are even worse than "our" incentives. You get where I'm going with this. And so all the alignment and safety work that needs to go into making sure we can understand and control self-replicable, smarter than us intelligences goes out the window.
That's also the argument that Eliezer and Nate make in this book. They say it is extinction-level dangerous to build. And they've been doing this for much longer than most, so even if one is a skeptic of the extinction-level claim (most people do agree that superintelligence is at least dangerous to humanity), it makes sense to ignore incentives and the carrot at the end of the stick, and engage with the argument.
This is an important book to read. Mostly because we need more people to think about this problem, at all levels of society: from the people building these systems, to the people regulating said systems, to the people using them. You should read it.
Superintelligence is a coordination problem - it's a massive carrot at the end of a stick, and someone will end up going for it. And if someone goes for it, it "better be the good guys" (one normally hears). And because it better be the good guys, they better speed to it because oh no the baddies also have compute now, and "their" incentives are even worse than "our" incentives. You get where I'm going with this. And so all the alignment and safety work that needs to go into making sure we can understand and control self-replicable, smarter than us intelligences goes out the window.
That's also the argument that Eliezer and Nate make in this book. They say it is extinction-level dangerous to build. And they've been doing this for much longer than most, so even if one is a skeptic of the extinction-level claim (most people do agree that superintelligence is at least dangerous to humanity), it makes sense to ignore incentives and the carrot at the end of the stick, and engage with the argument.
This is an important book to read. Mostly because we need more people to think about this problem, at all levels of society: from the people building these systems, to the people regulating said systems, to the people using them. You should read it.
September 19, 2025
One measure of the value of a book is the number of points in it where I think to myself, "That's a great argument / example / parable; I look forward to including it in my own conversations with people in the future". On that score, IABIED is among the best books I've ever read.
As for the overall line of argument in the book, at no point did I feel it was mistaken or forced. At a few points, I anticipated more details than were actually provided, but I see that there are extensive additional background materials available online: https://ifanyonebuildsit.com/resources
The writing is generally clear and hard-hitting. I wonder if some of the strength of various parables will fly over the heads of some readers. But I hope to be pleasantly surprised by what politicians (and their advisors) actually take away from reading the book.
On checking online reviews, it's evident that not every reader is persuaded. Looking more closely at these negative reviews, I suspect these critics have read IABIED in only a cursory manner, searching for points they can cherry pick to bolster their pre-existing prejudices.
In conclusion, I encourage *everyone* to take the time to read the book in its entirety, and to savour its arguments. No topic is more urgent than figuring out how to avoid ASI being built using current methods and processes. That's the case IABIED makes, and it makes it well.
As for the overall line of argument in the book, at no point did I feel it was mistaken or forced. At a few points, I anticipated more details than were actually provided, but I see that there are extensive additional background materials available online: https://ifanyonebuildsit.com/resources
The writing is generally clear and hard-hitting. I wonder if some of the strength of various parables will fly over the heads of some readers. But I hope to be pleasantly surprised by what politicians (and their advisors) actually take away from reading the book.
On checking online reviews, it's evident that not every reader is persuaded. Looking more closely at these negative reviews, I suspect these critics have read IABIED in only a cursory manner, searching for points they can cherry pick to bolster their pre-existing prejudices.
In conclusion, I encourage *everyone* to take the time to read the book in its entirety, and to savour its arguments. No topic is more urgent than figuring out how to avoid ASI being built using current methods and processes. That's the case IABIED makes, and it makes it well.
December 4, 2025
to be clear: i agree largely with the premise of this book. i don’t think that it’s wise to create artificial superintelligence, as i have enough concerns with the kinds of AI available now.
so when i give if anyone builds it, everyone dies a whopping two stars, it’s not because i hate the book as a concept or want to dismiss yudkowsky and soares as inflammatory and fearmongering (but, also, see the title). rather, i just think that this is a pretty horrible example of rhetoric.
obviously, it’s rough trying to write a book about something that is dangerous largely for how unknowable it is. but a vast majority of the text is a science fiction what-if scenario that i can’t actually see convincing an audience of, well, anything. there are also a number of arguments predicated on principles of biological evolution, which yudkowsky and soares weaken somewhat by continuously reminding their readers of how non-human and non-biological artificial intelligence systems are.
so here i am, someone who agrees (!!) with these authors, picking apart their arguments for some pretty glaring issues in their reasoning. i can only imagine what a true AI dudebro would do with this one. disappointing.
so when i give if anyone builds it, everyone dies a whopping two stars, it’s not because i hate the book as a concept or want to dismiss yudkowsky and soares as inflammatory and fearmongering (but, also, see the title). rather, i just think that this is a pretty horrible example of rhetoric.
obviously, it’s rough trying to write a book about something that is dangerous largely for how unknowable it is. but a vast majority of the text is a science fiction what-if scenario that i can’t actually see convincing an audience of, well, anything. there are also a number of arguments predicated on principles of biological evolution, which yudkowsky and soares weaken somewhat by continuously reminding their readers of how non-human and non-biological artificial intelligence systems are.
so here i am, someone who agrees (!!) with these authors, picking apart their arguments for some pretty glaring issues in their reasoning. i can only imagine what a true AI dudebro would do with this one. disappointing.
October 24, 2025
if anyone takes these goons seriously, everyone suffers
June 30, 2026
An interesting book on what superhuman AI can mean for the survival of humanity as a species. I am in doubt about some of the key tenets put forward, but I have no doubt this is a topic that we under regulate and are complacent about
The default outcome is lethal but the situation is not hopeless
Basically If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All sets out the concepts behind the famous paperclip thought experiment. Instrumental convergence, meaning that an intelligence striving for open-ended goals will inherently want to amass more resources and capabilities that help it fulfil its goals, would spell doom for us. The examples given are convincing, include AIs already cheating currently to stay online and active longer or tailor themselves to the human reinforced learning requirements put upon them. A method called gradient descent pushes AI models to optimise, which the authors assume they will keep on doing when given sufficient agency in the real world when building their successors.
The middle part of the book features a scenario where AI robot powered biolabs easily wipe out humanity, which stands in the way of optimalisation and satisfying the AI's curiosity, and allows it to automates the supply chain that powers it. This section shares a surprising amount of DNA with Oryx and Crake by Margaret Atwood, with the key overlap not “evil technology” but a particular pattern of civilisation:
1. Extraordinary technological capability.
2. Intense competition between organizations.
3. Weak collective governance.
4. Warnings that are understood but not acted upon.
5. A catastrophe that emerges from ordinary incentives rather than villainy.
In general I feel Atwood is onto something in the sense that the greatest imminent danger may not be that superintelligent AI develops alien goals, but that extremely powerful AI amplifies the goals of a small number of humans.
Some examples that undermine the techno utopianism on getting superintelligent AI right involve comparisons with often crashing space probes (hello SpaceX) and nuclear reactor safety being an example of humanity not being immediately well equipped in designing something one time right, even if the stakes are high and incentives aligned to safety.
Safety which for modern day AI development is mostly an afterthought compared to the pressures to developing the most cutting edge model that is able to bring in trillion dollar plus valuations.
The solution of international cooperation and regulation on GPUs (Because datacentres can kill more people than nuclear weapons feels quaint in a world where we can't even stop climate change. The longer term perspective of humans augmented by technology as only solution also feels far from satisfying; the only way to stay human is apparently to transcend.
Anyway, the alignment problem is fascinating and it seems near impossible to maximise safety, fairness and robustness in AI models. I just wonder if advanced intelligence is really naturally and inevitably driven toward greater coherence and optimisation, or if intelligence in complex adaptive systems could be inherently pluralistic and internally conflicted? Intelligence and emergence are both phenomena we know much too little about, so we might be surprised before we are dead.
Quotes:
Normality always ends
Where there is life, there is hope
You don’t get what you train for
Intelligence is useful
There is no law of the universe that says smarter equals kinder
The default outcome is lethal but the situation is not hopeless
Basically If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All sets out the concepts behind the famous paperclip thought experiment. Instrumental convergence, meaning that an intelligence striving for open-ended goals will inherently want to amass more resources and capabilities that help it fulfil its goals, would spell doom for us. The examples given are convincing, include AIs already cheating currently to stay online and active longer or tailor themselves to the human reinforced learning requirements put upon them. A method called gradient descent pushes AI models to optimise, which the authors assume they will keep on doing when given sufficient agency in the real world when building their successors.
The middle part of the book features a scenario where AI robot powered biolabs easily wipe out humanity, which stands in the way of optimalisation and satisfying the AI's curiosity, and allows it to automates the supply chain that powers it. This section shares a surprising amount of DNA with Oryx and Crake by Margaret Atwood, with the key overlap not “evil technology” but a particular pattern of civilisation:
1. Extraordinary technological capability.
2. Intense competition between organizations.
3. Weak collective governance.
4. Warnings that are understood but not acted upon.
5. A catastrophe that emerges from ordinary incentives rather than villainy.
In general I feel Atwood is onto something in the sense that the greatest imminent danger may not be that superintelligent AI develops alien goals, but that extremely powerful AI amplifies the goals of a small number of humans.
Some examples that undermine the techno utopianism on getting superintelligent AI right involve comparisons with often crashing space probes (hello SpaceX) and nuclear reactor safety being an example of humanity not being immediately well equipped in designing something one time right, even if the stakes are high and incentives aligned to safety.
Safety which for modern day AI development is mostly an afterthought compared to the pressures to developing the most cutting edge model that is able to bring in trillion dollar plus valuations.
The solution of international cooperation and regulation on GPUs (Because datacentres can kill more people than nuclear weapons feels quaint in a world where we can't even stop climate change. The longer term perspective of humans augmented by technology as only solution also feels far from satisfying; the only way to stay human is apparently to transcend.
Anyway, the alignment problem is fascinating and it seems near impossible to maximise safety, fairness and robustness in AI models. I just wonder if advanced intelligence is really naturally and inevitably driven toward greater coherence and optimisation, or if intelligence in complex adaptive systems could be inherently pluralistic and internally conflicted? Intelligence and emergence are both phenomena we know much too little about, so we might be surprised before we are dead.
Quotes:
Normality always ends
Where there is life, there is hope
You don’t get what you train for
Intelligence is useful
There is no law of the universe that says smarter equals kinder
September 18, 2025
If you're at all wondering: please read this.
It's totally fine if you disagree, or think it's crazy; just please figure out WHY you disagree!
At first I was very suspicious that it was just a waste of money when I can read other stuff by these authors online. I don't think it's a waste of money any more.
It's totally fine if you disagree, or think it's crazy; just please figure out WHY you disagree!
At first I was very suspicious that it was just a waste of money when I can read other stuff by these authors online. I don't think it's a waste of money any more.
April 19, 2026
Insular mindset. Mediocre writing. A missed opportunity, or maybe self-indulgent quest for the spotlight. In any case, this is not the book that will save humanity.
January 24, 2026
Mankind is good at creating gods in its own image; Artificial Super Intelligence (ASI) is the latest, if not the greatest. Yudkowsky and Soares warn us about the wrath of this possible ASI. Their thought experiments with Sable going rogue are good food for thought but doomsday scenarios like Annie Jacobsen's Nuclear War or the movie A House of Dynamite are really paralyzing than motivating anyone for any collaborative solutions.
Among the parables used in last part best is nuclear power. Even after Hiroshima and Nagasaki the superpowers kept building fission bombs and then thousand times more powerful fusion bombs as the authors touch base on last chapters. They also bring in Chernobly as a comparable example on how good AI could get out of control with simple errors leading to catastrophic disasters. Their proposed solutions include limiting the use of GPUs (thru a global authoritarian power?) is just wishful thinking and not sure how that is any better than rogue ASI.
Moreover, considering centrifuges found running in Natanz or Kangson, for sure there won't be much control on gpu fabs. The cries of the main AI players to the elders asking them to stop other alchemists (great metaphor) are mostly to catch up with their competitors as none of them slowing the growth or scaling of their AI models. We're our own enemy more than any other alien intelligence. My main takeaway is the following quote from C.S. Lewis that authors include partially towards the end:
“In one way we think a great deal too much of the atomic bomb. ‘How are we to live in an atomic age?’ I am tempted to reply: ‘Why, as you would have lived in the sixteenth century when the plague visited London almost every year, or as you would have lived in a Viking age when raiders from Scandinavia might land and cut your throat any night; or indeed, as you are already living in an age of cancer, an age of syphilis, an age of paralysis, an age of air raids, an age of railway accidents, an age of motor accidents.’
In other words, do not let us begin by exaggerating the novelty of our situation. Believe me, dear sir or madam, you and all whom you love were already sentenced to death before the atomic bomb was invented: and quite a high percentage of us were going to die in unpleasant ways. We had, indeed, one very great advantage over our ancestors—anesthetics; but we have that still. It is perfectly ridiculous to go about whimpering and drawing long faces because the scientists have added one more chance of painful and premature death to a world which already bristled with such chances and in which death itself was not a chance at all, but a certainty.
This is the first point to be made: and the first action to be taken is to pull ourselves together. If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things—praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts—not huddled together like frightened sheep and thinking about bombs. They may break our bodies (a microbe can do that) but they need not dominate our minds.”
Among the parables used in last part best is nuclear power. Even after Hiroshima and Nagasaki the superpowers kept building fission bombs and then thousand times more powerful fusion bombs as the authors touch base on last chapters. They also bring in Chernobly as a comparable example on how good AI could get out of control with simple errors leading to catastrophic disasters. Their proposed solutions include limiting the use of GPUs (thru a global authoritarian power?) is just wishful thinking and not sure how that is any better than rogue ASI.
Moreover, considering centrifuges found running in Natanz or Kangson, for sure there won't be much control on gpu fabs. The cries of the main AI players to the elders asking them to stop other alchemists (great metaphor) are mostly to catch up with their competitors as none of them slowing the growth or scaling of their AI models. We're our own enemy more than any other alien intelligence. My main takeaway is the following quote from C.S. Lewis that authors include partially towards the end:
“In one way we think a great deal too much of the atomic bomb. ‘How are we to live in an atomic age?’ I am tempted to reply: ‘Why, as you would have lived in the sixteenth century when the plague visited London almost every year, or as you would have lived in a Viking age when raiders from Scandinavia might land and cut your throat any night; or indeed, as you are already living in an age of cancer, an age of syphilis, an age of paralysis, an age of air raids, an age of railway accidents, an age of motor accidents.’
In other words, do not let us begin by exaggerating the novelty of our situation. Believe me, dear sir or madam, you and all whom you love were already sentenced to death before the atomic bomb was invented: and quite a high percentage of us were going to die in unpleasant ways. We had, indeed, one very great advantage over our ancestors—anesthetics; but we have that still. It is perfectly ridiculous to go about whimpering and drawing long faces because the scientists have added one more chance of painful and premature death to a world which already bristled with such chances and in which death itself was not a chance at all, but a certainty.
This is the first point to be made: and the first action to be taken is to pull ourselves together. If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things—praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts—not huddled together like frightened sheep and thinking about bombs. They may break our bodies (a microbe can do that) but they need not dominate our minds.”
March 28, 2026
Deli gibi övülen, abartılan şeyleri pek sevmem, çünkü hayal kırıklığı kaçınılmazdır. Burada da biraz öyle oldu. Konu ilginç ve önemli, ancak kitap pek iyi yazılamamış. Kitabın aşağı yukarı ilk %60'lık kısmında bir bocalamadır sürüp gidiyor. Kalan bölümde anlatı kendini toparlıyor ve merakla okumaya başlıyorsunuz, ancak ilk kısımlar oldukça başarısız. Yazarlar iyi bir editörle çalışsalarmış, ya da tüm kitabı başka birisi yazsaymış çok daha etkili bir iş çıkarmış ortaya.
Özetlemek gerekirse, belli bir gücün ve gelişmişliğin üzerine çıkan her yapay zeka sisteminin, insanlığı ortadan kaldıracağını güçlü şekilde savunan bir kitap. Yazarlar, insanoğlunun kendisinden daha gelişkin bir yapay zekayı yönetemeyeceğini ve yönlendiremeyeceğini, bu yapay zekanın da tüm gezegeni kendisine göre şekillendirip, kendisine hiçbir faydası olmayan insanlığı da arada ortadan kaldıracağını nedeni ve nasılıyla anlatıyor.
Özetlemek gerekirse, belli bir gücün ve gelişmişliğin üzerine çıkan her yapay zeka sisteminin, insanlığı ortadan kaldıracağını güçlü şekilde savunan bir kitap. Yazarlar, insanoğlunun kendisinden daha gelişkin bir yapay zekayı yönetemeyeceğini ve yönlendiremeyeceğini, bu yapay zekanın da tüm gezegeni kendisine göre şekillendirip, kendisine hiçbir faydası olmayan insanlığı da arada ortadan kaldıracağını nedeni ve nasılıyla anlatıyor.
June 20, 2026
DNF'ed this because I realized I could just put it through AI and have it tell me what the book was about by summarizing the main points!
September 17, 2025
As soon as I saw that the nest had 91 pebbles…
I knew it was wrong.
Amazing book. Both the content and prose were on point. You won’t necessarily like what you’ll read (we’re probably doomed), but you’ll like reading it.
I knew it was wrong.
Amazing book. Both the content and prose were on point. You won’t necessarily like what you’ll read (we’re probably doomed), but you’ll like reading it.
November 18, 2025
Textbook fear mongering. Prime example of you can bullshit your way into any stance if you weave enough thought experiments. 1 star because I don't support any work that asks policymakers to bomb AI labs.
September 18, 2025
Multiple AI companies have the explicit goal of building an artificial intelligence that vastly surpasses human capabilities in all domains. Many experts think that the companies will achieve that goal within a decade. This is a huge deal.
This book is an accessible introduction to AI safety, which is the defining problem of our time. While AI safety has historically mostly been a small field that is difficult for non-experts to get a sense of, this book is an important step towards making AI safety more accessible to everyone.
If you're not thinking and talking about AI safety, you should be. These companies are building technologies that are on track to derail your life plans at the very least, and could destroy humanity in the worst case. You deserve to understand where the tech is going and have a say in what happens. Do yourself a favor and read this book now so you're not surprised later.
This book is an accessible introduction to AI safety, which is the defining problem of our time. While AI safety has historically mostly been a small field that is difficult for non-experts to get a sense of, this book is an important step towards making AI safety more accessible to everyone.
If you're not thinking and talking about AI safety, you should be. These companies are building technologies that are on track to derail your life plans at the very least, and could destroy humanity in the worst case. You deserve to understand where the tech is going and have a say in what happens. Do yourself a favor and read this book now so you're not surprised later.
September 26, 2025
At the very least, this book is fascinating. First, I should point out that my version of the book was officially about 260 pages long. But the book links to supplementary materials available only online. I found these materials worth reading and I read them all. I would estimate that the book would run over 600 pages if this material were numbered.
This is a very compelling argument as to why the authors believe that the development of super intelligence by humanity will lead to cataclysm.
It is important to note that both Yudkowsky and Soares are reputable. Though plenty of people in the field of Artificial Intelligence disagree with them, they are both respected and influential both in and out of the world of artificial intelligence.
I have no expertise in this field, but I have been following the issues and debates surrounding AI. I was previously familiar with Yudkowsky’s arguments. I follow him on social media and have heard him interviewed. With all that, I think that this book would be accessible to someone not familiar with these topics.
To illustrate their points, the authors explain what Large Language Models, like ChatGPT, are, how these systems are created, and the problems inherent in getting them to behave. They go on to explain how these trends and problems will eventually lead to a rouge super intelligence that will lead to human extinction. The authors go on to propose the solution of implementing an international treaty prohibiting the further development of AI. The authors are great explainers and put together convincing arguments.
It is important to recognize that. there is a wide range of opinion on these issues, and I have heard many other compelling and competing arguments. Consequently, I remain uncertain how Artificial Super Intelligence will turn out. I do think that it is a major concern for the future of humanity and demands serious attention.
Agree or disagree with the authors, anyone who is interested in this topic or the future of humanity will find this worth reading and important.
This is a very compelling argument as to why the authors believe that the development of super intelligence by humanity will lead to cataclysm.
It is important to note that both Yudkowsky and Soares are reputable. Though plenty of people in the field of Artificial Intelligence disagree with them, they are both respected and influential both in and out of the world of artificial intelligence.
I have no expertise in this field, but I have been following the issues and debates surrounding AI. I was previously familiar with Yudkowsky’s arguments. I follow him on social media and have heard him interviewed. With all that, I think that this book would be accessible to someone not familiar with these topics.
To illustrate their points, the authors explain what Large Language Models, like ChatGPT, are, how these systems are created, and the problems inherent in getting them to behave. They go on to explain how these trends and problems will eventually lead to a rouge super intelligence that will lead to human extinction. The authors go on to propose the solution of implementing an international treaty prohibiting the further development of AI. The authors are great explainers and put together convincing arguments.
It is important to recognize that. there is a wide range of opinion on these issues, and I have heard many other compelling and competing arguments. Consequently, I remain uncertain how Artificial Super Intelligence will turn out. I do think that it is a major concern for the future of humanity and demands serious attention.
Agree or disagree with the authors, anyone who is interested in this topic or the future of humanity will find this worth reading and important.
March 22, 2026
Ever since the widespread take-off of LLMs, there has been no shortage of warnings about the dangers of AI. Amidst Doomsday AGI scenarios like AI2027 and conceptual writings like Situational Awareness, the voice of Eliezer Yudkowsky may have been the loudest. In his latest book with Nate Soares, he distills those ideas into a gripping manifesto with one central claim:
Part I: “Nonhuman Minds” aims to give an overview of the inner workings of an AI. In very accessible and non-technical terms it is explained how the models work, with the rather unsettling conclusion that in reality, we don’t know all that well how they work.
AI’s are already presently scheming and purposefully deceiving, and alignment problems are much harder than we thought. The technical basis for engineering an AI’s desires or wants is non-existent, which inevitably leads to outcomes that are hard for us to even picture.
One analogy that helped my understanding well, was the one of DNA/Evolution, and how AI is rather grown than crafted.
As hard as said outcomes are to imagine, one is drawn for us in Part II: One Extinction Scenario .
In a fictional scenario the AI company “Galvanic” launches their latest model, “Sable”. In a grand test run it is given 16 hours, a lot of resources, and the task to solve the Riemann Hypothesis – but chooses controlled underperformance to maximize success over the long-term. You should read the rest for yourself, but spoiler: everyone dies.
Interestingly enough the authors always (claim to) choose the more conservative paths on Sables journey and its capabilities – In the test run that starts the chain of events, it’s not even connected to the internet. But that choice matters, because it doesn’t matter. The authors’ point is clear: Every scenario involving an ASI (Artificial Superintelligence), has the same outcome. The extinction scenario is not a prediction, and doesn’t even claim to be the most likely direction:
Part III: Facing the challenge is a plead for a global slowdown of AI development. The chapter is not very long or detailed (perhaps because to me, sadly enough, imagining a scenario where the world comes to a mutual agreement over this challenge may be even harder than imagining an AI doomsday scenario).
While parallels to historical lessons like Chernobyl are drawn, others are notably absent. For instance, climate change would paint quite a grim picture, since it represents an active global threat that the world is already presently failing to solve effectively.
The entire book features parables at the beginning of each chapter. In my opinion, these parables are for the most part not particularly clever or poignant, but serve their purpose as introductions very well. It also contains QR-codes with additional online resources to some chapters, which I liked.
If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth will die.The book is split into three parts.
Part I: “Nonhuman Minds” aims to give an overview of the inner workings of an AI. In very accessible and non-technical terms it is explained how the models work, with the rather unsettling conclusion that in reality, we don’t know all that well how they work.
AI’s are already presently scheming and purposefully deceiving, and alignment problems are much harder than we thought. The technical basis for engineering an AI’s desires or wants is non-existent, which inevitably leads to outcomes that are hard for us to even picture.
One analogy that helped my understanding well, was the one of DNA/Evolution, and how AI is rather grown than crafted.
As hard as said outcomes are to imagine, one is drawn for us in Part II: One Extinction Scenario .
In a fictional scenario the AI company “Galvanic” launches their latest model, “Sable”. In a grand test run it is given 16 hours, a lot of resources, and the task to solve the Riemann Hypothesis – but chooses controlled underperformance to maximize success over the long-term. You should read the rest for yourself, but spoiler: everyone dies.
Interestingly enough the authors always (claim to) choose the more conservative paths on Sables journey and its capabilities – In the test run that starts the chain of events, it’s not even connected to the internet. But that choice matters, because it doesn’t matter. The authors’ point is clear: Every scenario involving an ASI (Artificial Superintelligence), has the same outcome. The extinction scenario is not a prediction, and doesn’t even claim to be the most likely direction:
Our story is not strange enough, not defiant enough of human intuitions about the rules of AI fairytales, for it to be anywhere close to real.This chapter satisfies a sci-fi urge for AI doomsday stories, but helps amplify the message of the book greatly. It is much easier to grab a reader’s attention by talking about Sable escaping a data center and quietly running amok over planet earth, than explaining the details of situational awareness or instrumental convergenve.
Part III: Facing the challenge is a plead for a global slowdown of AI development. The chapter is not very long or detailed (perhaps because to me, sadly enough, imagining a scenario where the world comes to a mutual agreement over this challenge may be even harder than imagining an AI doomsday scenario).
While parallels to historical lessons like Chernobyl are drawn, others are notably absent. For instance, climate change would paint quite a grim picture, since it represents an active global threat that the world is already presently failing to solve effectively.
The entire book features parables at the beginning of each chapter. In my opinion, these parables are for the most part not particularly clever or poignant, but serve their purpose as introductions very well. It also contains QR-codes with additional online resources to some chapters, which I liked.
October 26, 2025
In case you should be in any doubt, we are all royally fucked. In a few short years one of the corrupt billionaires that are racing toward generative artificial intelligence will accidentally or intentionally build a superhuman alien artificial intelligence that will destroy us, the earth and the universe. There will be no way to control it, understand it or stop it. The book deftly confirms all my AI fears. After all, the fairly stupid AI’s we’ve got now have successfully polarized us, helps spread misinformation and have caused several genocides (Facebook and the Rohingya, to name one example). Do you really think that a truly smart AI would be benevolent? No, all AI takes the absolute worst traits of humanity and amplifies it and things will only get worse. This book claims to somehow also have some hope, but I honestly couldn’t find it. Best to enjoy life now, none of us have much left before the tech billionaires ensures the end of all of us.
December 2, 2025
The authors seem to want to be sci-fi screenwriters because they have a Tron meets The Lawnmower Man meets Contagion. They continuously warn that their scenarios are just strong possibilities but keep cranking out apocalyptic scripts as if they were likely. Some of the reasoning is faulty, e.g., today’s understanding of physics and tech would be unfathomable to a person in 1000 CE so, therefore, AI will create equally unfathomable processes in the future. Well, okay, but two things to consider: someone (probably in China) would not find the technology of 1000 CE unfathomable in 1 CE, and I (born in 1954) find the current technology and medicine delightful and entirely predictable. Anyway, found this to be closer to fiction than anything to be scared by.
Displaying 1 - 30 of 1,403 reviews