Joseph’s review of Superintelligence: Paths, Dangers, Strategies > Likes and Comments
157 likes · Like
http://waitbutwhy.com/2015/01/artific...
I actually read that article at about the same time I was reading Superintelligence. And frankly I had the same problems with it that I had in the book, plus some extra annoyance with the whole idea of viewing something as vague and non-quantifiable as "human progress" as a simple exponential.
And again, I understand the argument for recursive self-improvement, it's just falling completely flat as a compelling argument since there's currently no ideas on the table for how recursive self-improvement is going to work. That would also require better metrics for intelligence, and a host of other things that make this argument almost the perfect example of what I meant in section II of my review. Not to mention that it also ascribes godlike powers to recursively self-improving AGIs that I don't find particularly persuasive.
Did you think there was something in it I missed? If so, could you point it out?
Ok. No, I think we just disagree. I guess we'll find out if we get to live for another 50 years, which in my case would be a superjackpot.
Right on. Thanks to your precision and thoroughness here, I don't have to try to write the same review.
We need an unike button. A typical freshman comment where the broad picture is missed for irrelevant self gloating.
The idea is to replace the human software developer. Since humans generally don't understand how their brains do what they do, it's hard for the human practitioner to guess when a computer could do the same things.
Similarly, everything I know about speaking English is irrelevant to guessing how long it will take for computers to speak and understand English as well as I do. I have no real idea of how I do this, so I have no idea whether computers will do it next year or in 100 years. My ability to do something doesn't give me any special insight into how I do it, or how a computer might do it.
But the natural language people have made some progress. Already we have Cortana and Siri as consumer products with some rudimentary conversational ability. When computers achieve some real conversational fluency, they might have crossed a threshold that might then spill into other domains, such as getting a computer to analyze and improve its own code.
The point is that you can't extrapolate from past experience to what happens after some threshold gets passed. By analogy, picture a yardstick sitting on a tabletop. You can push the yardstick slowly off the edge. The yardstick stays horizontal at first. Nothing changes as you keep pushing the yardstick, until you reach the point where almost half the yardstick is hanging over the edge. At that point, a very slight nudge causes a huge change as the yardstick topples over the edge, and all extrapolations from pushing the first half become invalid.
Another analogy is the earthquake that occurs after two tectonic plates build up stress by grinding and straining against each other for decades. Nothing much happens as the stress builds and builds, until some threshold of tensile strength in the rock gets passed and all the pent-up energy lets go at once.
As yet another analogy, before the invention of the telescope, progress in astronomy was very slow. Before astronomers had any idea the telescope was coming, they couldn't have guessed how it would speed things up.
With AI development there may be such thresholds lying in our future - maybe in our near future. This is a reasonable suspicion because we just passed a few. Not long ago people were saying we wouldn't have self-driving road vehicles, or a computer Go champion, for decades. Yet here they are.
Speaking for myself, I'll be skeptical about strong AI being just around the corner as long as we don't even have self-maintaining computers. Forget about computers achieving superintelligence - just give me a computer that runs and continually diagnoses and fixes all of its own errors, or at least tells me coherently how to fix whatever is breaking it. When computers stop needing human babysitters, then I'll start to worry. But this could happen at pretty much any time.
And because we don't know anything: we know it's coming soon! ;)
"Speaking for myself, I'll be skeptical about strong AI being just around the corner as long as we don't even have self-maintaining computers. Forget about computers achieving superintelligence - just give me a computer that runs and continually diagnoses and fixes all of its own errors, or at least tells me coherently how to fix whatever is breaking it. When computers stop needing human babysitters, then I'll start to worry. But this could happen at pretty much any time."
And that's pretty much my whole objection right there. Nobody knows how to build that, we don't really have any good ideas for how to build that, and unless we do get a crazy breakthrough out of nowhere I'm not optimistic that we'll see progress toward this ideal anytime soon.
I really, really do understand the arguments presented in the book, but until I can see somebody in software making progress toward actually having something self-maintaining built, I'm not buying any sort of immediacy to this problem. I don't begrudge people who do, but I have other things to worry about.
I don't think Bostrom claimed anywhere that Superintelligence was a "near-term existential threat". In fact he's in favor of delaying its onset as much as we can, until we figure out how to control it.
By the time AI stop needing babysitters, it’s too late.
So not only are we going to build a godlike AI, it will also be self-maintaining and self-sustaining, and a literal god. Yes, this sounds exactly like real software engineering work that real people do.
You’re just strawmanning and being rather uncivil.
Furthermore, you seem to have fixated on the idea that you believe ASI will take longer than Bostrom and the AI expert community estimates. Fine. Accurate long term predictions are nearly impossible. But that doesn’t mean we don’t need to work on this problem. AI alignment is an existential problem, whether in 25 years or 200. We should solve it as soon as possible so that when we do develop ASI, we’re safe.
I'm not at all convinced that ASI is even a thing, and the more I work with actual software systems that exist in the world, the more trying to solve any sort of "generic" alignment problem seems like an ill-defined waste of time. Nobody knows how to build this stuff, nobody knows how to interact with it, the space you need to work in is so broad that any truly generic solution is going to be pretty useless when applied to things that have a chance to actually exist.
I've wasted more time trying to get academic code to work on things in the real world than I care to reflect on, and those people are still trying to talk about machines that actually do things.
You’re falling prey to WYSIATI (What You See Is All There Is). You’re an accomplished software engineer who deals with boots-on-the-ground projects. Both the implementation and verification of ASI seem far-fetched from your reference frame. If you spent more time trying to understand the core assumptions (orthogonality, bootstrapping, and alignment), you’ll understand the « danger » part more, and if you spent more time thinking about whether your probability assignments are reasonable for long term projection of what’s possible (see: Superforecasting, Black Swan), I think you might realize you’re way too confident in your negative assessment.
But yes, we can do things like prove an agent with a utility function will retain that utility function as it changes itself. That’s a fundamental result. We can build on basic results like that to provide more meaningful guarantees.
It seems strange that in your conclusion you say that you "remain pretty unconvinced of AI as a relatively near-term existential threat", when the book never actually argues that AGI is a "near-term existential threat". In fact, according to the polls of AI researchers referenced by Kaj Sotala and Stuart Armstrong, those concerned with AI-safety believe AGI will arrive later than most professors that teach the subject.
Babysitters? No, human designed controls - computers would likely be more self sustained were it not for intentionally designed controls. So you want a computer that updates and run diagnostics without prompts, behaves as it wishes at OS level? Do you think this is not possible? Of course it is! The question is do you really think this is what consumers want and what has driven development? How fast would there be an uproar over control and privacy - this thinking does not apply to self contained potential AI systems that are not designed in the same manner...
Thank you Joseph for a great review.
I haven't read the book but seen a lecture from Bostrom and thought that his arguments were shallow, full of unstated ridiculous assumptions and not worth caring about. His book was mentioned frequently by the media, so wondered if maybe his writing is better, but as I suspected, it's not worth reading.
I haven't met any serious person in the field that believes in "strong"-AI, usually it's people that have a shallow interest sparked from watching Hollywood movies, that are afraid of general/strong-AI.
Disclaimer, I have worked as a business analyst and project manager for a machine-learning startup for 4 years, so I have knowledge and interact daily with people in the field. The problem with most of the fears, is the assumption of a conscious machine, which is not something that will happen. Also seems like most of the scaremongers don't understand that even if something is technically possible to do in theory, if it's economically unlikely, it will not happen and that is the case when it comes to this "super-intelligence".
Samu wrote: "Thank you Joseph for a great review.
I haven't read the book but seen a lecture from Bostrom and thought that his arguments were shallow, full of unstated ridiculous assumptions and not worth cari..."
I agree, and that's my main criticism of Bostrom and co. I don't see any economic incentive of creating Strong-AI, and therefore, until such an incentive exists, I don't think that AI can be a major existential threat. But I wouldn't dismiss Bostrom so fast. He's trying to grapple with a problem that is inherently hard for humans to comprehend and navigate (superintellgence, higher-than-human intelligence), and so many of his arguments will look silly and shallow.
Using a pure practical near-term business-oriented methodology may not be sufficient to understand AI and its potential benefits/threats. That's like looking at an individual leaf on a tree, and thinking that you understand the forest.
His work is either the silliest or the most important philosophical work ever done. Only time will tell which one it is.
I mean, the point I'm making is that if Bostrom really did have some prescient takes on the future of AI/machine learning/intelligence, I'd expect him to sound like he knows what he's talking about when he talks about things as they are. But he doesn't, he doesn't have a grasp on how software is actually designed and built (independent of any business motive, just the bare facts about how tech works). He doesn't get how machine learning currently works, nor does he have an appreciation for the really big, intractable problems in software development. And ultimately he doesn't have a metric for intelligence that makes sense on its own nor when I try to apply it to the way I know intelligence already works. Too many of the things he needed to make explicit for his argument to be at all convincing are lazy handwaves, and none of the people I know who actually know how to create software take this stuff seriously because he doesn't seem to know what he's talking about. In short:
- I have no rational basis for believing that general intelligence as a concept makes sense, or that it even can increase to the level Bostrom and Yudkowsky et al. claim it can
- I have no reason to believe that recursive self-improvement with no human interaction is a thing we're anywhere close to seeing, if indeed it's feasible to begin with. Bostrom's arguments in support of it consist of the same I.J. Good quote everyone uses and a mathematical model too simple to be worthwhile as an argument
- climate change is going to kill us all first anyway, so if y'all wanna fight x-risk, get off your computer and plant a tree or something
Learn the lesson St. Anselm didn't: describing something as possible doesn't make it exist.
Super intelligence?
Recent successful applications of machine learning techniques have unfortunately lent an unearned veneer of credibility to the claims of people that lack a thorough grounding in computer science to theorize about AI. It is a present day incarnation of the medieval scholastic arguments about how many angels can dance on the head of a pin.
Computers are currently just tools we’re clumsily using for good or bad to amplify our existing capabilities. But popularity of “Looming AI Threat” as a genre seems to correlate with the building realization that we lack workable leavers to effect any meaningful change in our post capitalist societies.
If you’re truly concerned about the control problem, you would be directing your efforts to curtailing the negative impacts of the artificial construct of corporations. These have largely superseded the artificial construct of a nation state in their dominance over the consequential decisions that pose an imminent threat to our continued existence as species on this planet.
The problem of self-improving intelligent systems driven by goals that are utterly indifferent to human well-being is very real. But the threat is not awaiting us in the future, conditioned on unknown advances in computer science. It is embodied in the collective intelligence systems all around us that were divorced from moral and ethical principles. Climate change is the effect. The root cause of this existential risk to our civilization is the optimization strategy we have chosen to fully maximize the efficiency of the system: removing all safeties.
We have some intuitions on computability theory concerning singularity.
First, the Kleene fixed-point theorem https://en.wikipedia.org/wiki/Kleene%... proves that there are Turing Machines that can replicate itself with the same capacity or even with greater capacity (it is, recognizes more languages).
Second, this is not only applicable to Turing Machines but is actually applicable to hardware https://en.wikipedia.org/wiki/Von_Neu....
The following step is in the book. If a machine, that can interact with the environment, is only acting in an optimization manner, it could conclude that self-replicating itself would maximize his goals. Some examples of pure optimization out of control: https://deepmind.com/blog/article/Spe...
Now suppose the paper clip example, it has a box that counts the number of clips and a machine that produces it. It uses a reinforcement learning algorithm, so it has an exploitation step and an exploration step, in the exploitation it does what it learns, in the exploration, it tries something new, supposes that it can sample metaprograms from GPT-3 and one of this metaprograms is a method to produce a quine https://en.wikipedia.org/wiki/Quine_(... (by 1 and 2 it is possible) and also produce a clip, so it starts to self-replicating itself and getting more clips, this behavior generates a greater reward so in the exploitation phase it will keep continuously replicating itself while doing clips. End of the world?
I just finished the book and Bostrom says (on the last page) that an intelligence explosion could be decades into the future. So your complaint about the threat being painted out to be near-term is unfounded.
As for the oversimplification of the concepts, I don't think the average reader would want it any other way. It's not an AI textbook, after all.
Very interesting to read this thread in 2023!
"Whereof One Cannot Speak, Thereof One Must Request Funding"
An interesting review to read in Dec. 2024. Admittedly, I’m a low knowledge user of my (many) devices, but have had significant concerns about AI for years. Reading from someone with experience in the industry who doesn’t (or didn’t) share my concerns is fascinating and I am curious if the advances in the years since this book was published have changed the OPs opinion or perspective on AI/superintelligence.
back to top
date
newest »
newest »
message 1:
by
Tom LA
(new)
Jul 01, 2015 09:07PM
http://waitbutwhy.com/2015/01/artific...
reply
|
flag
I actually read that article at about the same time I was reading Superintelligence. And frankly I had the same problems with it that I had in the book, plus some extra annoyance with the whole idea of viewing something as vague and non-quantifiable as "human progress" as a simple exponential.And again, I understand the argument for recursive self-improvement, it's just falling completely flat as a compelling argument since there's currently no ideas on the table for how recursive self-improvement is going to work. That would also require better metrics for intelligence, and a host of other things that make this argument almost the perfect example of what I meant in section II of my review. Not to mention that it also ascribes godlike powers to recursively self-improving AGIs that I don't find particularly persuasive.
Did you think there was something in it I missed? If so, could you point it out?
Ok. No, I think we just disagree. I guess we'll find out if we get to live for another 50 years, which in my case would be a superjackpot.
Right on. Thanks to your precision and thoroughness here, I don't have to try to write the same review.
We need an unike button. A typical freshman comment where the broad picture is missed for irrelevant self gloating.
"Though in principle I can't think of anything that prevents the formation of some forms of superintelligence, everything I know about software development makes me think that any progress will be slow and gradual, occasionally punctuated with a new trick or two that allows for somewhat faster (but still gradual) increases in some domains."
The idea is to replace the human software developer. Since humans generally don't understand how their brains do what they do, it's hard for the human practitioner to guess when a computer could do the same things.
Similarly, everything I know about speaking English is irrelevant to guessing how long it will take for computers to speak and understand English as well as I do. I have no real idea of how I do this, so I have no idea whether computers will do it next year or in 100 years. My ability to do something doesn't give me any special insight into how I do it, or how a computer might do it.
But the natural language people have made some progress. Already we have Cortana and Siri as consumer products with some rudimentary conversational ability. When computers achieve some real conversational fluency, they might have crossed a threshold that might then spill into other domains, such as getting a computer to analyze and improve its own code.
The point is that you can't extrapolate from past experience to what happens after some threshold gets passed. By analogy, picture a yardstick sitting on a tabletop. You can push the yardstick slowly off the edge. The yardstick stays horizontal at first. Nothing changes as you keep pushing the yardstick, until you reach the point where almost half the yardstick is hanging over the edge. At that point, a very slight nudge causes a huge change as the yardstick topples over the edge, and all extrapolations from pushing the first half become invalid.
Another analogy is the earthquake that occurs after two tectonic plates build up stress by grinding and straining against each other for decades. Nothing much happens as the stress builds and builds, until some threshold of tensile strength in the rock gets passed and all the pent-up energy lets go at once.
As yet another analogy, before the invention of the telescope, progress in astronomy was very slow. Before astronomers had any idea the telescope was coming, they couldn't have guessed how it would speed things up.
With AI development there may be such thresholds lying in our future - maybe in our near future. This is a reasonable suspicion because we just passed a few. Not long ago people were saying we wouldn't have self-driving road vehicles, or a computer Go champion, for decades. Yet here they are.
Speaking for myself, I'll be skeptical about strong AI being just around the corner as long as we don't even have self-maintaining computers. Forget about computers achieving superintelligence - just give me a computer that runs and continually diagnoses and fixes all of its own errors, or at least tells me coherently how to fix whatever is breaking it. When computers stop needing human babysitters, then I'll start to worry. But this could happen at pretty much any time.
And because we don't know anything: we know it's coming soon! ;)
"Speaking for myself, I'll be skeptical about strong AI being just around the corner as long as we don't even have self-maintaining computers. Forget about computers achieving superintelligence - just give me a computer that runs and continually diagnoses and fixes all of its own errors, or at least tells me coherently how to fix whatever is breaking it. When computers stop needing human babysitters, then I'll start to worry. But this could happen at pretty much any time."And that's pretty much my whole objection right there. Nobody knows how to build that, we don't really have any good ideas for how to build that, and unless we do get a crazy breakthrough out of nowhere I'm not optimistic that we'll see progress toward this ideal anytime soon.
I really, really do understand the arguments presented in the book, but until I can see somebody in software making progress toward actually having something self-maintaining built, I'm not buying any sort of immediacy to this problem. I don't begrudge people who do, but I have other things to worry about.
I don't think Bostrom claimed anywhere that Superintelligence was a "near-term existential threat". In fact he's in favor of delaying its onset as much as we can, until we figure out how to control it.
By the time AI stop needing babysitters, it’s too late.
So not only are we going to build a godlike AI, it will also be self-maintaining and self-sustaining, and a literal god. Yes, this sounds exactly like real software engineering work that real people do.
You’re just strawmanning and being rather uncivil. Furthermore, you seem to have fixated on the idea that you believe ASI will take longer than Bostrom and the AI expert community estimates. Fine. Accurate long term predictions are nearly impossible. But that doesn’t mean we don’t need to work on this problem. AI alignment is an existential problem, whether in 25 years or 200. We should solve it as soon as possible so that when we do develop ASI, we’re safe.
I'm not at all convinced that ASI is even a thing, and the more I work with actual software systems that exist in the world, the more trying to solve any sort of "generic" alignment problem seems like an ill-defined waste of time. Nobody knows how to build this stuff, nobody knows how to interact with it, the space you need to work in is so broad that any truly generic solution is going to be pretty useless when applied to things that have a chance to actually exist.I've wasted more time trying to get academic code to work on things in the real world than I care to reflect on, and those people are still trying to talk about machines that actually do things.
You’re falling prey to WYSIATI (What You See Is All There Is). You’re an accomplished software engineer who deals with boots-on-the-ground projects. Both the implementation and verification of ASI seem far-fetched from your reference frame. If you spent more time trying to understand the core assumptions (orthogonality, bootstrapping, and alignment), you’ll understand the « danger » part more, and if you spent more time thinking about whether your probability assignments are reasonable for long term projection of what’s possible (see: Superforecasting, Black Swan), I think you might realize you’re way too confident in your negative assessment.But yes, we can do things like prove an agent with a utility function will retain that utility function as it changes itself. That’s a fundamental result. We can build on basic results like that to provide more meaningful guarantees.
It seems strange that in your conclusion you say that you "remain pretty unconvinced of AI as a relatively near-term existential threat", when the book never actually argues that AGI is a "near-term existential threat". In fact, according to the polls of AI researchers referenced by Kaj Sotala and Stuart Armstrong, those concerned with AI-safety believe AGI will arrive later than most professors that teach the subject.
Babysitters? No, human designed controls - computers would likely be more self sustained were it not for intentionally designed controls. So you want a computer that updates and run diagnostics without prompts, behaves as it wishes at OS level? Do you think this is not possible? Of course it is! The question is do you really think this is what consumers want and what has driven development? How fast would there be an uproar over control and privacy - this thinking does not apply to self contained potential AI systems that are not designed in the same manner...
Thank you Joseph for a great review. I haven't read the book but seen a lecture from Bostrom and thought that his arguments were shallow, full of unstated ridiculous assumptions and not worth caring about. His book was mentioned frequently by the media, so wondered if maybe his writing is better, but as I suspected, it's not worth reading.
I haven't met any serious person in the field that believes in "strong"-AI, usually it's people that have a shallow interest sparked from watching Hollywood movies, that are afraid of general/strong-AI.
Disclaimer, I have worked as a business analyst and project manager for a machine-learning startup for 4 years, so I have knowledge and interact daily with people in the field. The problem with most of the fears, is the assumption of a conscious machine, which is not something that will happen. Also seems like most of the scaremongers don't understand that even if something is technically possible to do in theory, if it's economically unlikely, it will not happen and that is the case when it comes to this "super-intelligence".
Samu wrote: "Thank you Joseph for a great review. I haven't read the book but seen a lecture from Bostrom and thought that his arguments were shallow, full of unstated ridiculous assumptions and not worth cari..."
I agree, and that's my main criticism of Bostrom and co. I don't see any economic incentive of creating Strong-AI, and therefore, until such an incentive exists, I don't think that AI can be a major existential threat. But I wouldn't dismiss Bostrom so fast. He's trying to grapple with a problem that is inherently hard for humans to comprehend and navigate (superintellgence, higher-than-human intelligence), and so many of his arguments will look silly and shallow.
Using a pure practical near-term business-oriented methodology may not be sufficient to understand AI and its potential benefits/threats. That's like looking at an individual leaf on a tree, and thinking that you understand the forest.
His work is either the silliest or the most important philosophical work ever done. Only time will tell which one it is.
I mean, the point I'm making is that if Bostrom really did have some prescient takes on the future of AI/machine learning/intelligence, I'd expect him to sound like he knows what he's talking about when he talks about things as they are. But he doesn't, he doesn't have a grasp on how software is actually designed and built (independent of any business motive, just the bare facts about how tech works). He doesn't get how machine learning currently works, nor does he have an appreciation for the really big, intractable problems in software development. And ultimately he doesn't have a metric for intelligence that makes sense on its own nor when I try to apply it to the way I know intelligence already works. Too many of the things he needed to make explicit for his argument to be at all convincing are lazy handwaves, and none of the people I know who actually know how to create software take this stuff seriously because he doesn't seem to know what he's talking about. In short:- I have no rational basis for believing that general intelligence as a concept makes sense, or that it even can increase to the level Bostrom and Yudkowsky et al. claim it can
- I have no reason to believe that recursive self-improvement with no human interaction is a thing we're anywhere close to seeing, if indeed it's feasible to begin with. Bostrom's arguments in support of it consist of the same I.J. Good quote everyone uses and a mathematical model too simple to be worthwhile as an argument
- climate change is going to kill us all first anyway, so if y'all wanna fight x-risk, get off your computer and plant a tree or something
Learn the lesson St. Anselm didn't: describing something as possible doesn't make it exist.
Super intelligence?Recent successful applications of machine learning techniques have unfortunately lent an unearned veneer of credibility to the claims of people that lack a thorough grounding in computer science to theorize about AI. It is a present day incarnation of the medieval scholastic arguments about how many angels can dance on the head of a pin.
Computers are currently just tools we’re clumsily using for good or bad to amplify our existing capabilities. But popularity of “Looming AI Threat” as a genre seems to correlate with the building realization that we lack workable leavers to effect any meaningful change in our post capitalist societies.
If you’re truly concerned about the control problem, you would be directing your efforts to curtailing the negative impacts of the artificial construct of corporations. These have largely superseded the artificial construct of a nation state in their dominance over the consequential decisions that pose an imminent threat to our continued existence as species on this planet.
The problem of self-improving intelligent systems driven by goals that are utterly indifferent to human well-being is very real. But the threat is not awaiting us in the future, conditioned on unknown advances in computer science. It is embodied in the collective intelligence systems all around us that were divorced from moral and ethical principles. Climate change is the effect. The root cause of this existential risk to our civilization is the optimization strategy we have chosen to fully maximize the efficiency of the system: removing all safeties.
We have some intuitions on computability theory concerning singularity.First, the Kleene fixed-point theorem https://en.wikipedia.org/wiki/Kleene%... proves that there are Turing Machines that can replicate itself with the same capacity or even with greater capacity (it is, recognizes more languages).
Second, this is not only applicable to Turing Machines but is actually applicable to hardware https://en.wikipedia.org/wiki/Von_Neu....
The following step is in the book. If a machine, that can interact with the environment, is only acting in an optimization manner, it could conclude that self-replicating itself would maximize his goals. Some examples of pure optimization out of control: https://deepmind.com/blog/article/Spe...
Now suppose the paper clip example, it has a box that counts the number of clips and a machine that produces it. It uses a reinforcement learning algorithm, so it has an exploitation step and an exploration step, in the exploitation it does what it learns, in the exploration, it tries something new, supposes that it can sample metaprograms from GPT-3 and one of this metaprograms is a method to produce a quine https://en.wikipedia.org/wiki/Quine_(... (by 1 and 2 it is possible) and also produce a clip, so it starts to self-replicating itself and getting more clips, this behavior generates a greater reward so in the exploitation phase it will keep continuously replicating itself while doing clips. End of the world?
I just finished the book and Bostrom says (on the last page) that an intelligence explosion could be decades into the future. So your complaint about the threat being painted out to be near-term is unfounded.As for the oversimplification of the concepts, I don't think the average reader would want it any other way. It's not an AI textbook, after all.
Very interesting to read this thread in 2023!
"Whereof One Cannot Speak, Thereof One Must Request Funding"
An interesting review to read in Dec. 2024. Admittedly, I’m a low knowledge user of my (many) devices, but have had significant concerns about AI for years. Reading from someone with experience in the industry who doesn’t (or didn’t) share my concerns is fascinating and I am curious if the advances in the years since this book was published have changed the OPs opinion or perspective on AI/superintelligence.
