More on this book
Community
Kindle Notes & Highlights
At some point, a group of atoms became arranged into a complex pattern that could both maintain and replicate itself.
Bit of a leap!
What’s replicated isn’t matter (made of atoms) but information (made of bits) specifying how the atoms are arranged. When a bacterium makes a copy of its DNA, no new atoms are created, but a new set of atoms are arranged in the same pattern as the original, thereby copying the information. In other words, we can think of life as a self-replicating information-processing system whose information (software) determines both its behavior and the blueprints for its hardware.
Life forms summarized
Relevant questions
This substrate independence of computation implies that AI is possible: intelligence doesn’t require flesh, blood or carbon atoms.
AI researchers have often been accused of over-promising and under-delivering, but in fairness, some of their critics don’t have the best track record either. Some keep moving the goalposts, effectively defining intelligence as that which computers still can’t do, or as that which impresses us.
Winograd Schema Challenge
Differences are even more extreme internationally where, in 2013, the combined wealth of the bottom half of the world’s population (over 3.6 billion people) is the same as that of the world’s eight richest people43—a statistic that highlights the poverty and vulnerability at the bottom as much as the wealth at the top.
So what career advice should we give our kids?
A friend of mine recently joked with me that perhaps the very last profession will be the very first profession: prostitution. But then he mentioned this to a Japanese roboticist, who protested: “No, robots are very good at those things!”
Ha! Five year old example
Yet all these scenarios have two features in common: 1. A fast takeoff: the transition from subhuman to vastly superhuman intelligence occurs in a matter of days, not decades. 2. A unipolar outcome: the result is a single entity controlling Earth.
This can be remedied by building a Gatekeeper, a superintelligence with the goal of interfering as little as necessary to prevent the creation of another superintelligence.
Theodicy problem
If in the distant future our cosmos has been settled by high-tech zombie AIs, then it doesn’t matter how fancy their intergalactic architecture is: it won’t be beautiful or meaningful, because there’s nobody and nothing to experience it—it’s all just a huge and meaningless waste of space.
With our present level of intelligence and emotional maturity, we humans have a knack for miscalculations, misunderstandings and incompetence, and as a result, our history is full of accidents, wars and other calamities that, in hindsight, essentially nobody wanted.
In other words, the second law of thermodynamics has a life loophole: although the total entropy must increase, it’s allowed to decrease in some places as long as it increases even more elsewhere. So life maintains or increases its complexity by making its environment messier.
In other words, the real risk with AGI isn’t malice but competence. A superintelligent AI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble. As I mentioned in chapter 1, people don’t think twice about flooding anthills to build hydroelectric dams, so let’s not place humanity in the position of those ants.
From my perspective, a conscious person is simply food, rearranged.
Ha!
Andy
The Asilomar AI Principles
My experiences over the past few years have increased my optimism for two separate reasons. First, I’ve witnessed the AI community come together in a remarkable way to constructively take on the challenges ahead, often in collaboration with thinkers from other fields.
The second reason I’ve grown more optimistic is that the FLI experience has been empowering. What had triggered my London tears was a feeling of inevitability: that a disturbing future may be coming and there was nothing we could do about it. But the next three years dissolved my fatalistic gloom.
For reasons I’ll soon explain, I think that a great first step is working on becoming a mindful optimist, if you aren’t one already. To be a successful mindful optimist, it’s crucial to develop positive visions for the future.
In summary, if we can create a more harmonious human society characterized by cooperation toward shared goals, this will improve the prospects of the AI revolution ending well.
In other words, we're doomed
Of course you can vote at the ballot box and tell your politicians what you think about education, privacy, lethal autonomous weapons, technological unemployment and other issues. But you also vote every day through what you choose to buy, what news you choose to consume, what you choose to share and what sort of role model you choose to be.
