More on this book
Community
Kindle Notes & Highlights
The fear of machines turning evil is another red herring. The real worry isn’t malevolence, but competence.
It’s natural for us to rate the difficulty of tasks relative to how hard it is for us humans to perform them, as in figure 2.1. But this can give a misleading picture
I just typed the word “word,” and my laptop represented it in its memory as the 4-number sequence 119 111 114 100, storing each of those numbers as 8 bits (it represents each lowercase letter by a number that’s 96 plus its order in the alphabet).
If you email your friend a document to print, the information may get copied in rapid succession from magnetizations on your hard drive to electric charges in your computer’s working memory, radio waves in your wireless network, voltages in your router, laser pulses in an optical fiber and, finally, molecules on a piece of paper.
our human DNA stores about 1.6 gigabytes,
In other words, the hardware is the matter and the software is the pattern. This substrate independence of computation implies that AI is possible: intelligence doesn’t require flesh, blood or carbon atoms.
“The city councilmen refused the demonstrators a permit because they feared violence.” “The city councilmen refused the demonstrators a permit because they advocated violence.”
that are being pursued around the world: verification, validation, security and control.
This was NASA’s second super-expensive bug: their Mariner 1 mission to Venus exploded after launch from Cape Canaveral on July 22, 1962, after the flight-control software was foiled by an incorrect punctuation mark.
Elon Musk argued that what we need right now from governments isn’t oversight but insight:
If it sounds far-off, consider that most of our economy is already owned by another form of non-human entity: corporations, which are often more powerful than any one person
not decades, and the stakes are high: autonomous weapons have been described as the third revolution in warfare, after gunpowder and nuclear arms.
John F. Kennedy emphasized when announcing the Moon missions that hard things are worth attempting when success will greatly benefit the future of humanity.
those who stand to gain most from an arms race aren’t superpowers but small rogue states and non-state actors such as terrorists, who gain access to the weapons via the black market once they’ve been developed.
First, by replacing old jobs with ones requiring more skills, technology has rewarded the educated: since the mid-1970s, salaries rose about 25% for those with graduate degrees while the average high school dropout took a 30% pay
Second, they claim that since the year 2000, an ever-larger share of corporate income has gone to those who own the companies as opposed to those who work there—and that as long as automation continues, we should expect those who own the machines to take a growing fraction of the pie.
If you go into law, don’t be the paralegal who reviews thousands of documents for the discovery phase and gets automated away, but the attorney who counsels the client and presents the case in court.
As mechanical muscles made horses redundant, will mechanical minds do the same to humans?
we want jobs because they can provide us with income and purpose, but given the opulence of resources produced by machines, it should be possible to find alternative ways of providing both the income and the purpose without jobs.
Advocates argue that basic income is more efficient than alternatives such as welfare payments to the needy, because it eliminates the administrative hassle of determining who qualifies.
Interestingly, technological progress can end up providing many valuable products and services for free even without government intervention. For example, people used to pay for encyclopedias, atlases, sending letters and making phone calls, but now anyone with an internet connection gets access to all these things at no cost—together
Voltaire wrote in 1759 that “work keeps at bay three great evils: boredom, vice and need.”
Forecasting is tough because, when you’re exploring uncharted territory, you don’t know how many mountains separate you from your destination.
our DNA gave us the goal of having sex because it “wants” to be reproduced, but now that we humans have understood the situation, many of us choose to use birth control, thus staying loyal to the goal itself rather than to its creator or the principle that motivated the goal.
since we won’t get an intelligence explosion until the cost of doing human-level work drops below human-level hourly wages.
many people today fail to realize their full creative potential because they need to devote time and energy to less creative activities just to earn a living.
monotheistic religions believe in or hope for. If someone asks the superintelligent AI “Does God exist?” after it’s switched on, it could repeat a joke by Stephen Hawking and quip “It does now!”
The Catholic Church is the most successful organization in human history in the sense that it’s the only one to have survived for two millennia,
Given our human obsession with limits, it’s fitting that the best-selling copyrighted book of all time is The Guinness Book of World Records.
After all, information is very different from the resources that humans usually fight over, in that you can simultaneously give it away and keep
although dinosaurs ruled Earth for over 100 million years, a thousand times longer than we modern humans have been around, evolution didn’t seem to inevitably push them toward higher intelligence and inventing telescopes or computers.
More extreme examples of the brain rebelling against its genes include choosing to commit suicide or spend life in celibacy to become a priest, monk or nun.
We rebel because by design, as agents of bounded rationality, we’re loyal only to our feelings.
Although our brains evolved merely to help copy our genes, our brains couldn’t care less about this goal since we have no feelings related to genes—indeed, during most of human history, our ancestors didn’t even know that they had genes.
People might realize why their genes make them feel lust, yet have little desire to raise fifteen children, and therefore choose to hack their genetic programming by combining the emotional rewards of intimacy with birth control. They might realize why their genes make them crave sweets yet have little desire to gain weight, and therefore choose to hack their genetic programming by combining the emotional rewards of a sweet beverage with zero-calorie artificial sweeteners.
Teleology is the explanation of things in terms of their purposes rather than their causes, so we can summarize the first part of this chapter by saying that our Universe keeps getting more teleological.
When we build a machine to help us, it can be hard to perfectly align its goals with ours. For example, a mousetrap may mistake your bare toes for a hungry rodent, with painful results.
As long as we build only relatively dumb machines, the question isn’t whether human goals will prevail in the end, but merely how much trouble these machines can cause humanity before we figure out how to solve the goal-alignment problem.
In other words, the time window during which you can load your goals into an AI may be quite short: the brief period between when it’s too dumb to get you and too smart to let you.
If you look up “consciousness” in the 1989 Macmillan Dictionary of Psychology, you’re informed that “Nothing worth reading has been written on it.”1 As
Traditionally, we humans have often founded our self-worth on the idea of human exceptionalism: the conviction that we’re the smartest entities on the planet and therefore unique and superior. The rise of AI will force us to abandon this and become more humble.
saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom. Isaac Asimov
