More on this book
Kindle Notes & Highlights
whether or not you think the singularity is near, the very idea deserves some serious thought.
If the intellect becomes, not only the producer, but also a product of technology, then a feedback cycle with unpredictable and potentially explosive consequences can result.
Even if the arguments of the futurists are flawed, we need only assign a small probability to the anticipated event for it to command our most sincere attention. For the consequences for humanity, if a technological singularity did indeed occur, would be seismic.
the ability to re-engineer the brain from bottom to top suggests the possibility of more radical forms of cognitive enhancement and re-organization.
intellectual prowess is what sets human beings apart from the rest of the animals, and chess is a quintessentially intellectual pursuit.
Common Sense and Creativity
A human-level artificial general intelligence must display a similar blend of common sense and creativity if it is to perform comparable feats.
variety of schools in current AI research, and attempt to extrapolate from there.
At one end of this axis we find AIs that have been engineered from scratch, according to principles quite different from those that govern biological intelligence. At the other end of the axis are machines based on neural networks that copy biological brains down to a fine level of physical detail.
Whole Brain Emulation
Copying the Brain
The human brain contains an astonishing number of neurons—more than 80 billion.
human behavior is determined by physical processes in the brain that mediate between its incoming sensory signals and its outgoing motor signals.
The idea of human whole brain emulation is technologically problematic and philosophically challenging. It’s a theme we will return to in due course.
Understanding the neural basis of language is, of course, a major target for neuroscience.
Machine Learning
Universal Artificial Intelligence
the essence of our humanity is the capacity to transcend the contingencies of biology.
explaining consciousness in scientific terms,
the “hard problem” and the “easy problem.”
Three cognitive attributes that seem to be not only necessary for consciousness but also intimately tied together are (1) an apparent sense of purpose, (2) an awareness of the world and the ongoing situation, and (3) the ability to integrate knowledge, perception, and action.
self-reproducing superintelligent machines would be in a good position to colonize the galaxy. From a large enough perspective, it might be seen as human destiny to facilitate this future, even though (unenhanced) humans themselves are physically and intellectually too feeble to participate in it.
Little bits of specialized AI technology are increasingly finding their way into everyday applications. But today’s AI technology is a long way from human-level artificial general intelligence, from AI that possesses common sense and creativity.
AI technology has the potential to reshape human society within a few generations.
we are, as individuals and as a society, highly dependent on information technology, and sophisticated artificial intelligence is only likely to increase that dependence.
An increase in the sophistication of specialized AI technology does look likely in the short term. However, unless it is achieved by the brute-force brain emulation route, human-level artificial general intelligence may require a conceptual breakthrough (or a series of breakthroughs). There are too many unknown unknowns to guess when such a breakthrough might be made.
transhumanism.1 Transhumanists advocate the use of technology to transcend the biological limitations of the human body and its brain.
More radically, the technology of whole brain emulation, discussed in chapter 2, could (arguably) be used to upload a person’s mind into a computational substrate with the aim of rendering it permanently invulnerable to disease or decay.
The transhumanist approach to keeping up with superintelligence is not merely to use technology but to merge with it.
The result would be a new sort of human being, a bio-machine hybrid species with potentially far greater intellectual capabilities than an ordinary person.
Mind Uploading Many transhumanists will not be satisfied with the vision of artificial intelligence alone colonizing the stars. They would like to see humanity along for the trip. But the brevity of human life makes this impractical given the limitations imposed by the speed of light.
The conquest of death by means of technology is a fundamental goal of transhumanism, and mind uploading is one way to approach this goal.
The most important philosophical question to settle is simply whether or not whole brain emulation for humans preserves personal identity
the re-creation of consciousness is not the same as personal survival, the preservation of the self.
Human beings are themselves dangerous creatures, their very natures forged in the ruthlessly competitive crucible of natural selection.
Both the promise and the threat of machine superintelligence are far greater. If we slip up, if we fail to put the right safeguards in place before an intelligence explosion occurs, then we, as a species, might not survive.
the challenge for the AI’s developers is to carefully design its initial reward function to ensure that the resulting behavior is desirable.
A less apocalyptic, but no less grandiose vision assigns humanity a central role in creating a form of AI that will spread out among the stars, eventually to fill the galaxy with intelligence and consciousness.
Deep learning A machine learning technique that involves multiple, hierarchically organized layers of artificial neurons.
Fermi’s paradox The puzzle, first articulated by Enrico Fermi, that our planet seems never to have been visited by extraterrestrials despite the fact that there has been ample time for any sufficiently advanced extraterrestrial civilization to have spread throughout the galaxy.
Great filter In the context of Fermi’s paradox, the hypothesized cause for the demise of any sufficiently advanced extraterrestrial civilization before it has a chance to spread throughout the galaxy. The development of hostile machine superintelligence is one candidate. The term was coined by Robin Hanson.
