More on this book
Community
Kindle Notes & Highlights
A few hundred thousand years ago, in early human (or hominid) prehistory, growth was so slow that it took on the order of one million years for human productive capacity to increase sufficiently to sustain an additional one million individuals living at subsistence level. By 5000 bc, following the Agricultural Revolution, the rate of growth had increased to the point where the same amount of growth took just two centuries. Today, following the Industrial Revolution, the world economy grows on average by that amount every ninety minutes.
most technologies that will have a big impact on the world in five or ten years from now are already in limited use, while technologies that will reshape the world in less than fifteen years probably exist as laboratory prototypes. Twenty years may also be close to the typical duration remaining of a forecaster’s career, bounding the reputational risk of a bold prediction.
McCarthy’s dictum that when something works it is no longer called AI.
Algorithmic high-frequency traders account for more than half of equity shares traded on US markets.
The situation is analogous to the history of heavier than air flight, where birds, bats and insects clearly demonstrated the possibility before our culture mastered it.
Before we would get things to work perfectly, we would probably get things to work imperfectly.
Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
As noted, the rate-limiting step in human intelligence is not how fast raw data can be fed into the brain but rather how quickly the brain can extract meaning and make sense of the data.
In order for the thoughts of one brain to be intelligible to another, the thoughts need to be decomposed and packaged into symbols according to some shared convention that allows the symbols to be correctly interpreted by the receiving brain. This is the job of language.
That there are multiple paths does not entail that there are multiple destinations.
Many machines and nonhuman animals already perform at superhuman levels in narrow domains. Bats interpret sonar signals better than man, calculators outperform us in arithmetic, and chess programs beat us in chess.
World population, for example, has increased by at least a factor of a thousand since the Pleistocene.9 On this basis alone, current levels of human collective intelligence could be regarded as approaching superintelligence relative to a Pleistocene baseline.
Even within the range of present human variation we see that some functions benefit greatly from the labor of one brilliant mastermind as opposed to the joint efforts of myriad mediocrities.
The sluggishness of neural signals limits how big a biological brain can be while functioning as a single processing unit.
In a fast takeoff scenario, humanity’s fate essentially depends on preparations previously put in place.
During a slow takeoff, there would be plenty of time for the news to get out. In a moderate takeoff, by contrast, it is possible that developments would be kept secret as they unfold.
Eliezer Yudkowsky, an AI theorist who has written extensively on the future of machine intelligence,
They are designed to operate for much of the time in unsupervised mode (i.e. to learn hidden structure in unlabeled data in the absence of error or reward signal, without human guidance)
Imitation creates a headwind that disadvantages the leader and benefits laggards, especially if intellectual property is weakly protected.
In human-run organizations, economies of scale are counteracted by bureaucratic inefficiencies and agency problems, including difficulties in keeping trade secrets.
The Romans, despite the high price commanded by imported silk cloth in their empire, never learnt the art of silk manufacture.
The story of porcelain-making also features long lags. The craft was practiced in China during the Tang Dynasty around ad 600 (and might have been in use as early as ad 200), but was mastered by Europeans only in the eighteenth century.
At present, intelligence agencies do not appear to be looking very hard for promising AI projects or other forms of potentially explosive intelligence amplification.
United States concealed its atomic bomb project from the Soviet Union. The United States did collaborate on the Manhattan Project with Britain and Canada.32 Similarly, the United Kingdom concealed its success in breaking the German Enigma code from the Soviet Union, but shared it—albeit with some difficulty—with the United States.
It is important not to anthropomorphize superintelligence when thinking about its potential impacts.
The association is strengthened when we observe that the people who are good at working with computers tend themselves to be nerds.
The magnitudes of the advantages are such as to suggest that rather than thinking of a superintelligent AI as smart in the sense that a scientific genius is smart compared with the average human being, it might be closer to the mark to think of such an AI as smart in the sense that an average human being is smart compared with a beetle or a worm.
But what will its goals be? What is the relation between intelligence and motivation in an artificial agent?
Unfortunately, because a meaningless reductionistic goal is easier for humans to code and easier for an AI to learn, it is just the kind of goal that a programmer would choose to install in his seed AI if his focus is on taking the quickest path to “getting the AI to work” (without caring much about what exactly the AI will do, aside from displaying impressively intelligent behavior). We will revisit this concern shortly. Intelligent
The orthogonality thesis Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.
agents will tend to instrumentally value many kinds of information.
If intelligence and knowledge come at a cost, such as time and effort expended in acquisition, or increased storage or processing requirements, then the agent might prefer less knowledge and less intelligence.
A great deal of resource accumulation is motivated by social concerns—gaining status, mates, friends, and influence, through wealth accumulation and conspicuous consumption.
An unfriendly AI of sufficient intelligence realizes that its unfriendly final goals will be best realized if it behaves in a friendly manner initially, so that it will be let out of the box.
AI might find subtle ways of concealing its true capabilities and its incriminating intent.
A system motivated to promote our interests might be making a mistake if it allowed us to shut it down or to construct another, potentially unfriendly AI.)
The treacherous turn—While weak, an AI behaves cooperatively (increasingly so, as it gets smarter). When the AI gets sufficiently strong—without warning or provocation—it strikes, forms a singleton, and begins directly to optimize the world according to the criteria implied by its final values.
We humans are sometimes saved from wrongdoing by the anticipation that we would feel guilty afterwards if we lapsed.
Final goal: “Act so as to avoid the pangs of bad conscience” Perverse instantiation: Extirpate the cognitive module that produces guilt feelings
Now it might be suggested that the remedy here is obvious. (But how obvious was it before it was pointed out that there was a problem here in need of remedying?)
The claim is that it is much easier to convince oneself that one has found a solution than it is to actually find a solution.
Their performance in preliminary trials helps us make reasonable inferences about their future reliability. Such behavioral methods are defeated in the case of superintelligence because of the strategic planning ability of general intelligence.
unavailing, we must look for alternatives. We can divide potential control methods into two broad classes: capability control methods, which aim to control what the superintelligence can do; and motivation selection methods, which aim to control what it wants to do.
The AI could be allowed to communicate only via a low-bandwidth teletype interface. This would make it slightly harder to psychologically manipulate the gatekeepers (precluding the use of emotive sound and imagery and tricks such as flashing hypnotic patterns to induce a trancelike state).
Consider a billionaire who uses her fortune to set up a large charitable foundation. Once created, the foundation may be powerful—more powerful than most individuals, including its founder, who might have donated most of her wealth.
Even without any designated knowledge base at all, a sufficiently superior mind might be able to learn much by simply introspecting on the workings of its own psyche—the design choices reflected in its source code, the physical characteristics of its circuitry.
Tripwires differ from incentive methods in that they do not rely on the system being aware of the consequences of engaging in forbidden activities.
The final goal given to the AI in this example could be something along the lines of “achieve that which we would have wished the AI to achieve if we had thought about the matter long and hard.” Further
whereas the truth itself is a Schelling point (a salient place for agreement in the absence of communication). So if the oracles achieve consensus, it might be a sign that they gave the true answer.
A sovereign is a system that has an open-ended mandate to operate in the world in pursuit of broad and possibly very long-range objectives.
