The Technological Singularity

by Murray Shanahan

If the intellect becomes, not only the producer, but also a product of technology, then a feedback cycle with unpredictable and potentially explosive consequences can result. For when the thing being engineered is intelligence itself, the very thing doing the engineering, it can set to work improving itself. Before long, according to the singularity hypothesis, the ordinary human is removed from the loop, overtaken by artificially intelligent machines or by cognitively enhanced biological intelligence and unable to keep pace.

In short, a human being is a generalist, a jack of all trades. A human chess champion can do a whole lot more than just play chess. Moreover a human being is adaptive. Fixing photocopiers is not an innate capability. It is learned. Had the office worker been born in a different century or a different culture, she would have acquired a different set of skills. And if she has the misfortune to lose her present job, she can re-train for another one. The achievements of AI research in a variety of specialist domains (chess being just one among many success stories) contrast starkly with the field’s failure to produce a machine with general purpose, adaptive intelligence. So how could we produce artificial general intelligence? Before we can speculate in an informed way about machine superintelligence, we need to answer this question.3

Human intelligence, for all its glorious achievements, is fundamentally an extension of animal intelligence, and the human capacities for language, reason, and creativity all rest on a sensorimotor foundation.

The hallmark of properly general intelligence is the ability to adapt an existing behavioral repertoire to new challenges, and to do so without recourse to trial and error or to training by a third party.

The second major requirement for general intelligence is creativity. The sort of creativity in question is not that of a great artist or composer or mathematician, but something every human being is capable of, something displayed by children in abundance. It is the ability to innovate, to generate novel behavior, to invent new things or devise new ways to use old things.

Creativity and common sense complement each other. Creativity enables the individual to come up with new actions, but a commonsense understanding of the everyday world is needed to anticipate the consequences of those actions.

A fine example of apparently spontaneous innovation was reported in 2002 by a team of scientists from Oxford led by animal cognition researcher Alex Kacelnik.4 They were studying tool use in captive New Caledonian crows (an especially clever species), using an experimental apparatus comprising a small bucket containing food and a tall tube. To challenge the birds, the bucket was lowered into the tube, so that the handle was just out of reach. The birds were provided with pieces of bent wire, which they soon learned to use as hooks to lift the food-bucket out. However, on one occasion, when no hooks were available to the birds, only a piece of straight wire was left in their enclosure. Without ever having been trained to do so, one of the birds, Betty, jammed one end of the wire into a hole in the apparatus and bent it into a hook, which she then used to retrieve the food.

human behavior is determined by physical processes in the brain that mediate between its incoming sensory signals and its outgoing motor signals.

The business of whole brain emulation can be envisioned as a three-stage process: mapping, simulation, and embodiment.1

Suppose that we can genetically engineer the mouse so that every neuron in its brain contains a sequence embedded in its DNA that is unique to that neuron, a kind of “DNA barcode.” Then, with every neuron individually barcoded, the mouse’s brain could be “infected” with an otherwise harmless virus that has been specially engineered to carry genetic material across synaptic gaps, enabling DNA from the pre-synaptic neuron to recombine with DNA from the post-synaptic neuron. This would produce new strands of DNA, each containing a pair of barcodes representing the existence of a synaptic connection between the two neurons in question. The brain of the mouse would thus become a repository of billions of genetically encoded records of pairwise connections between neurons. The task then would be to extract these data, which could be done using DNA sequencing technology. With this method, a neuron-level connectome would be obtainable without the costly intermediate step, in terms of data and computation, of submicron-scale imaging and image processing. Moreover the bottleneck with this method, the speed and cost of DNA sequencing, has undergone years of exponential improvement in the aftermath of the human genome project.

Nanotechnology has numerous potential applications, many of which are relevant to this book. But for now we’ll confine our attention to the business of brain activity mapping. At the nano-scale, even the soma of a neuron, whose characteristic size is a few millionths of a meter, looks big. So we can imagine creating swarms of nano-scale robots capable of swimming freely in the brain’s network of blood vessels, each one then attaching itself like a limpet to the membrane of a neuron or close to a synapse.4 There it would sit, sensing the neuron’s fluctuating membrane potential or detecting spike events, and transmitting this information live to a fleet of micro-scale way-station devices near the cortical surface. The job of these way stations would be to harvest incoming data from the numerous “neuro-limpets” and to broadcast to the outside world, where the data can be collected by the neuroscientist.

Neuroscience has yet to answer this question. But even if the answer is favorable, the scale-up from a mouse brain to a human brain (and to human-level intelligence) is huge. The engineering challenge here is not merely to achieve the required number of FLOPS (floating point operations per second) but to do so in a small volume and with low power consumption. The average human brain (male) occupies a mere 1,250 cm3 and consumes just 20 W. By contrast, the Tianhe-2, the world’s most powerful supercomputer in 2013, consumes 24 MW and is housed in a complex occupying 720 m2. Yet it still has only a fraction of the computing power needed to simulate a human brain under even the most conservative assumptions. In short, massive parallelism notwithstanding, it may be necessary to look beyond conventional digital computers to achieve human-level AI via the whole brain emulation route.

The last of these options raises the possibility of a whole virtual society of artificial intelligences living in a simulated environment. Liberated from the constraints of real biology and relieved of the need to compete for resources such as food and water, certain things become feasible for a virtual society that are not feasible for a society of agents who are confined to wetware. For example, given sufficient computing resources, a virtual society could operate at hyper-real speeds. Every millisecond that passed in the virtual world could be simulated in, say, one-tenth of a millisecond in the real world. If a society of AIs inhabiting such a virtual world were to work on improving themselves or on creating even more intelligent successors, then from the standpoint of the real world their progress would be duly accelerated. And if they were able to direct their technological expertise back out to the real world and help improve the computational substrate on which they depended, then the rate of this acceleration would in turn be accelerated. This is one route to a singularity-like scenario. The result would be explosive technological change, and the consequences would be unpredictable.

Once a mouse-scale whole brain emulation has been achieved, there are compelling reasons to think that human-level AI would not be far off. There are a number of ways the transition could be made. The most obvious is simply to scale up the emulation process and apply it to the human brain. It would be hard engineering, for sure, but no conceptual breakthroughs would be required. But is it realistic to expect the relevant enabling technologies, such as computer processing power and storage capacity, to carry on improving at a fast enough rate? Moore’s law has to end somewhere. Perhaps it will grind to a halt somewhere in the three orders of magnitude between mouse-scale whole brain emulation and human-scale whole brain emulation.

In the story of the motorbike designers, the team of AIs has an enormous competitive advantage over their human rivals simply by working very fast. If the AIs in question were brain-like, this would amount to their operating in faster than real time. This is the simplest and most obvious way to exploit the liberation from biological constraints that results from migration to a computational substrate. But the migration from biology opens up many more possibilities for enhancing the capabilities of brain-inspired artificial intelligence. Consider all the ways in which human workers are hampered by their animal nature. Humans need to eat, for example, and to sleep. But even a biologically highly realistic whole brain emulation—a faithful synthetic copy of a specific brain—could to a large extent be relieved of these needs. While real brains require a blood supply to provide energy, in the form of glucose, to enable neurons to function, a simulated brain has no such requirements, at least not at the level of the simulation.

In short, a brain-inspired human-level AI wouldn’t have to waste time finding food, preparing it, and eating it. Nor would it have to spend time (or as much time, in the case of whole brain emulation) unproductively asleep. The time duly saved could be devoted to work, and the resulting increase in its effective workload would confer the same sort of advantage as acceleration, albeit on a less dramatic scale. Of course, most humans would object to having their mealtimes and their sleep replaced by work. But the reward function of a designer brain could be tuned differently. A willing intellectual slave who never eats or sleeps and wants nothing more than to work would be many corporations’ idea of the perfect employee, especially if they don’t require wages. Eliminating the need for food and sleep is one straightforward way to exploit the liberation from biology. Other relatively conservative techniques for getting the most out of brain-inspired AI are easy to imagine. Many humans enhance their cognitive performance using the tried-and-tested pharmaceutical trick of caffeine ingestion. Hallucinogens such as psilocybin (the active ingredient in magic mushrooms) have often been claimed to promote creativity, their legal status notwithstanding. In a simulated brain the effects of such drugs can themselves be simulated, without any unwanted side effects on the rest of the body. Moreover there’s no need to stick to pharmaceutically realistic interventions. With innumerable easi...

Perhaps the most potent factor in the likely development of superintelligence, whether we’re talking about brain-based AI or AI engineered from scratch, is the prospect of recursive self-improvement. The idea is straightforward. A human-level AI is, by definition, capable of matching humans in almost every sphere of intellectual activity. One such sphere of intellectual activity is the construction of artificial intelligence. A first generation human-level artificial intelligence would be in much the same position as the human engineers that created it. Both species of engineer, biological and artificial, might call upon techniques like those just discussed to boost intelligence. However, the next generation of AIs, those whose intelligence is slightly above human level, will be better at engineering AI than any human. A sufficiently brilliant human neuroscientist could open up whole new vistas of theory, unearthing principles we can hardly imagine today, with far-reaching implications for neural engineering and brain-based artificial intelligence. A team of brilliant artificially intelligent neuroscientists working at superhuman speeds, or otherwise exploiting the possibilities afforded by liberation from biology, would be even more effective. They would be in a position to produce the next generation of brain-based AIs more rapidly than the previous generation was produced by its human developers. Each successive generation would appear more quickly than the last, follow...

According to the prescription, artificial general intelligence can be realized by (1) devising the right reward function, (2) implementing an effective learning technique to build a model of the world, and (3) deploying a powerful optimization method capable of maximizing expected reward given that learned model.

Then again, the distinction between virtual and physical embodiment would become less relevant if an AI could easily migrate between virtual reality and physical reality (much like the characters in the Matrix triology), taking on a robot body as an avatar in order to interact with the physical world. This would be one way in which a disaffected and rebelious AI (or indeed a malicious or malfunctioning AI) could escape the confines of virtual reality and wreak havoc in the real world. But there are other ways that require nothing more than Internet access. Consider Stuxnet, the weaponized computer virus that infiltrated computers in an Iranian nuclear facility, where it took control of centrifuges that were being used to enrich uranium.

A more palatable strategy would be to provide the very best living conditions for the AIs, and to reward them for doing their jobs well. As with a human workforce, this policy is likely to be most productive in the long run, is less dangerous, and raises fewer ethical issues. Taking this liberal approach to its limit, we can imagine a sufficiently human-like AI being given the same legal status and the same rights as a human. At the same time it would acquire moral responsibilities and would be subject to the law like any person. Perhaps the eventual result would be a society in which biological and artificial intelligence coexisted harmoniously, as envisaged in the Culture novels of Iain Banks. This vision of the future has considerable appeal. If the transition from human-level AI to superintelligence is inevitable, then it would be a good idea to ensure that artificial intelligence inherits basic human motives and values. These might include intellectual curiosity, the drive to create, to explore, to improve, to progress. But perhaps the value we should inculcate in AI above all others is compassion toward others, toward all sentient beings, as Buddhists say. And despite humanity’s failings—our war-like inclinations, our tendency to perpetuate inequality, and our occasional capacity for cruelty—these values do seem to come to the fore in times of abundance. So the more human-like an AI is, the more likely it will be to embody the same values, and the more likely it is t...

Three cognitive attributes that seem to be not only necessary for consciousness but also intimately tied together are (1) an apparent sense of purpose, (2) an awareness of the world and the ongoing situation, and (3) the ability to integrate knowledge, perception, and action.

are those that are AI-complete. A problem is said to be AI-complete if achieving human-level AI is a prerequisite for building a computer that can solve it. Passing the Turing Test (properly) is an AI-complete problem, as is (professional standard) machine translation. Occupations such as lawyer, company executive, market researcher, scientist, programmer, psychiatrist, and

Some authors, notably Ray Kurzweil, have made very precise predictions about when machine superintelligence will arise. Writing in 2005, Kurzweil claimed that by the year 2045 the quantity of nonbiological intelligence on the planet will substantially exceed that of the entire human population.2 He based his projections on exponential technological trends, extrapolating them into the future. The best known of these exponential trends is Moore’s law, which we have already encountered several times. This states that the number of transistors that can be fabricated on a given area of silicon doubles roughly every eighteen months.

There are two opposing mistakes that are commonly made in discussions of artificial intelligence by those who don’t work in the field, and especially by the media. The first mistake is to give the impression that artificial intelligence is already here, or is just around the corner. Little bits of specialized AI technology are increasingly finding their way into everyday applications. But today’s AI technology is a long way from human-level artificial general intelligence, from AI that possesses common sense and creativity. A chatbot that is programmed to crack a few jokes or a humanoid robot whose eyes can follow you around a room can easily give a contrary impression. But, as AI skeptics will quickly and rightly point out, this is just an illusion. Yet the same skeptics would be making a mistake of their own to suppose that human-level artificial general intelligence will never happen. Kurzweil’s timeline may be out (or it may not). But as the preceding chapters have argued, there are a number of plausible paths to human-level AI and beyond, and every step along each of those paths is technologically feasible. It doesn’t matter what the timetable is, unless you’re hoping for the singularity to occur just in time to catalyze medical research that will prolong your life. But more important than your life or mine is the world we bequeath to future generations, and this is likely to be profoundly reshaped by the advent of human-level AI. As Friedrich Nietzsche said, above th...

In short, the total amount of paid work that developed economies require humans to do is likely to decrease substantially. If this happens, things could go a number of ways. On the one hand, we might see a more divided society in which the most lucrative work is carried out by a small subset of the population. This highly educated and highly creative elite would buck the trend by pursuing the few remaining occupations where humans still outperform machines, such as entrepreneurship or a creative vocation. The remainder of the population would be out of work. But their basic needs would be more than met. Indeed this is likely to be a time of abundance, with an ever-increasing variety of goods and services available even to the economically less empowered.

Alternatively, we might see a more equitable society, one in which education of the highest quality is afforded to everyone and creativity is universally promoted and duly rewarded. If a system could be instituted in which leisure activities that have social value also had monetary value, then the distinction between paid work and leisure would break down. For example, the writer and information technology critic Jaron Lanier has proposed a system of micropayments whereby every item of data or digital content that an individual produces would generate income for that individual each time it is consumed.6 Perhaps this, or some similar arrangement, could facilitate a more even distribution of power, wealth, and resources. Perhaps it could also stimulate an era of unprecedented cultural expression in which people are no longer tied down by the need to work but are free to pursue art, music, literature, or whatever takes their fancy. But to bring this about might require considerable social and political will. The self-perpetuating tendency for power, wealth, and resources to concentrate in the hands of a few is a historical invariant. In this respect nothing is likely to change in an era of disruptive specialized AI technology. Control of the means of production—in this case the AI technology—will most likely remain in the hands of a small number of powerful corporations and individuals. It would be no surprise if, at the same time, popular culture is pushed to the lowest com...

To conclude this chapter, I want to tell you a story. The story is set in the near future, at a time when some of the artificial intelligence technology we have been discussing has matured, but perhaps not yet to the point where human-level AI has been created. The story is about three AI systems. The first is a marketing AI that belongs to a large multinational corporation, which we will call Moople Corp. The second system is a police AI operated by the US government. The third system is a security AI controlled by the government of a small developing country. The story begins when Moople Corp.’s marketing AI is given responsibility for maximizing the pre-sales of their new wearable computing device. After due deliberation, using the complex model of human behavior that Moople Corp. has built up from its unfathomably deep data vaults and applying the latest, most powerful optimization techniques, the marketing AI comes up with a plan. To excite the market, it announces a pre-launch giveaway. Two hundred of the wearable devices will be handed out for free in one of its flagship stores on a first-come, first-served basis. As required by US law, the marketing AI notifies the local police AI of the pre-launch event because it expects it to attract a crowd. Indeed, when it hears of the event, the police AI estimates (using its own model of human behavior) that 5,000 people will turn up to the flagship store. Moreover the police AI calculates that there is a 10 percent chance o...

But the story has a coda. For one of Moople’s most senior execs, the tragic death of the security guard precipitates a period of profound soul-searching. Eventually this leads her to renounce her considerable material wealth and devote her life to reducing depression among those whose jobs have been taken away by AI technology and are forced into a life of pointless leisure. In due course, the foundation she endows will become a worldwide movement, bringing light into countless lives where before there was only darkness. In short, everything goes exactly as Moople’s other AI had planned all along. Ah, yes, I forgot to mention something. There is another system. Moople Corp.’s ethics AI is much consulted by the company’s employees. It was the ethics AI that advised deployment of the marketing AI in the first place. Based not only on its model of human behavior but also its model of the marketing AI, the ethical AI anticipated the death of the security guard (who was terminally ill, with no access to medical facilities in the developing country) and correctly predicted the effect it would have on the senior Moople exec. So the moral of the story is that unintended consequences can be good as well as bad. What matters is that the reward function of every powerful AI is designed right.

So one approach to keeping up with machine superintelligence might be simply to employ sophisticated AI technology as a tool, amplifying human intelligence noninvasively, so to speak. In essence, this is what humans have done since the invention of writing. But transhumanists aim for more than this. The transhumanist approach to keeping up with superintelligence is not merely to use technology but to merge with it. No one who uses a calculator says that it feels like a part of their mind in the way that someone who has mastered a tool such as a paintbrush might say it feels like a part of their body. The machinations of the calculator are hidden from the user, who simply takes its results as given. We have far more intimate, albeit imperfect, access to the reasoning processes that go on in our own heads, and this intimacy facilitates reflection and cognitive integration. A properly transhumanist perspective on cognitive enhancement demands the same level of intimacy. The enhanced human would be neither a user of AI technology nor a member of a hybrid human–computer team. Rather, interfaced directly to their brain, sophisticated AI technology would be become a part of their mind, conferring unmediated access to its computational processes. The result would be a new sort of human being, a bio-machine hybrid species with potentially far greater intellectual capabilities than an ordinary person. The rest of society would then have to decide how to treat such people, while they...

To sharpen the dilemma, suppose that one of the simulations is terminated after a period of time, say one week. And never mind Murray, suppose that the biological original is you. Suppose you have a terminal illness and have been given six months to live. But you are a billionaire and can afford to undergo whole brain emulation. You are convinced that mind uploading through brain emulation preserves personal identity. So it is your best hope of survival. But you must undertake the procedure now, while your brain is healthy. Then you are told that, as a safeguard, two emulations must be built (in case one is a failure). After a week, if both are functioning correctly, one will be terminated. You are about to sign the papers, but you can’t stop asking yourself which of the two emulations would actually be you. Which body would you wake up in? Isn’t there a chance that you will find yourself reincarnated as a perfectly healthy, functional emulation but then, after a week, be cruelly terminated? How would that be better than forgoing the upload altogether and accepting your present fate? It would be little comfort to know that the other you was doing fine and looking forward to a long life. Surely it’s better to enjoy six months more of guaranteed life than to take the risk of getting just one week. (Of course, you could insist on just one emulation being built, but this is a thought experiment.) Having reflected on this, would you still undergo the procedure?

It’s important to remember that what we’re talking about here is not the first wave of disruptive (specialized) AI technology that was characterized in chapter 6. We are talking about a second wave of disruptive AI technology, something that would only arrive if we managed to develop human-level artificial general intelligence. The social, legal, and political challenges of sophisticated specialized AI technology are considerable. But no doubt we will muddle through, hopefully emerging as a better, more fulfilled society with fewer problems. Both the promise and the threat of machine superintelligence are far greater. If we slip up, if we fail to put the right safeguards in place before an intelligence explosion occurs, then we, as a species, might not survive.

But why stop there? Not only is there an entire planet to exploit (Earth), with a large quantity of matter to re-organize into paperclip factories, there are other planets in our solar system, plus numerous asteroids and moons. Ultimately, as Bostrom argues, if this rogue AI were sufficiently intelligent, it could end up “converting first the Earth and then increasingly large chunks of the observable universe into paperclips.”9 The example, of course, is frivolous. But the moral of the story is not. In contrast to a specialized AI, the intellectual compass of a superhuman-level artificial general intelligence is at least as great as ours, while its powers to shape everything within its compass according to its reward function are far greater. Not only is this world its oyster, so is everything in the universe that is accessible from here.

“The AI neither hates you, nor loves you, but you are made out of atoms that it can use for something else.”11 A system that was intent on accumulating as many resources as possible, without regard to the law or to morality, that was willing to deploy force to defend itself against attempts to stop it, and that was capable of outwitting humans at every turn, would be an engine of unspeakable destruction. Moreover a rogue AI of this nature wouldn’t stop its destructive rampage until it had appropriated everything. It wouldn’t stop at the abject surrender of humanity (if it even noticed). It wouldn’t stop at the extermination of all life on Earth (unless the continuation of life on Earth subserved its reward function). It would just keep on going, turning everything into computronium, into paperclip factories, or whatever (less fanciful) resources it needed. The worst-case situation is reminiscent of the so-called gray goo scenario described by nanotechnology pioneer Eric Drexler, wherein self-replicating nano-scale robots literally eat the planet as they proliferate exponentially.12 But unlike a tide of dumb nanobots, a rogue artificial superintelligence would be able to use thinking to overcome any resistance.

Perhaps the most obvious approach to the problem of rendering an AI safe is to impose a limit on its physical capabilities, and to ensure that it cannot do anything that would revoke this limit. However, this is easier said than done. Suppose that we tried to limit an AI’s ability to act directly on the world. So the AI isn’t endowed with a robot body, nor is it connected to any physical pieces of equipment or infrastructure. The only way for it to interact with the outside world is through language. Surely the AI would then have no means to accumulate resources or to deploy military force. We would be safe. Unfortunately, this is not true. Human dictators don’t need to act directly on the physical world. Rather, they rely on persuading other people to do their bidding. Not only would a superhuman-level AI be more adept at manipulating human behavior than the most Machiavellian dictator, it would have a great deal more to offer. Indeed, even if the AI were confined to a secure facility with no access to the outside world, we wouldn’t be safe. Before long, those with the power to release it into the wild would likely succumb to its promises and/or threats.

Perhaps technological development always follows a similar path for every civilization, everywhere in the galaxy. When a civilization’s technology reaches a certain level, it becomes easy to engineer a self-improving artificial general intelligence. Yet, at that point, the obstacles to making it safe remain insurmountable. Even if the danger is widely understood, someone (some blob, some hive, or whatever) somewhere on the planet in question is bound to make one. After that, everything is paperclips, so to speak. All is lost.