New Dark Age: Technology and the End of the Future

by James Bridle

One of the arguments often made in response to weak public understanding of technology is a call to increase technological education – in its simplest formulation, to learn to code. Such a call is made frequently by politicians, technologists, pundits and business leaders, and it is often advanced in nakedly functional and pro-market terms: the information economy needs more programmers, and young people need jobs in the future. This is a good start, but learning to code is not enough, just as learning to plumb a sink is not enough to understand the complex interactions between water tables, political geography, ageing infrastructure, and social policy that define, shape and produce actual life support systems in society. A simply functional understanding of systems is insufficient; one needs to be able to think about histories and consequences too. Where did these systems come from, who designed them and what for, and which of these intentions still lurk within them today?

The second danger of a purely functional understanding of technology is what I call computational thinking. Computational thinking is an extension of what others have called solutionism: the belief that any given problem can be solved by the application of computation. Whatever the practical or social problem we face, there is an app for it. But solutionism is insufficient too; this is one of the things that our technology is trying to tell us. Beyond this error, computational thinking supposes – often at an unconscious level – that the world really is like the solutionists propose. It internalises solutionism to the degree that it is impossible to think or articulate the world in terms that are not computable. Computational thinking is predominant in the world today, driving the worst trends in our societies and interactions, and must be opposed by a real systemic literacy. If philosophy is that fraction of human thought dealing with that which cannot be explained by the sciences, then systemic literacy is the thinking that deals with a world that is not computable, while acknowledging that it is irrevocably shaped and informed by computation.

The weakness of ‘learning to code’ alone might be argued in the opposite direction too: you should be able to understand technological systems without having to learn to code at all, just as one should not need to be a plumber to take a shit, nor to live...

Thinking through machines predates the machines themselves. The existence of calculus proves that some problems may be tractable before it is possible to solve them practically.

an essay entitled ‘As We May Think’, published in the Atlantic in 1945, the engineer and inventor Vannevar Bush wrote, There is a growing mountain of research. But there is increased evidence that we are being bogged down today as specialisation extends. The investigator is staggered by the findings and conclusions of thousands of other workers – conclusions which he cannot find time to grasp, much less to remember, as they appear. Yet specialisation becomes increasingly necessary for progress, and the effort to bridge between disciplines is correspondingly superficial.9

Programming Hippo took almost a year, and when it was ready it was run continuously on the SSEC, twenty-four hours a day, seven days a week, for several months. The result of the calculations was at least three full simulations of a hydrogen bomb explosion: calculations carried out in full view of the public, in a shopfront in New York City, without anyone on the street being even slightly aware of what was going on. The first full-scale American thermonuclear test based on the Hippo calculations was carried out in 1952; today, all the major nuclear powers possess hydrogen bombs. Computational thinking – violent, destructive, and unimaginably costly, in terms of both money and human cognitive activity – slipped out of view. It became unquestioned and unquestionable, and as such it has endured.

All contemporary computation stems from this nexus: military attempts to predict and control the weather, and thus to control the future.

The Kremlin is surrounded by a similar field, as first discovered by players of Pokémon GO, who found their in-game characters teleported blocks away while trying to play the location-based game in the centre of Moscow.

An airport is a canonical example of what geographers call ‘code/space’.30 Code/spaces describe the interweaving of computation with the built environment and daily experience to a very specific extent: rather than merely overlaying and augmenting them, computation becomes a crucial component of them, such that the environment and the experience of it actually ceases to function in the absence of code.

In a study in the NASA Ames Advanced Concepts Flight Simulator, crews were given contradictory fire warnings during preparation for takeoff. The study found that 75 per cent of the crews following the guidance of an automated system shut down the wrong engine, whereas when following a traditional paper checklist only 25 per cent did likewise, despite both having access to additional information that should have influenced their decision. The tapes of the simulations showed that those following the automated system made their decisions faster and with less discussion, suggesting that the availability of an immediate suggested action prevented them looking deeper into the problem.35

the counterintuitive premise of the new dark age: the more obsessively we attempt to compute the world, the more unknowably complex it appears.

In a 2008 article in Wired magazine entitled ‘End of Theory’, Chris Anderson argued that the vast amounts of data now available to researchers made the traditional scientific process obsolete.5 No longer would they need to build models of the world and test them against sampled data. Instead, the complexities of huge and totalising data sets would be processed by immense computing clusters to produce truth itself: ‘With enough data, the numbers speak for themselves.’ As an example, Anderson cited Google’s translation algorithms, which, with no knowledge of the underlying structures of languages, were capable of inferring the relationship between them using extensive corpora of translated texts. He extended this approach to genomics, neurology, and physics, where scientists are increasingly turning to massive computation to make sense of the volumes of information they have gathered about complex systems. In the age of big data, he argued, ‘correlation is enough. We can stop looking for models.’

As the historians of science Albert van Helden and Thomas Hankins put it in 1994, ‘Because instruments determine what can be done, they also determine to some extent what can be thought.’

Known as the Kuznets curve, after its Nobel Prize–winning inventor, this doctrine claims that economic inequality first increases as societies industrialise, but then decreases as mass education levels the playing field and results in wider political participation.

On May 10, 2010, the Dow Jones Industrial Average, a stock market index that tracks thirty of the largest privately owned companies in the United States, opened lower than the previous day, falling slowly over the next few hours in response to the debt crisis in Greece. But in the early afternoon, something very strange happened. At 2:42 p.m., the index started to fall rapidly. In just five minutes, some 600 points – representing billions of dollars in value – were wiped off the market. At its lowest point, the index was a thousand points below the previous day’s average, a difference of almost 10 per cent of its total value, and the biggest ever single-day fall in the market’s history. By 3:07 p.m. – in just twenty-five minutes – it recovered almost all of those 600 points – becoming the largest and fastest swing ever. In the chaos of those twenty-five minutes, 2 billion shares, worth $56 billion, changed hands. Even more worryingly, and for reasons still not fully understood, many orders were executed at what the SEC called ‘irrational prices’: as low as a penny, or as high as $100,000.27 The event became known as the ‘flash crash’, and it is still being investigated and argued over years later.

Despite increasingly sophisticated systems of both computation and visualisation, we are no closer today to truly understanding exactly how machine learning does what it does; we can only adjudicate the results.

a Taiwanese-American strategy consultant named Joz Wang purchased a new Nikon Coolpix S630 camera for Mother’s Day, but when she tried to take a family photo, the camera repeatedly refused to capture an image. ‘Did someone blink?’ read the error message. The camera, preprogrammed with software to wait until all its subjects were looking, eyes open, in the right direction, failed to account for the different physiognomy of non-Caucasians.

Since the 1950s, Kodak had distributed test cards featuring a white woman and the phrase ‘Normal’ in order to calibrate their films. Jean-Luc Godard refused to use Kodak film on assignment in Mozambique in the seventies, claiming it was racist. But only when two of their biggest clients, the confectionary and furniture industries, complained that dark chocolate and dark chairs were difficult to photograph did the company address the need to image dark bodies.

The map is thus multidimensional, extending in more directions than the human mind can hold. As one Google engineer commented, when pursued by a journalist for an image of such a system, ‘I do not generally like trying to visualise thousand-dimensional vectors in three-dimensional space.’25 This is the unseeable space in which machine learning makes its meaning.

By contrast, when the Google Brain–powered AlphaGo software defeated the Korean Go professional Lee Sedol, one of the highest-rated players in the world, something had changed. In the second of five games, AlphaGo played a move that stunned Sedol and spectators alike, placing one of its stones on the far side of the board, and seeming to abandon the battle in progress. ‘That’s a very strange move,’ said one commentator. ‘I thought it was a mistake,’ said the other. Fan Hui, another seasoned Go player who had been the first professional to lose to the machine six months earlier, said of it: ‘It’s not a human move. I’ve never seen a human play this move.’ And he added, ‘So beautiful.’26 In the history of the 2,500-year-old game, nobody had ever played like this. AlphaGo went on to win the game, and the series.

Things get stranger still when these dream images start to interleave with our own memories. Robert Elliott Smith, an artificial intelligence researcher at University College London, returned from a family holiday in France in 2014 with a phone full of photos. He uploaded a number of them to Google+, to share them with his wife, but while browsing through them he noticed an anomaly.30 In one image, he saw himself and his wife at a table in a restaurant, both smiling at the camera. But this photograph had never been taken. At lunch one day, his father had held the button down on his iPhone a little long, resulting in a burst of images of the same scene. Smith uploaded two of them, to see which his wife preferred. In one, he was smiling, but his wife was not; in the other, his wife was smiling, but he was not. From these two images, taken seconds apart, Google’s photo-sorting algorithms had conjured a third: a composite in which both subjects were smiling their ‘best’. The algorithm was part of a package called AutoAwesome (since renamed, simply, ‘Assistant’), which performed a range of tweaks on uploaded images to make them more ‘awesome’ – applying nostalgic filters, turning them into charming animations, and so forth. But in this case, the result was a photograph of a moment that had never happened: a false memory, a rewriting of history.

But perhaps the most extraordinary result derived from Advanced Chess, which is normally played by matched human-machine pairs, occurs when human and machine play against a solo machine. Since Deep Blue, many computer programmes have been developed that can beat any human with ease and efficiency: increases in data storage and processing power mean that supercomputers are no longer required for the task. But even the most powerful contemporary programme can be defeated by a skilled player with access to their own computer – even one less powerful than their opponent. Cooperation between human and machine turns out to be a more potent strategy than the most powerful computer alone.

This formulation, which has come to be known in US law as the ‘Glomar response’, creates a third category of statement between affirmation and renunciation, between truth and falsehood. Often shortened to ‘neither confirm nor deny’, or simply NCND,

The operation of surveillance, and our complicity in it, is one of the most fundamental characteristics of the new dark age, because it insists on a kind of blind vision: everything is illuminated, but nothing is seen. We have become convinced that throwing light upon the subject is the same thing as thinking it, and thus having agency over it. But the light of computation just as easily renders us powerless – either through information overload, or a false sense of security.

It is a nightmarish scene, yet one that seems to embody the conditions of a new dark age. Our vision is increasingly universal, but our agency is ever more reduced. We know more and more about the world, while being less and less able to do anything about it. The resulting sense of helplessness, rather than giving us pause to reconsider our assumptions, seems to be driving us deeper into paranoia and social disintegration: more surveillance, more distrust, an ever-greater insistence on the power of images and computation to rectify a situation that is produced by our unquestioning belief in their authority.

In all of these cases, surveillance reveals itself as a wholly retroactive enterprise, incapable of acting in the present and entirely subservient to the established and utterly compromised interests of power. What was missing in Rwanda and Srebrenica was not evidence of an atrocity, but the willingness to act upon it.