How Innovation Works: And Why It Flourishes in Freedom

by Matt Ridley

Innovations come in many forms, but one thing they all have in common, and which they share with biological innovations created by evolution, is that they are enhanced forms of improbability. That is to say, innovations, be they iPhones, ideas or eider ducklings, are all unlikely, improbable combinations of atoms and digital bits of information.

And innovation is potentially infinite because even if it runs out of new things to do, it can always find ways to do the same things more quickly or for less energy.

The more ordered and improbable our world becomes, the richer we become, and, as a consequence, the more disordered the universe becomes overall.

Innovation, then, means finding new ways to apply energy to create improbable things, and see them catch on.

What made most of us, not just in the West but in China and Brazil too, unprecedentedly rich, so the economic historian Deirdre McCloskey says, was ‘innovationism’: the habit of applying new ideas to raising living standards. No other explanation of the great enrichment of recent centuries makes any sense. Trade had been expanding for centuries, and colonial exploitation with it, and these alone were unable to give anything like the order of magnitude of improvement in incomes that happened.

before. The income of the average Ethiopian has doubled in a decade; the number of people living in extreme poverty has dipped below 10 per cent for the first time in history; malaria mortality has plummeted; war has ceased altogether in the western hemisphere and become much rarer in the Old World, too; frugal LED lights have replaced both incandescent and fluorescent bulbs; telephone conversations have essentially become free on Wi-Fi. Some things have got worse, of course, but most trends are positive. All this is due to innovation.

Most innovation is a gradual process. The modern obsession with disruptive innovation, a phrase coined by the Harvard professor Clayton Christensen in 1995, is misleading. Even when a new technology does upend an old one, as digital media has done to newspapers, the effect begins very slowly, gathers pace gradually and works by increments, not leaps and bounds. Innovation often disappoints in its early years, only to exceed expectations once it gets going, a phenomenon I call the Amara hype cycle, after Roy Amara, who first said that we underestimate the impact of innovation in the long run but overestimate it in the short run.

Before 1700 there were two main kinds of energy used by human beings: heat and work. (Light came mainly from heat.) People burned wood or coal to keep warm and cook food; and they used their muscles, or those of horses and oxen, or rarely a water wheel or a windmill, to move things, to do work. These two kinds of energy were separate: wood and coal did no mechanical work; wind, water and oxen did no warming.

The twentieth century saw only one innovative source of energy on any scale: nuclear power. (Wind and solar, though much improved and with a promising future, still supply less than 2 per cent of global energy.) In terms of its energy density, nuclear is without equal: an object the size of a suitcase, suitably plumbed in, can power a town or an aircraft carrier almost indefinitely.

Following the Three-Mile Island accident in 1979, and Chernobyl in 1986, activists and the public demanded greater safety standards. They got them. According to one estimate, per unit of power, coal kills nearly 2,000 times as many people as nuclear; bioenergy fifty times; gas forty times; hydro fifteen times; solar five times (people fall off roofs installing panels) and even wind power kills nearly twice as many as nuclear. These numbers include the accidents at Chernobyl and Fukushima. Extra safety requirements have simply turned nuclear power from a very, very safe system into a very, very, very safe system.

According to the correspondents, Emmanuel Timonius and Giacomo Pylarini, both physicians working in the Ottoman Empire, the pus from a smallpox survivor would be mixed with the blood in a scratch on the arm of a healthy person.

This beautifully simple, carefully designed, experiment, known as ‘Darriet et al. 1984’, became famous in the small world of malaria and insect control, though it has never achieved the celebrity it deserves in the popular media.

The greatest killer of the modern world is no longer a germ, but a habit: smoking. It directly kills more than six million people every year prematurely, perhaps contributing indirectly to another million deaths. The innovation of smoking, brought from the Americas to the Old World in the 1500s, is one of humankind’s biggest mistakes.

Failure is only the opportunity to begin again more intelligently. HENRY FORD

In 1988 two Brazilian scientists, Joanna Döbereiner and Vladimir Cavalcante, noticed something peculiar. Some fields of sugar cane were producing consistent yields without having received any fertilizer for decades. They searched inside the plant tissues and found a bacterium, Gluconacetobacter diazotrophicus, that was fixing nitrogen from the air. This ability is found in legumes such as peas and beans, thanks to a symbiosis between the plants and bacteria that live in special nodules on the roots.

The lesson of wheeled baggage is that you often cannot innovate before the world is ready. And that when the world is ready, the idea will be already out there, waiting to be employed: in America, at least. Nothing like this happened in Communist Russia or Mao’s China.

In fact, we now know that both Bell and Gray were beaten to the telephone by Antonio Meucci, an Italian who emigrated to Cuba, then New York. He was experimenting with ‘a vibrating diaphragm and an electrified magnet’, the key ingredients of the telephone receiver, back in 1857 and filed a patent caveat in 1871. He built lots of devices and even used them to communicate between floors in his house in Staten Island. The reason history forgot Meucci is because, unlike the determined Bell, he raised no money to develop the idea or defend his patents, and his candle factory went broke, leaving him in poverty and bankruptcy. He was an inventor, but not an innovator.

The origination of the computer is as mysterious and confusing as that of far more ancient and uncertain innovations. There is nobody who deserves the accolade of the inventor of the computer. There is instead a regiment of people who made crucial contributions to a process that was so incremental and gradual, cross-fertilized and networked, that there is no moment or place where it can be argued that the computer came into existence, any more than there is a moment when a child becomes an adult.

Innovators are often unreasonable people: restless, quarrelsome, unsatisfied and ambitious.

Moore’s Law kept on going not just for ten years but for about fifty years, to everybody’s surprise. Yet it probably has now at last run out of steam. The atomic limit is in sight. Transistors have shrunk to less than 100 atoms across, and there are billions on each chip. Since there are now trillions of chips in existence, that means there are billions of trillions of transistors on Planet Earth. They are probably now within an order of magnitude of equalling the number of grains of sand on the planet. Most sand grains, like most microchips, are made largely of silicon, albeit in oxidized form. But whereas sand grains have random – and therefore probable – structures, silicon chips have highly non-random, and therefore improbable, structures.

Again and again people were caught out by the speed of the fall in cost of computing and communicating, leaving future commentators with a rich seam of embarrassing quotations to mine. Often it was those closest to the industry about to be disrupted who least saw it coming. Thomas Watson, the head of IBM, said in 1943 that ‘there is a world market for maybe five computers.’ Tunis Craven, commissioner of the Federal Communications Commission, said in 1961: ‘there is practically no chance communications space satellites will be used to provide better telephone, telegraph, television or radio service inside the United States.’ Marty Cooper, who has as good a claim as anybody to have invented the mobile phone, or cell phone, said, while director of research at Motorola in 1981: ‘Cellular phones will absolutely not replace local wire systems. Even if you project it beyond our lifetimes, it won’t be cheap enough.’ Tim Harford points out that in the futuristic film Blade Runner, made in 1982, robots are so life-like that a policeman falls in love with one, but to ask her out, he calls her from a payphone, not a mobile.

Social media took the world by surprise in another way too. Far from ushering in an era of utopian democratic enlightenment in which the world is flat, everybody is sharing and we all see each other’s point of view, it plunged us into a maze of echo chambers and filter bubbles in which we spend our time confirming our biases and railing against the opinions of others. It polarized, enraged, depressed, addicted and soured us.

Go is too complex a game for the brute-force techniques that helped Deep Blue, and the crucial ingredient of AlphaGo is its ability to learn. It was not taught the rules of Go, but intuited them from examples of games using neural networks (the latest version of the program does not consult human games at all). Thus the human beings who programmed AlphaGo have no idea why it chose the moves it did. Move 37 in Game 2 was described by an expert as ‘creative’ and ‘unique’, because it broke all the normal rules and seemed to be stupid. Lee Sedol took an unusually long time to respond, and although his counter was also unusually clever, he eventually lost the game following a series of equally brilliant moves by AlphaGo at moves 151, 157 and 159.

For the moment, the safest bet is that artificial intelligence will augment rather than replace people, as automation has done for centuries. Even in the case of chess playing, the most successful teams these days are ‘centaurs’, that is to say combinations of algorithms and people. The same will undoubtedly be true of driving. I already rely on my car to warn me when a car is passing me in an outside lane, or when a car is approaching as I back out of a parking space. Many more such ‘intelligent’ tricks will be at my disposal in the future, but the day when I settle into the car, tell it where to take me and go to sleep at the wheel is – in my opinion – a fair way off.

There is no great invention, from fire to flying, that has not been hailed as an insult to some god. J. B. S. HALDANE

Before the last two centuries, innovation was rare. A person could live his or her whole life without once experiencing a new technology: carts, ploughs, axes, candles, creeds and corn looked the same when you died as when you were born. Innovation happened but sporadically and slowly.

Long before farming was invented, human beings made a crucial innovation that transformed their fortunes: the dog. It was the first animal to be domesticated and become an ecological companion of people all over the world, hunting alongside them to their mutual benefit, before later being selected for a huge variety of specialized roles.

His team’s conclusion, published in 2017, is that dogs diverged from wolves around 40,000 years ago, and subsequently split into two (eastern and western) branches of the dog family tree around 20,000 years ago: Chinese village dogs are genetically distinct from European breeds after this date. This suggests that the domestication happened only once, and that it was between 20,000 and 40,000 years ago.

A fascinating long experiment conducted in Siberia since the 1960s shows how this would work and reveals something surprising about the evolution of tameness in both dogs and people. The experiment concerned foxes, but its point is more general. In 1937 Nikolai Belyaev, a well-known geneticist, was arrested and executed without trial for showing an unhealthy interest in Western genetic science. His brother Dmitri was just twenty at the time, but he went on to become a geneticist, too, though he was very careful to pay lip service to the prevailing environmentalist dogma of Stalinism. He went to work in a laboratory that studied fur-bearing animals, and in 1958 he moved to Novosibirsk to join the Institute of Cytology and Genetics of the Siberian Department of the Soviet Academy of Sciences. Here he decided to study silver foxes.

They showed little sign of adapting to human captivity and becoming tame. According to the Lysenko dogma then promoted by Nikita Khrushchev, captivity should itself cause tameness to appear, but it clearly was not happening. Belyaev decided to try selective breeding instead. He did this by a very simple trick. He bred from the least frightened foxes in every generation: the ones that snarled least when their cage was approached. Then he did the same for fox pups, selecting the friendliest, least timid and least aggressive pups. Out of a thousand pups bred every year, 200 were chosen as parents for the next generation. This went on for half a century.

Within a few more generations, Belyaev had enthusiastically tame foxes that rushed over to lick their human friends. What was most surprising, though, was that the foxes had also changed in appearance. They had curly tails, floppy ears, slightly more feminine heads and white patches on their foreheads – as are often found in domesticated cattle, horses and other pets. They also had larger litters and began breeding at younger ages and out of season. Belyaev repeated the experiment with mink and rats, with similar results.

Human beings are rarely like this. From birth we are amazingly tolerant of other people. It looks like we too are a domesticated species, selected by a bunch of Dr Belyaevs – each other – to be less reactively aggressive to strangers, the better to survive in urban, agricultural or dense hunter-gatherer settlements. At some point in human prehistory we must have weeded out people who had fast-migrating neural crest cells and hair-trigger reactions. Whether we did this by executing them, generation after generation, or by ostracizing them, or by sending them into battle, or some combination of the three, we continued doing so into more recent history, and the penal system does so even today.

Brains shrink during domestication in others species, too, including dogs. Wrangham writes: ‘The differences between modern humans and our earlier ancestors have a clear pattern. They look like the differences between a dog and a wolf.’ There is even now evidence of which genes were changed to achieve this result. For instance, the BRAF gene shows strong recent evolutionary selection in cats, horses and people and this relates to neural crest cell migration.

Perhaps a parallel with birds’ nests is helpful. These are artefacts, technologies even, built by brain-equipped vertebrates capable of learning. Yet the structure of a bird’s nest and the materials it is built from are characteristic of each species and vary little over thousands of miles or from decade to decade. Swallows make mud cups, wrens make moss balls, pigeons make stick platforms. Nest building is an innate instinct,

Perhaps toolmaking was an innate instinct in Homo erectus.

Mark Thomas and his colleagues at University College London wrote a paper in 2009 arguing that innovation in the Upper Paleolithic is all about demography. Dense populations inevitably spur human technological change, because they create the conditions in which people can specialize. The most striking evidence for this idea comes from Tasmania and concerns, not innovation, but ‘disinnovation’. The Tasmanian people became isolated 10,000 years ago when rising sea levels at the end of the ice age cut the island off from mainland Australia. They remained effectively uncontacted until Western explorers arrived. During these millennia of isolation, the population of the island was small – about 4,000 people – and not only showed little sign of technological innovation, but actually gave up some of the technologies that were there at the start. By the end, Tasmanians had no bone tools, no cold-weather clothing, no hafted tools, no nets, no barbed spears, fishing spears, spear throwers or boomerangs. In a key paper published in 2004, the anthropologist Joe Henrich explained this by reference to the sudden reduction in the ‘effective population size’ upon isolation. The Tasmanians went from being a small part of a huge population to the whole of a small population. That meant they could no longer draw upon the ideas and discoveries of many people. Given the need to learn skills, the technology shrank to what could be supported by limited specialization within a small population.

Some people try to live off raw food today, and the result is that they always lose weight, and suffer from infertility and chronic energy deficiency, however much they fill their bellies with nuts and fruits. A German study of more than 500 raw-food faddists, who ate most of their food raw, concluded that ‘a strict raw food diet cannot guarantee an adequate energy supply.’ And this was among people who were eating domesticated and easily digested fruits and vegetables rather than wild food, let alone trekking through forests energetically looking for food, as their chimpanzee equivalents would do while thriving on such diets. Most raw foodists have to include some cooked food in their diets. The human gut is just not adapted to extract enough energy from raw vegetables, raw meat, raw nuts or raw fruit.

Cooking predigests food. It gelatinizes starch, almost doubling its digestible energy. It denatures proteins, increasing the energy available from eating an egg or a steak by 40 per cent or so.

Homo erectus had discovered how to use a form of energy till now unavailable to mammals, trapped in wood and released by combustion. Human beings were thus stealing energy sources that had till now been the province of termites, fungi and bacteria. This was in effect an energy transition equivalent in its impact to the adoption of fossil fuels many millennia later.

Liberty is the parent of science and of virtue, and a nation will be great in both in proportion as it is free. THOMAS JEFFERSON

Every technology is a combination of other technologies; every idea a combination of other ideas.

Most inventors find that they need to keep ‘just trying’ things. Tolerance of error is therefore critical.

If error is a key part of innovation, then one of America’s greatest advantages has come from its relatively benign attitude to business failure. Bankruptcy laws in most American states have allowed innovators to ‘fail fast and fail often’ as the Silicon Valley slogan has it. In some states, the ‘homestead exemption’ essentially allows an entrepreneur to keep his or her home if their business fails under Chapter 7 bankruptcy rules. Those states with homestead exemptions have shown more innovation than those without.

The second paradox of the inevitability of invention is that it makes innovation look predictable, yet it is not. In retrospect, it is blindingly obvious that search engines would be the biggest and most profitable fruit of the internet. But did anybody see them coming? No.

Amara’s Law states that people tend to overestimate the impact of a new technology in the short run, but to underestimate it in the long run.

Innovation prefers fragmented governance One of the peculiar features of history is that empires are bad at innovation. Though they have wealthy and educated elites, imperial regimes tend to preside over gradual declines in inventiveness, which contribute to their eventual undoing.

By contrast, the Ottoman and Mughal empires managed to ban printing for more than three centuries. Istanbul, a great city of culture on the edge of Europe administering a vast empire of Christians as well as Muslims, resisted the new technology. It did so, precisely because it was the capital of an empire. In 1485 printing was banned by order of Sultan Bayezid II. In 1515 Sultan Selim I decreed that printing by Muslims was punishable by death. This was an unholy alliance: the calligraphers defending their business monopoly in cahoots with the priests defending their religious monopoly, by successfully lobbying the imperial authorities to keep printing at bay. Foreigners were eventually allowed to print books in foreign languages within the Ottoman Empire, but it was not until 1726 that a Hungarian convert to Islam, Ibrahim Muteferrika, managed to persuade the imperial authorities to allow secular (but not religious) books to be printed in Arabic. Had the lands ruled by the Sultans been fragmented into different political territories and different religions, it is impossible to believe that printing would not have happened sooner and spread faster.

In China, too, the periods of explosive innovation coincided with decentralized government, otherwise known as ‘warring states’. The strong empires, most notably the Ming, effectively put a stop to innovation as well as trade and enterprise more generally. David Hume, writing in the eighteenth century, already realized this truth, that China had stalled as a source of novelty because it was unified, while Europe took off because it was divided.

This fragmentation works best when it results in the creation of city states.

Innovation increasingly means using fewer resources rather than more The bigger cities get, the more productive and efficient they become, in terms of their use of energy to create improbability, just as the bodies of animals do: a whale burns proportionately less energy than a shrew and so lives longer, has a bigger brain and behaves in a more complicated way. London proportionately burns less energy than Bristol, has a bigger collective brain and behaves in a more complicated way. The same is true throughout the economy. Those who say that indefinite growth is impossible, or at least unsustainable, in a world of finite resources are therefore wrong, for a simple reason: growth can take place through doing more with less.