How Innovation Works: And Why It Flourishes in Freedom

by Matt Ridley

Innovation often disappoints in its early years, only to exceed expectations once it gets going, a phenomenon I call the Amara hype cycle, after Roy Amara, who first said that we underestimate the impact of innovation in the long run but overestimate it in the short run.

He then tried to infect Phipps with smallpox itself and showed that he was immune to it. This demonstration proof, not the vaccination itself, was his real contribution and the reason he had such an impact.

So, yet again, innovation proves to be gradual and to begin with the unlettered and ordinary people, before the elite takes the credit. That is perhaps a little unfair on Jenner, who, like Lady Mary Wortley, deserves fame for persuading the world to adopt the practice.

Science was beginning to catch up with technology.

So the source of the invention of chlorination, like that of vaccination, is enigmatic and confused. Only in retrospect can it be seen as a disruptive and successful innovation that saved millions of lives. It evolved rather slowly, probably from serendipitous beginnings in largely mistaken ideas.

Here Fleming showed, using test tubes deformed to resemble jagged wounds, that antiseptics like carbolic acid were counterproductive, because they killed the body’s own white blood cells without reaching the gangrene-causing bacteria deep in the crevices of the wounds. Instead, Fleming and Wright argued, wounds should be cleaned with saline solution. It was an important discovery, and one that doctors treating the wounded almost completely ignored, because it felt all wrong not to dress wounds with antiseptics.

The story of penicillin reinforces the lesson that even when a scientific discovery is made, by serendipitous good fortune, it takes a lot of practical work to turn it into a useful innovation.

She knew that the monkey kidney tissue culture used to grow the Salk vaccine sometimes itself sickened with viral infections, because of monkey viruses, and she worried that these contaminating viruses might be included with the vaccine, and might cause cancer in people. In June 1959 in her own time she did experiments to show that monkey kidney cultures could indeed cause cancers in hamsters, at the site of inoculation.

The contaminating virus was eventually isolated, christened SV40 and studied in detail by others. We now know that almost every single person vaccinated for polio in America between 1954 and 1963 was probably exposed to monkey viruses, of which SV40 – the fortieth to be described – was just one.

The innovation of smoking, brought from the Americas to the Old World in the 1500s, is one of humankind’s biggest mistakes.

Enter innovation. The decline of smoking in Britain has accelerated sharply in recent years, largely because of the spread of an alternative way of getting nicotine hits (which are not known to be harmful in themselves), using high technology instead of smoke: the electronic cigarette.

He asked Sutherland to explain and was intrigued by the thought that the risks of vaping might be the lesser of two evils – like vaccinating to prevent smallpox or chlorinating to prevent typhoid. Or like distributing clean needles to heroin addicts to prevent HIV infection,

Putting steam engines on ships happened around the same time, but it was not until the second half of the 1800s, and the invention of the screw propeller to replace the paddle wheel, that ocean-going steam could challenge sail for price as well as speed. Sailing technology peaked in the late 1860s with the launch of the Cutty Sark and other fast clippers. The story of the screw propeller shows all the usual elements of an innovation: a long prehistory, simultaneous breakthroughs by two rivals, then incremental evolution over many years. The idea had actually been around since the 1600s and it kept popping up in the eighteenth century, but by the 1830s paddle steamers were everywhere instead.

Nikolaus Otto, a grocery salesman, came up with this design in 1876 after sixteen years of trying to improve on Lenoir’s engine. He had enough success along the way to make and sell stationary engines and to expand his firm, which became Deutz – still a leading engine maker. Although Otto sold many engines, he was not interested in developing a car, so two of his employees, Gottlieb Daimler and Wilhelm Maybach,

The success of the internal-combustion engine is mainly a thermodynamic one. As Vaclav Smil has argued, the key metric is grams per watt (g/W): how much mass it takes to generate a certain amount of energy. Human beings and draught animals operate at about 1,000 g/W. Steam engines got that down to about 100 g/W. The Mercedes 35hp was more like 8.5 g/W and the Model T Ford just 5 g/W. And the cost just kept on falling. In 1913 somebody earning the average American wage would have had to work 2,625 hours to earn enough to buy a Model T. In 2013, on the average wage, he or she would have needed to work just 501 hours, or 18 per cent as long, to afford to buy a Ford Fiesta equipped with seat belts, air bags, side windows, rear-view mirror, heating, a speedometer and windscreen wipers – none of which were in a Model T.

Diesel started from first scientific principles. He became obsessed with the thermodynamics of the Carnot cycle, a theoretical idea by which an internal-combustion engine could reach 100 per cent efficiency, turning heat into work without changing temperature. In the 1890s he strove to get some way towards this goal by inventing an engine that used excess air and high compression, so that the fuel was ignited purely by compression, rather than a spark.

The truly extraordinary improvement in the safety record of air travel is an example of gradual but pervasive innovation with real impact. In 2017, for the first time, there was no death as a result of a commercial passenger jet crashing. There were fatal crashes involving cargo planes, private planes and propeller aircraft, but no commercial passenger jets. Yet that year also saw a record 37 million commercial flights. The number of airline accident fatalities in the world has declined steadily from over 1,000 people a year in the 1990s to just 59 in 2017, even as the number of people flying has greatly increased.

The decline in air accidents is as steep and impressive as the decline in the cost of microchips as a result of Moore’s Law.

The science that explained this hunger for nitrogen came much later, with the discovery that every building block in a protein or DNA molecule must contain several nitrogen atoms, and that though the air consisted mostly of nitrogen atoms they were bound together in tight pairs, triple covalent bonds between each pair of atoms. Vast energy was needed to break these bonds and make nitrogen useful. In the tropics, frequent lightning strikes provided such energy, keeping the land a little more fertile, while in paddy-rice agriculture, algae and other plants fix nitrogen from the air to replenish the soil.

Kings and conquerors also coveted ionized nitrogen (not that they knew it as such), with which to make gunpowder and wage war.

Today, ammonia plants use about one-third as much energy to make a tonne of ammonia as they did in Bosch’s day. About 1 per cent of global energy is used in nitrogen fixation, and that provides about half of all fixed nitrogen atoms in the average human being’s food.

The so called Green Revolution of the 1960s and 1970s was about new varieties of crop, but the key feature of these new varieties was that they could absorb more nitrogen and yield more food without collapsing

The best plants seemed to retain the short stature of Daruma and the high yield of Turkey Red. The station head, Gonjirô Inazuka, selected the most promising lines and in 1935 began to market a true-breeding new wheat variety under the name Nôrin-10.

The dwarfing versions of genes in Nôrin-10 (which turned out to be two mutations known as Rht1 and Rht2 that make the plant less responsive to growth hormones) thus changed the world, in combination with fertilizer fixed from the air.

Berliner isolated the bacterium behind the infection and named it Bacillus thuringiensis. It turned out to be the same creature that had been killing the Japanese silkworms. Bt, as it came to be known, possessed an ability to kill the caterpillars of any moth or butterfly because of a gene for producing a crystallized protein that was lethal to such insects.

Francisco Mojica, who was born nearby, earned a doctorate here in 1993 studying the genes of this creature. And he noticed something rather odd. Hidden in part of its genome was a distinctive sequence of the same thirty letters, repeated over and over again, each repeat being separated by a sequence of 35–39 letters that was different in every case. The repeat sequence was often a palindrome – it spelled the same text backwards and forwards. Mojica looked in another, related salt-loving microbe and found roughly the same pattern, though with a different sequence.

Then, one day in 2003, Mojica had a lucky break. He took one of the non-repetitive ‘spacer’ sequences, between the palindromes, from a gut bacterium, and put it into a database of gene sequences to see what it matched. Eureka. The answer came back: it matched the gene of a virus, specifically a bacteriophage virus, known as ‘phage’ for short.

They concluded: ‘It would be a delusion not to consider the GMO bans in Europe as having had a strong negative impact on the future of biotechnology on the continent.’

Between 1960 and 2010, the acreage of land needed to produce a given quantity of food has declined by about 65 per cent. Had this not happened, pretty well every acre of forest, wetland and nature reserve in the world would have been cultivated or grazed, and the Amazon rain forest would have been far more severely destroyed.

His sources were Arab, and the greatest of them was Al Khwarizmi, the mathematician whose name survives into English in the word ‘algorithm’.

‘A debt minus zero is a debt. A fortune minus zero is a fortune. Zero minus zero is a zero. A debt subtracted from zero is a fortune. A fortune subtracted from zero is a debt. The product of zero multiplied by a debt or fortune is zero.’

Apart from that, there is not a lot to say about Mr Cumming. He was granted a patent on ‘a water closet upon a new construction’. It included many of the features we know today, most critically the S-trap. It flushed from an overhead cistern, and a little water remained in the double bend of the pipe to act as a barrier to smell.

Allen improved the water closet by causing the water to spiral around the bowl when flushed. Around this time Bramah had another accident, and while laid up turned his mind to improving the water closet further. He patented his design in 1778, with a hinged flap instead of a sliding valve and a series of other tweaks to the design.

The key problem now was standardization. The United States government and then the International Standards Organization wrestled for years with what would be the best size and shape for a ‘standard container’.

The computer, as we know it, has four indispensable ingredients to distinguish it from a mere calculator. It must be digital (in particular binary), electronic, programmable and general purpose – that is, capable of carrying out any logical task, at least in principle. In addition, it must actually work. After an exhaustive survey of many claims, the historian Walter Isaacson concludes that the first machine to meet all these criteria is the ENIAC, the Electronic Numerical Integrator and Computer, which began operating towards the end of 1945 at the University of Pennsylvania.

A better candidate to challenge the ENIAC’s claim might be Colossus, the computer built at Bletchley Park in Britain to crack German codes. Colossus preceded the ENIAC by almost two years, the first version being finished in December 1943 and the second, larger version going operational in June 1944: within a few weeks it had decoded some of Hitler’s orders in the battle for Normandy. Colossus was fully electronic, digital (and binary, unlike the ENIAC) and programmable. But it was designed as a single- not a general-purpose machine. Besides, even in the 1970s its story was still shrouded in secrecy, so its influence on later machines was smaller. There again, even if you give Colossus its due, to whom should we give the credit for designing it?

However, new hardware was also crucial to this change, and it came from a surprising source: the computer games industry. The central feature of a computer is the CPU, or central processing unit. This includes one or a few ‘cores’, which do the calculations, and lots of cache memory. For most tasks this is fine, but the games industry found that in creating realistic, apparently three-dimensional images, it needed a different type of chip: one with hundreds of cores that can handle hundreds of software threads at a time. This ‘graphics processing unit’, or GPU, does not replace the CPU but it does augment it, and has proved invaluable for making deep learning, through back propagation, possible.

It turned out that in selecting for docility, Belyaev had also selected for genetic mutations that came with other traits: a domestication syndrome. In particular, he had unwittingly promoted a delay in the migration of the animal’s ‘neural crest’ cells during development. These cells disperse throughout the embryo and give rise to certain tissues within organs such as the skin and the brain. Most of the cells that produce black pigment derive from the neural crest, and it is the paucity of such cells in the head that give domesticated animals their white blaze on the face.

Human beings are rarely like this. From birth we are amazingly tolerant of other people. It looks like we too are a domesticated species, selected by a bunch of Dr Belyaevs – each other – to be less reactively aggressive to strangers, the better to survive in urban, agricultural or dense hunter-gatherer settlements.

The startling idea here (as I argued in The Rational Optimist) is that some time before 150,000 years ago human beings had become reliant on a collective, social brain mediated through specialization and exchange.

Innovations like steam and social media change culture. Fire was an innovation that went one step further and changed human anatomy. Nobody yet knows for sure when fire was invented or where. According to hints in the archaeological evidence it could have been half a million years ago or two million and it could have happened once or many times.

Some people try to live off raw food today, and the result is that they always lose weight, and suffer from infertility and chronic energy deficiency, however much they fill their bellies with nuts and fruits. A German study of more than 500 raw-food faddists, who ate most of their food raw, concluded that ‘a strict raw food diet cannot guarantee an adequate energy supply.’ And this was among people who were eating domesticated and easily digested fruits and vegetables rather than wild food, let alone trekking through forests energetically looking for food, as their chimpanzee equivalents would do while thriving on such diets. Most raw foodists have to include some cooked food in their diets. The human gut is just not adapted to extract enough energy from raw vegetables, raw meat, raw nuts or raw fruit. Which is very odd, when you think about it, because it is not true of any other species, including domesticated ones like dogs. Every human society that has been contacted cooks food, however simple their ecosystem and their dependence on particular species: from the Inuit to the Sentinelese to the Fuegians. Every hunter-gatherer society revolves around the cooking fire.

Cooking predigests food. It gelatinizes starch, almost doubling its digestible energy. It denatures proteins, increasing the energy available from eating an egg or a steak by 40 per cent or so. It is like having an external extra stomach. Cooking therefore explains why we have small teeth, small stomachs and a gut that is only a little over half as big as in other apes, relative to our body weight. This small gut costs us less to run – 10 per cent less energy is burned by people just keeping the alimentary canal alive, compared with other apes. So the cooking fire not only provides us with energy, but also saves us energy.

Charles Townes, who won the Nobel Prize for the physics behind the laser in 1964, was fond of quoting an old cartoon. It shows a beaver and a rabbit looking up at the Hoover dam: ‘No, I didn’t build it myself,’ says the beaver. ‘But it’s based on an idea of mine.’

Or take the invention of genetic fingerprinting, a technology that has proved invaluable in the conviction of the guilty, but even more so in the exoneration of the innocent; and that has been so widely applied in paternity and immigration disputes that it is safe to say DNA unexpectedly had a far greater impact outside medicine than inside it, in the 1990s.

Every technology is a combination of other technologies; every idea a combination of other ideas. As Erik Brynjolfsson and Andrew McAfee put it: ‘Google self-driving cars, Waze, Web, Facebook, Instagram are simple combinations of existing technology.’ But the point is true more generally.

The parallel with human innovation could not be clearer. Innovation happens, as I put it a decade ago, when ideas have sex. It occurs where people meet and exchange goods, services and thoughts. This explains why innovation happens in places where trade and exchange are frequent and not in isolated or underpopulated places: California rather than North Korea, Renaissance Italy rather than Tierra del Fuego.

Darwinians are beginning to realize that recombination is not the same as mutation and the lesson for human innovation is significant. DNA sequences change by errors in transcription, or mutations caused by things like ultraviolet light. These little mistakes, or point mutations, are the fuel of evolution. But, as the Swiss biologist Andreas Wagner has argued, such small steps cannot help organisms cross ‘valleys’ of disadvantage to find new ‘peaks’ of advantage.

Wagner argues that sudden shifts of whole chunks of DNA, through crossing over, or through so-called mobile genetic elements, are necessary to allow organisms to leap across these valleys. The extreme case is hybridization.

Wagner cites numerous studies which support the conclusion that ‘recombination is much more likely to preserve life – up to a thousand times more likely – than random mutation is.’ This is because whole working genes, or parts of genes, can be given new jobs, where a step-by-step change would find only worse results.