Where Good Ideas Come from: The Natural History of Innovation

by Steven Johnson

What lingers in the back of Darwin’s mind, in the days and weeks to come, is not the beauty of the submarine grotto but rather the “infinite numbers” of organic beings.

We now call this phenomenon Darwin’s Paradox: so many different life forms, occupying such a vast array of ecological niches, inhabiting waters that are otherwise remarkably nutrient-poor.

It is a story about the innovative persistence of life. And as he mulls the thought, there is a hint of something else in his mind, a larger, more encompassing theory that might account for the vast scope of life’s innovations. The forms of things unknown are turning, slowly, into shapes.

Presaging a line he would publish thirty years later in the most famous passage from On the Origin of Species, Darwin writes, “I can hardly explain the reason, but there is to my mind much grandeur in the view of the outer shores of these lagoon-islands.” In time, the reason would come to him.

And so Max Kleiber charted a path that would be followed by countless sandal-wearing, nonconformist war protesters in the decades to come. He moved to California. Kleiber set up shop at the agricultural college run by the University of California at Davis, in the heart of the fertile Central Valley.

After a formidable series of measurements in his Davis lab, Kleiber discovered that this scaling phenomenon stuck to an unvarying mathematical script called “negative quarter-power scaling.

As the science writer George Johnson once observed, one lovely consequence of Kleiber’s law is that the number of heartbeats per lifetime tends to be stable from species to species. Bigger animals just take longer to use up their quota.

Several years ago, the theoretical physicist Geoffrey West decided to investigate whether Kleiber’s law applied to one of life’s largest creations: the superorganisms of human-built cities. Did the “metabolism” of urban life slow down as cities grew in size? Was there an underlying pattern to the growth and pace of life of metropolitan systems?

But there was one fundamental difference: the quarter-power law governing innovation was positive, not negative. A city that was ten times larger than its neighbor wasn’t ten times more innovative; it was seventeen times more innovative. A metropolis fifty times bigger than a town was 130 times more innovative.

West’s power laws suggested something far more provocative: that despite all the noise and crowding and distraction, the average resident of a metropolis with a population of five million people was almost three times more creative than the average resident of a town of a hundred thousand.

It is one of the great truisms of our time that we live in an age of technological acceleration; the new paradigms keep rolling in, and the intervals between them keep shortening.

In fact, if you look at the entirety of the twentieth century, the most important developments in mass, one-to-many communications clock in at the same social innovation rate with an eerie regularity. Call it the 10/10 rule: a decade to build the new platform, and a decade for it to find a mass audience.

So Hurley, Chen, and Karim cobbled together a rough beta for a service that would correct these deficiencies, raised less than $10 million in venture capital, hired about two dozen people, and launched YouTube, a website that utterly transformed the way video information is shared online. Within sixteen months of the company’s founding, the service was streaming more than 30 million videos a day.

There are many ways to measure innovation, but perhaps the most elemental yardstick, at least where technology is concerned, revolves around the job that the technology in question lets you do. All other things being equal, a breakthrough that lets you execute two jobs that were impossible before is twice as innovative as a breakthrough that lets you do only one new thing.

This is a book about the space of innovation. Some environments squelch new ideas; some environments seem to breed them effortlessly. The city and the Web have been such engines of innovation because, for complicated historical reasons, they are both environments that are powerfully suited for the creation, diffusion, and adoption of good ideas.

The argument of this book is that a series of shared properties and patterns recur again and again in unusually fertile environments. I have distilled them down into seven patterns, each one occupying a separate chapter. The more we embrace these patterns—in our private work habits and hobbies, in our office environments, in the design of new software tools—the better we will be at tapping our extraordinary capacity for innovative thinking.3

In the language of complexity theory, these patterns of innovation and creativity are fractal: they reappear in recognizable form as you zoom in and out, from molecule to neuron to pixel to sidewalk.

Every economics textbook will tell you that competition between rival firms leads to innovation in their products and services. But when you look at innovation from the long-zoom perspective, competition turns out to be less central to the history of good ideas than we generally think. Analyzing innovation on the scale of individuals and organizations—as the standard textbooks do—distorts our view.

It creates a picture of innovation that overstates the role of proprietary research and “survival of the fittest” competition. The long-zoom approach lets us see that openness and connectivity may, in the end, be more valuable to innovation than purely competitive mechanisms.

The academic literature on innovation and creativity is rich with subtle distinctions between innovations and inventions, between different modes of creativity: artistic, scientific, technological. I have deliberately chosen the broadest possible phrasing—good ideas—to suggest the cross-disciplinary vantage point I am trying to occupy. The good ideas in this survey range from software platforms to musical genres to scientific paradigms to new models for government. My premise is that there is as much value to be found in seeking the common properties across all these varied forms of innovation and creativity as there is value to be found in documenting the differences between them. The poet and the engineer (and the coral reef) may seem a million miles apart in their particular forms of expertise, but when they bring good ideas into the world, similar patterns of development and collaboration shape that process.

And so as soon as his newborn incubator had been installed at Maternité, the fragile infants warmed by hot water bottles below the wooden boxes, Tarnier embarked on a quick study of five hundred babies. The results shocked the Parisian medical establishment: while 66 percent of low-weight babies died within weeks of birth, only 38 percent died if they were housed in Tarnier’s incubating box. You could effectively halve the mortality rate for premature babies simply by treating them like hatchlings in a zoo.

By late 2008, when an MIT professor named Timothy Prestero visited the hospital, all eight were out of order, the victims of power surges and tropical humidity, along with the hospital staff’s inability to read the English repair manual. The Meulaboh incubators were a representative sample: some studies suggest that as much as 95 percent of medical technology donated to developing countries breaks within the first five years of use. Prestero had a vested interest in those broken incubators, because the organization he founded, Design that Matters, had been working for several years on a new scheme for a more reliable, and less expensive, incubator, one that recognized complex medical technology was likely to have a very different tenure in a developing world context than it would in an American or European hospital. Designing an incubator for a developing country wasn’t just a matter of creating something that worked; it was also a matter of designing something that would break in a non-catastrophic way. You couldn’t guarantee a steady supply of spare parts, or trained repair technicians. So instead, Prestero and his team decided to build an incubator out of parts that were already abundant in the developing world.

The scientist Stuart Kauffman has a suggestive name for the set of all those first-order combinations: “the adjacent possible.” The phrase captures both the limits and the creative potential of change and innovation.

The strange and beautiful truth about the adjacent possible is that its boundaries grow as you explore those boundaries. Each new combination ushers new combinations into the adjacent possible. Think of it as a house that magically expands with each door you open. You begin in a room with four doors, each leading to a new room that you haven’t visited yet. Those four rooms are the adjacent possible. But once you open one of those doors and stroll into that room, three new doors appear, each leading to a brand-new room that you couldn’t have reached from your original starting point. Keep opening new doors and eventually you’ll have built a palace.

When dinosaurs such as the velociraptor evolved a new bone called the semi-lunate carpal (the name comes from its half-moon shape), it enabled them to swivel their wrists with far more flexibility. In the short term, this gave them more dexterity as predators, but it also opened a door in the adjacent possible that would eventually lead, many millions of years later, to the evolution of wings and flight. When our ancestors evolved opposable thumbs, they opened up a whole new cultural branch of the adjacent possible: the creation and use of finely crafted tools and weapons.

The Web has explored the adjacent possible of its medium far faster than any other communications technology in history. In early 1994, the Web was a text-only medium, pages of words connected by hyperlinks. But within a few years, the possibility space began to expand. It became a medium that let you do financial transactions, which turned it into a shopping mall and an auction house and a casino. Shortly afterward, it became a true two-way medium where it was as easy to publish your own writing as it was to read other people’s, which engendered forms that the world had never seen before: user-authored encyclopedias, the blogosphere, social network sites.

You can see the fingerprints of the adjacent possible in one of the most remarkable patterns in all of intellectual history, what scholars now call “the multiple”: A brilliant idea occurs to a scientist or inventor somewhere in the world, and he goes public with his remarkable finding, only to discover that three other minds had independently come up with the same idea in the past year. Sunspots were simultaneously discovered in 1611 by four scientists living in four different countries. The first electrical battery was invented separately by Dean Von Kleist and Cuneus of Leyden in 1745 and 1746. Joseph Priestley and Carl Wilhelm Scheele independently isolated oxygen between 1772 and 1774. The law of the conservation of energy was formulated separately four times in the late 1840s. The evolutionary importance of genetic mutation was proposed by S. Korschinsky in 1899 and then by Hugo de Vries in 1901, while the impact of X-rays on mutation rates was independently uncovered by two scholars in 1927. The telephone, telegraph, steam engine, photograph vacuum tube, radio—just about every essential technological advance of modern life has a multiple lurking somewhere in its origin story.

Good ideas are not conjured out of thin air; they are built out of a collection of existing parts, the composition of which expands (and, occasionally, contracts) over time.

But the truth is that technological (and scientific) advances rarely break out of the adjacent possible; the history of cultural progress is, almost without exception, a story of one door leading to another door, exploring the palace one room at a time.

Consider the legendary Analytical Engine designed by nineteenth-century British inventor Charles Babbage, who is considered by most technology historians to be the father of modern computing, though he should probably be called the great-grandfather of modern computing, because it took several generations for the world to catch up to his idea. Babbage is actually in the pantheon for two inventions, neither of which he managed to build during his lifetime. The first was his Difference Engine, a fantastically complex fifteen-ton contraption, with over 25,000 mechanical parts, designed to calculate polynomial functions that were essential to creating the trigonometric tables crucial to navigation. Had Babbage actually completed his project, the Difference Engine would have been the world’s most advanced mechanical calculator. When the London Science Museum constructed one from Babbage’s plans to commemorate the centennial of his death, the machine returned accurate results to thirty-one places in a matter of seconds. Both the speed and precision of the device would have exceeded anything else possible in Babbage’s time by several orders of magnitude.

The same cannot be said of Babbage’s other brilliant idea: the Analytical Engine, the great unfulfilled project of Babbage’s career, which he toiled on for the last thirty years of his life. The machine was so complicated that it never got past the blueprint stage, save a small portion that Babbage built shortly before his death in 1871. The Analytical Engine was—on paper, at least—the world’s first programmable computer.

All of us live inside our own private versions of the adjacent possible. In our work lives, in our creative pursuits, in the organizations that employ us, in the communities we inhabit—in all these different environments, we are surrounded by potential new configurations, new ways of breaking out of our standard routines.

The trick is to figure out ways to explore the edges of possibility that surround you. This can be as simple as changing the physical environment you work in, or cultivating a specific kind of social network, or maintaining certain habits in the way you seek out and store information.

The answer, as it happens, is delightfully fractal: to make your mind more innovative, you have to place it inside environments that share that same network signature: networks of ideas or people that mimic the neural networks of a mind exploring the boundaries of the adjacent possible. Certain environments enhance the brain’s natural capacity to make new links of association. But these patterns of connection are much older than the human brain, older than neurons even. They take us back, once again, to the origin of life itself.

Carbon atoms measure only 0.03 percent of the overall composition of the earth’s crust, and yet they make up nearly 20 percent of our body mass. That abundance highlights the unique property of the carbon atom: its combinatorial power.

But all of these theories share a common motif: the combinatorial power of the carbon atom.

The computer scientist Christopher Langton observed several decades ago that innovative systems have a tendency to gravitate toward the “edge of chaos”: the fertile zone between too much order and too much anarchy. (The notion is central to Stuart Kauffman’s idea of the adjacent possible, as well.)

Those connective carbon atoms swirling in the primordial soup formed a high-density liquid network. The 100 billion neurons in your brain form another kind of liquid network: densely interconnected, constantly exploring new patterns, but also capable of preserving useful structures for long periods of time.

Economists have a telling phrase for the kind of sharing that happens in these densely populated environments: “information spillover.

Looking at the past from this perspective makes one thing clear: somewhere within a thousand years of the first cities emerging, human beings invented a whole new way of inventing. A strong correlation exists between those dense settlements and the dramatic surge in the societal innovation rate. But is there a causal relationship between the two? The chart alone cannot tell us, and we do not know enough about the specific histories of these innovations to document how essential the urban context was to their creation. But the circumstantial evidence is strong.

The pattern was repeated in the explosion of commercial and artistic innovation that emerged in the densely settled hill towns of Northern Italy, the birthplace of the European Renaissance.

Once again, the rise of urban networks triggers a dramatic increase in the flow of good ideas. It is not a coincidence that Northern Italy was the most urbanized region in all of Europe during the fourteenth and fifteenth centuries.

First codified by the Franciscan friar and mathematician Luca Pacioli in 1494, the double-entry method had been used for at least two centuries by Italian bankers and merchants.

Whatever its roots, the technique first became commonplace in the trade capitals of Italy—Genoa, Venice, and Florence—as the merchants of the early Renaissance shared tips among themselves on how best to manage their finances.

Double-entry accounting illustrates a key principle in the emergence of markets: when economic systems shift from feudal structures to the nascent forms of modern capitalism, they become less hierarchical and more networked.

Koestler was a great believer in the creative power that emerges when different intellectual disciplines collide. But he seems to have had little interest in the environments that make those collisions possible: living environments, office environments, media environments.

In group interactions, for instance, exchanges between scientists could be formally coded as “clarification” or “agreement and elaboration” or “

Instead, most important ideas emerged during regular lab meetings, where a dozen or so researchers would gather and informally present and discuss their latest work. If you looked at the map of idea formation that Dunbar created, the ground zero of innovation was not the microscope. It was the conference table.

But architects and interior designers are learning how to build work environments that facilitate liquid networks in more permanent structures.

Exploring the adjacent possible can be as simple as opening a door. But sometimes you need to move a wall.