Where Good Ideas Come from: The Natural History of Innovation

by Steven Johnson

But dismissing it on those grounds fundamentally misses the point. Williams stumbled across a provocative and surprising idea that was nonetheless incomplete. But if that hunch had connected with another equally provocative idea, one that emerged three weeks later and five hundred miles away, the Phoenix memo might well have transformed the history of the early twenty-first century.

A metropolis shares one key characteristic with the Web: both environments are dense, liquid networks where information easily flows along multiple unpredictable paths. Those interconnections nurture great ideas, because most great ideas come into the world half-baked, more hunch than revelation.

But the snap judgments of intuition—as powerful as they can be—are rarities in the history of world-changing ideas. Most hunches that turn into important innovations unfold over much longer time frames.

During those twenty years, Priestley dabbled in a dozen different fields, concocted hundreds of novel experiments in his home lab, engaged in extensive conversations with the leading intellectuals of the day. A minuscule percentage of that time was devoted directly to the problem of plant respiration. He just kept it alive in the back of his mind. Sustaining the slow hunch is less a matter of perspiration than of cultivation. You give the hunch enough nourishment to keep it growing, and plant it in fertile soil, where its roots can make new connections. And then you give it time to bloom.

Unlike more technically intricate scientific breakthroughs, it seems somehow appropriate that the basic evolutionary algorithm should just pop into the mind in a moment of recognition. (Darwin’s great supporter, T. H. Huxley, is said to have exclaimed, on hearing the natural selection argument for the first time, “How incredibly stupid not to think of that.”) Darwin’s account also possesses a strangely poetic symmetry, because years later, when Alfred Russel Wallace independently hit upon the theory of natural selection, he claimed his breakthrough had been inspired by Malthus as well.

All of which means we cannot say definitively that Darwin hit upon the idea for his theory of natural selection on September 28, 1838. The best we can do is say that he did not possess the idea when he embarked on his enquiry in the summer of 1837, and that he had it in an enduring form by November of 1838.

Keeping a slow hunch alive poses challenges on multiple scales. For starters, you have to preserve the hunch in your own memory, in the dense network of your neurons. Most slow hunches never last long enough to turn into something useful, because they pass in and out of our memory too quickly, precisely because they possess a certain murkiness.

We can track the evolution of Darwin’s ideas with such precision because he adhered to a rigorous practice of maintaining notebooks where he quoted other sources, improvised new ideas, interrogated and dismissed false leads, drew diagrams, and generally let his mind roam on the page. We can see Darwin’s ideas evolve because on some basic level the notebook platform creates a cultivating space for his hunches; it is not that the notebook is a mere transcription of the ideas, which are happening offstage somewhere in Darwin’s mind. Darwin was constantly rereading his notes, discovering new implications. His ideas emerge as a kind of duet between the present-tense thinking brain and all those past observations recorded on paper. Somewhere in the middle of the Indian Ocean, a train of association compels him to revisit his notes on the fauna of the Galápagos archipelago from five months before. As he reads through his observations, a new thought begins to take shape in his mind, which provokes a whole new set of notes that will only make complete sense to Darwin two years later, after the Malthus episode.

John Locke first began maintaining a commonplace book in 1652, during his first year at Oxford. Over the next decade he developed and refined an elaborate system for indexing the book’s content. Locke thought his method important enough that he appended it to a printing of his canonical work, An Essay Concerning Human Understanding. Locke’s approach seems almost comical in its intricacy, but it was a response to a specific set of design constraints: creating a functional index in only two pages that could be expanded as the commonplace book accumulated more quotes and observations:

In the bibliographic history of epic miscellany, another British title deserves mention alongside Erasmus Darwin’s commonplace book: an immensely popular Victorian how-to guide with the memorable title Enquire Within Upon Everything

In his own account of the Web’s origins, Tim Berners-Lee makes no attempt to collapse the evolution of his marvelous idea into a single epiphany. The Web came into being as an archetypal slow hunch: from a child’s exploration of a hundred-year-old encyclopedia, to a freelancer’s idle side project designed to help him keep track of his colleagues, to a deliberate attempt to build a new information platform that could connect computers across the planet. Like Darwin’s great understanding of life’s tangled web, Berners-Lee’s idea needed time—at least a decade’s worth—to mature:

Early in its history, Google famously instituted a “20-percent time” program for all Google engineers: for every four hours they spend working on official company projects, the engineers are required to spend one hour on their own pet project, guided entirely by their own passions and instincts. (Modeled on a similar program pioneered by 3M known as “the 15-percent rule,” Google’s system is officially called “Innovation Time Off.

Google News launched in September of 2002, which means StoryRank went from a hunch in Krishna Bharat’s mind to a shipping product in one year. Nine years after 9/11, the FBI is still using the Automated Case Support system.

There is nothing mystical about the role of dreams in scientific discovery. While dream activity remains a fertile domain for research, we know that during REM sleep acetylcholine-releasing cells in the brain stem fire indiscriminately, sending surges of electricity billowing out across the brain.

For reasons that are not entirely understood, large clusters of neurons will regularly fire at the exact same frequency. (Imagine a discordant jazz band, each member following a different time signature and tempo, that suddenly snaps into a waltz at precisely 120 beats per minute.) This is what neuroscientists call phase-locking. There is a kind of beautiful synchrony to phase-locking—millions of neurons pulsing in perfect rhythm.

But scientists believe that the sudden adoption of sex is also a kind of biological innovation strategy: in challenging times, an organism needs new ideas to meet those new challenges. Reproducing asexually makes perfect sense during prosperous periods: if life is good, keep doing what you’re doing.

Organizations like IBM and Procter & Gamble, who have a long history of profiting from patented, closed-door innovations, have embraced open innovation platforms over the past decade, sharing their leading-edge research with universities, partners, suppliers, and customers.

Google has a company-wide e-mail list where employees can suggest new features or products; each suggestion can then be rated on a scale of 0 (“Dangerous or harmful”) to 5 (“Great idea! Make it so.

Thomas Kuhn makes a comparable argument for the role of error in The Structure of Scientific Revolutions. Paradigm shifts, in Kuhn’s argument, begin with anomalies in the data, when scientists find that their predictions keep turning out to be wrong. When Joseph Priestley first placed a mint plant in a bell jar to deprive it of oxygen, he expected that the plant would die, just as mice or spiders perished in the same circumstances. But he was wrong: the plant thrived. In fact, it thrived even if you burned all the oxygen out of the jar before placing the plant in it. Priestley’s error energized him to investigate this strange behavior, and it ultimately led him to one of the founding discoveries of what we now call ecosystem science:

In other words, when subjects were exposed to inaccurate descriptions of the slides, they became more creative.

Nemeth has gone on to document the same phenomenon at work in dozens of different environments: mock juries, boardrooms, academic seminars. Her research suggests a paradoxical truth about innovation: good ideas are more likely to emerge in environments that contain a certain amount of noise and error.

Sex keeps the door to the adjacent possible open by just a crack, so that we can adapt to the changing pressures or opportunities of our environment. By keeping the opening so narrow, it also keeps mutation rates in check, which is one crucial reason the asexual bacteria have such markedly higher error rates than the multicellular life. Sex lets us learn from the mistakes of our genes.

“Perhaps the history of the errors of mankind, all things considered, is more valuable and interesting than that of their discoveries. Truth is uniform and narrow; it constantly exists, and does not seem to require so much an active energy, as a passive aptitude of soul in order to encounter it. But error is endlessly diversified.

In The Act of Creation, Arthur Koestler argued that “all decisive events in the history of scientific thought can be described in terms of mental cross-fertilization between different disciplines.

His research led him to one overwhelming conclusion, published in a seminal paper in 1975: big cities nurture subcultures much more effectively than suburbs or small towns.

The same pattern holds true for trades and businesses in large cities. As Jane Jacobs observed in The Death and Life of Great American Cities: “The larger a city, the greater the variety of its manufacturing, and also the greater both the number and the proportion of its small manufacturers.

Cities, then, are environments that are ripe for exaptation, because they cultivate specialized skills and interests, and they create a liquid network where information can leak out of those subcultures, and influence their neighbors in surprising ways. This is one explanation for superlinear scaling in urban creativity. The cultural diversity those subcultures create is valuable not just because it makes urban life less boring. The value also lies in the unlikely migrations that happen between the different clusters. A world where a diverse mix of distinct professions and passions overlap is a world where exaptations thrive.

In the late nineties, a Stanford Business School professor named Martin Ruef decided to investigate the relationship between business innovation and diversity. Ruef was interested in the coffeehouse model of diversity, not the “melting pot” political kind: the diversity of professions and disciplines, not of race or sexual orientation. Ruef interviewed 766 graduates of the school who had gone on to have entrepreneurial careers.

What Ruef discovered was a ringing endorsement of the coffeehouse model of social networking: the most creative individuals in Ruef’s survey consistently had broad social networks that extended outside their organization and involved people from diverse fields of expertise.

Diverse, horizontal social networks, in Ruef’s analysis, were three times more innovative than uniform, vertical networks.

The coffeehouse model of creativity helps explain one of those strange paradoxes of twenty-first-century business innovation. Even as much of the high-tech culture has embraced decentralized, liquid networks in their approach to innovation, the company that is consistently ranked as the most innovative in the world—Apple—remains defiantly top-down and almost comically secretive in its development of new products. You won’t ever see Steve Jobs or Jonathan Ive crowdsourcing development of the next-generation iPhone.

Apple’s approach, by contrast, is messier and more chaotic at the beginning, but it avoids this chronic problem of good ideas being hollowed out as they progress through the development chain. Apple calls it concurrent or parallel production. All the groups—design, manufacturing, engineering, sales—meet continuously through the product-development cycle, brainstorming, trading ideas and solutions, strategizing over the most pressing issues, and generally keeping the conversation open to a diverse group of perspectives.

Legendary innovators like Franklin, Snow, and Darwin all possess some common intellectual qualities—a certain quickness of mind, unbounded curiosity—but they also share one other defining attribute. They have a lot of hobbies.

Chance favors the connected mind.

As the Beagle departed, Darwin captured the miraculous nature of this explanation in his diary. “We must look at a Lagoon [island] as a monument raised by myriads of tiny architects,” he wrote, “to mark the spot where a former land lies buried in the depths of the ocean.

Most hotbeds of innovation have similar physical spaces associated with them: the Homebrew Computing Club in Silicon Valley; Freud’s Wednesday salon at 19 Berggasse; the eighteenth-century English coffeehouse. All these spaces were, in their own smaller-scale fashion, emergent platforms.

The history books like to condense these slower, evolutionary processes into eureka moments dominated by a single inventor, but most of the key technologies that powered the Industrial Revolution were instances of what scholars call “collective invention.” Textbooks casually refer to James Watt as the inventor of the steam engine, but in truth Watt was one of dozens of innovators who refined the device over the course of the eighteenth century.

That deliberate inefficiency doesn’t exist in the fourth quadrant. No, these non-market, decentralized environments do not have immense paydays to motivate their participants. But their openness creates other, powerful opportunities for good ideas to flourish. All of the patterns of innovation we have observed in the previous chapters—liquid networks, slow hunches, serendipity, noise, exaptation, emergent platforms—do best in open environments where ideas flow in unregulated channels.

The wonderful irony is that this historic opportunity comes to governments in part because of an innovation that they unleashed on the world: the Internet, probably the clearest example of the way that public- and private-sector innovation can complement each other. The generative platform of the Internet (and the Web) has created a space where countless fortunes have been made over the past thirty years, but the platform itself was created by the loose affiliation of information scientists around the world, funded, in large part, by the federal government of the United States.