Where Good Ideas Come From

by Steven Johnson

The tradition of the commonplace book contains a central tension between order and chaos, between the desire for methodical arrangement, and the desire for surprising new links of association. For some Enlightenment-era advocates, the systematic indexing of the comm...

Others, including Priestley and both Darwins, used their commonplace books as a repository for a vast miscellany of hunches.

Each rereading of the commonplace book becomes a new kind of revelation. You see the evolutionary paths of all your past hunches: the ones that turned out to be red herrings; the ones that turned out to be too obvious to write; even the ones that turned into entire books. But each encounter holds the promise that some long-forgotten hunch will connect in a new way with some emerging obsession. The beauty of Locke’s scheme was that it provided just enough order to find snippets when you were looking for them, but at the same time it allowed the main body of the commonplace book to have its own unruly, unplanned meanderings. Imposing too much order runs the risk of orphaning a promising hunch in a larger project that has died, and it makes it difficult for those ideas to mingle and breed when you revisit them. You need a system for capturing hunches, but not necessarily categorizing them, because categories can build barriers between disparate ideas, restrict them to their own conceptual islands. This is one way in which the human history of innovation deviates from the natural history. New ideas do not thrive on archipelagos.

Over a hundred editions of the guide were published, and it remained a common staple of British households well into the twentieth century. One musty copy of the book lingered into the 1960s, in the home of a pair of mathematicians living in the suburbs of London. The couple had a young son who was drawn to the “suggestion of magic” in the book’s title, and who spent hours exploring this “portal to the world of information.” The title stuck in the back of his mind, along with that wondrous feeling of exploring an immense trove of data. More than a decade later, he was working as a software consultant in a Swiss research lab, and found himself overwhelmed by the flow of information and the personnel churn in the organization. As a side project, he began tinkering with an application that would allow him to keep track of all that data. When it came time to give his program a name, his mind drifted back to that strange Victorian household encyclopedia from his youth. He called his application Enquire.

The application allowed you to store small blocks of information about people or projects as nodes in a connected network. It was easy to attach two-way pointers between nodes, so if you pulled up a person’s name, you could instantly see all the projects he or she was working on. The application proved to be genuinely informative, but the programmer soon switched jobs and abandoned the code. He started up another version, which he called Tangle, a few years later, but it never got off the ground. But then, almost ten years after he had first programmed Enquire, he began sketching out a more ambitious application that could make connections between documents stored on different computers, using hypertext links. For a while he struggled with the right name for his nascent platform, calling it an information “mine” or “mesh.” Eventually, he hit upon a different metaphor for the platform’s dense network. He called it the World Wide Web.

In his own account of the Web’s origins, Tim Berners-Lee makes no attempt to collapse the evolution of his marvelous idea into a single epiphany. The Web came into being as an archetypal slow hunch: from a child’s exploration of a hundred-year-old encyclopedia, to a freelancer’s idle side project designed to help him keep track of his colleagues, to a deliberate attempt to build a new information platform that could connect computers across the planet. Like...

Journalists have always asked me what the crucial idea was, or what the singular event was, that allowed the Web to exist one day when it hadn’t the day before. They are frustrated when I tell them there was no “Eureka!” moment . . . Inventing the World Wide Web involved my growing realization that there was a power in arranging ideas in an unconstrained, weblike way. And that awareness came to me through precisely that kind of process. The Web arose as the answer to an open challenge, through the swirling together of influences, ideas, and realizations from many sides, until...

He needed a work environment that carved out a space for slow hunches, cordoned off from all the immediate dictates of the day’s agenda. And he needed information networks that let those hunches travel to other minds, where they could be augmented and polished.

The final report of the Judiciary Committee investigation into the intelligence failings in the months prior to 9/11 explicitly cited this design principle of the Bureau’s information network as one of the key culprits, calling it “a ‘stove pipe’ mentality where crucial intelligence is pigeonholed into a particular unit and may not be shared with other units.”

In 2007, Robert Thatcher, a brain scientist at the University of South Florida, decided to study the vacillation between phase-lock and noise in the brains of dozens of children. While Thatcher found that the noise periods lasted, on average, for 55 milliseconds, he also detected statistically significant variation among the children. Some brains had a tendency to remain longer in phase-lock, others had noise intervals that regularly approached 60 milliseconds. When Thatcher then compared the brain-wave results with the children’s IQ scores, he found a direct correlation between the two data sets. Every extra millisecond spent in the chaotic mode added as much as twenty IQ points. Longer spells in phase-lock deducted IQ points, though not as dramatically.

Thatcher’s study suggests a counterintuitive notion: the more disorganized your brain is, the smarter you are.

William James, writing in the late 1880s, had no way of measuring synchronized neuron firing, but his description of the “highest order of minds” captures something of the chaos mode: Instead of thoughts of concrete things patiently following one another, we have the most abrupt cross-cuts and transitions from one idea to another, the most rarefied abstractions and discriminations, the most unheard-of combinations of elements . . . a seething caldron of ideas, where everything is fizzling and bobbing about in a state of bewildering activity, where partnerships can be joined or loosened in an instant, treadmill routine is unknown, and the unexpected seems the only law.

The water flea Daphnia lives in most freshwater ponds and swamps. Its spasmodic movements in the water are responsible for the “flea” description, but in reality Daphnia are tiny crustaceans, no more than a few millimeters long. Under normal conditions, Daphnia reproduce asexually, with females producing a brood of identical copies of themselves in a tiny pouch. In this mode, the Daphnia community is composed entirely of females. This reproductive strategy proves to be stunningly successful: in warm summer months, Daphnia will often be one of the most abundant organisms in a pond ecosystem. But when conditions get tough, when droughts or other ecological disturbances happen, or when winter rolls in, the water fleas make a remarkable transformation: they start producing males and switch to reproducing sexually. In part, this switch is attributable to the sturdier eggs produced by sexual reproduction, which are more capable of surviving the long months of winter. But scientists believe that the sudden adoption of sex is also a kind of biological innovation strategy: in challenging times, an organism needs new ideas to meet those new challenges. Reproducing asexually makes perfect sense during prosperous periods: if life is good, keep doing what you’re doing. Don’t mess with success by introducing new genetic combinations. But when the world gets more challenging—scarce resources, predators, parasites—you need to innovate. And the quickest path to innovation lies in making no...

The English language is blessed with a wonderful word that captures the power of accidental connection: “serendipity.” First coined in a letter written by the English novelist Horace Walpole in 1754, the word derives from a Persian fairy tale titled “The Three Princes of Serendip,” the protagonists of which were “always making discoveries, by accident and sagacity, of things they were not in quest of.” The contemporary novelist John Barth describes it in nautical terms: “You don’t reach Serendip by plotting a course for it. You have to set out in good faith for elsewhere and lose your bearings serendipitously.”

The challenge, of course, is how to create environments that foster these serendipitous connections, on all the appropriate scales: in the private space of your own mind; within larger institutions; and across the information networks of society itself.

The truth is, your mind contains a near-infinite number of ideas and memories that at any given moment are lurking outside your consciousness. Some tiny fraction of those thoughts are like Kekulé’s serpent: surprising connections that might help you unlock a door in the adjacent possible. But how do you get those particular clusters of neurons to fire at the right time?

The problem with assimilating new ideas at the fringes of your daily routine is that the potential combinations are limited by the reach of your memory.

You can immerse yourself in a single author’s perspective, but then it’s harder to create serendipitous collisions between the ideas of multiple authors. One way around this limitation is to carve out dedicated periods where you read a large and varied collection of books and essays in a condensed amount of time. Bill Gates (and his successor at Microsoft, Ray Ozzie) are famous for taking annual reading vacations.

For more than a decade now, I have been curating a private digital archive of quotes that I’ve found intriguing, my twenty-first-century version of the commonplace book. Some of these passages involve very focused research on a specific project; others are more random discoveries, hunches waiting to make a connection. Some of them are passages that I’ve transcribed from books or articles; others were clipped directly from Web pages. (In the past few years, thanks to Google Books and the Kindle, copying and storing interesting quotes from a book has grown far simpler.) I keep all these quotes in a database using a program called DEVONthink, where I also store my own writing: chapters, essays, blog posts, notes. By combining my own words with passages from other sources, the collection becomes something more than just a file storage system. It becomes a digital extension of my imperfect memory, an archive of all my old ideas, and the ideas that have influenced me. There are now more than five thousand distinct entries in that database, and more than 3 million words—sixty books’ worth of quotes, fragments, and hunches, all individually captured by me, stored in a single database.

Having all that information available at my fingertips is not just a quantitative matter of finding my notes faster. Yes, when I’m trying to track down an article I wrote many years ago, it’s now much easier to retrieve. But the qualitative change lies elsewhere: in finding documents that I’ve forgotten about altogether, finding documents that I didn’t know I was looking for. What makes the system truly powerful is the way that it fosters private serendipity.

DEVONthink features a clever algorithm that detects subtle semantic connections between distinct passages of text. These tools are smart enough to get around the classic search-engine failing of excessive specificity: searching for “dog” and missing all the articles that only have the word “canine” in them. Modern indexing software like DEVONthink’s learns associations between individual words by tracking the frequency with which words appear near each other. This can create almost lyrical connections between ideas. Several years ago, I was working on a book about cholera in London and queried DEVONthink for information about Victorian sewage systems. Because the software had detected that the word “waste” is often used alongside “sewage,” it directed me to a quote that explained the way bones evolved in vertebrate bodies: namely, by repurposing the calcium waste products created by the metabolism of cells. At first glance that might seem like an errant result, but it sent me off on a long and fruitful tan...

But I’m not at all confident that I would have made the initial connection without the help of the software. The idea was a true collaboration, two very different kinds of intelligence playing off one another, one carbon-based, the other silicon. When I’d first captured that quote about calcium and bone structure, I’d had no idea that it would ultimately connect to the history of London’s sewage system (or to a book about innovation). But there was something about that concept that intrigued me enough to store it in the database. It lingered there for years in the software’s primordial soup, a slow hunch waiting for its connection.

I use DEVONthink as an improvisational tool as well. I write a paragraph about something—let’s say it’s about the human brain’s remarkable facility for interpreting facial expressions. I then plug that paragraph into the software, and ask DEVONthink to find other passages in my archive that are similar. Instantly, a list of quotes appears on my screen: some delving into the neural architecture that triggers facial expressions, others exploring the evolutionary history of the smile, others dealing with the expressiveness of our near-relatives, the chimpanzees. Invariably, one or two of these triggers a new association in my head—perhaps I’ve forgotten about the chimpanzee connection—and so I select that quote, and ask the software to find a new batch of passages similar to it. Before long, a larger idea takes shape in my head, built upon the trail of associations the machine has assembled for me.

Compare that to the traditional way of exploring your files, where the computer is like a dutiful, but dumb, butler: “Find me that document about the chimpanzees!” That’s searching. The other feels radically different, so different that we don’t quite have a verb for it: it’s riffing, or exploring. There are false starts and red herrings, but there are just as many happy accidents and unexpected discoveries. Indeed, the fuzziness of the results is part of what makes the software so powerful. The serendipity of the system emerges out of two distinct forces. First, there is the connective power of the semantic algorithm, which is smart but also slightly unpredictable, thus creating a small amount of randomizing noise that makes the results more surprising. But that randomizing force is held in check by the fact that I have curated all these passages myself, which makes each individual connection far more likely to be useful to me in some way. When you start a new query in DEVONthink and look down at the initial results, at first glance they can sometimes seem jumbled and disconnected, but then you read through them in more detail, and i...

Consider this representative elegy to the “endangered joy of serendipity,” authored by a journalism professor named William McKeen: Think about the library. Do people browse anymore? We have become such a directed people. We can target what we want, thanks to the Internet. Put a couple of key words into a search engine and you find—with an irritating hit or miss here and there—exactly what you’re looking for. It’s efficient, but dull. You miss the time-consuming but enriching act of looking through shelves, of pulling down a book because the title interests you, or the binding . . . Looking for something and being surprised by what you find—even if it’s not what you set out looking for—is one of life’s great pleasures, and so far no software exists that can duplicate that experience.

The Internet scholar Ethan Zuckerman compared the front page of the New York Times with that of its Web cousin and found that the print version had twenty-three references on its front page to articles in the paper (either in the form of lead articles themselves, or short summaries teased below the fold). The front page at NYTimes.com, in Zuckerman’s study, contained 315 links to articles and other forms of content. If the architecture of serendipity lies in stumbling across surprising connections while scanning the front page, then the Web is more than ten times as serendipitous as the classic print newspaper.

If you want to build a daily reading list of eclectic and diverse perspectives, you can stitch one together in your RSS reader or your bookmarks bar in a matter of minutes, for no cost, while sitting on your couch. Just as important, you can use the Web to fill out the context when you do stumble across some interesting new topic.

If the commonplace book tradition tells us that the best way to nurture hunches is to write everything down, the serendipity engine of the Web suggests a parallel directive: look everything up.

Ironically, those walls have been erected with the explicit aim of encouraging innovation. They go by many names: patents, digital rights management, intellectual property, trade secrets, proprietary technology.

Yet a number of recent studies have suggested that brainstorming is less effective than its practitioners would like. One trouble with brainstorming is that it is finite in both time and space: a group gathers for an hour in a room, or for a daylong corporate retreat, they toss out a bunch of crazy ideas, and then the meeting disperses. Sometimes a useful connection emerges, but too often the relevant hunches aren’t in sync with one another. One employee has a promising hunch in one office, and two months later, another employee comes up with the missing piece that turns that hunch into a genuine insight. Brainstorming might bring those two fragments together, but the odds are against it.

The secret to organizational inspiration is to build information networks that allow hunches to persist and disperse and recombine. Instead of cloistering your hunches in brainstorm sessions or R&D labs, create an environment where brainstorming is something that is constantly running in the background, throughout the organization, a collective version of the 20-percent-time concept that proved so successful for Google and 3M.

Nemeth had deliberately introduced noise into the decision-making process, and what she found ran directly counter to our intuitive assumptions about truth and error. The groups that had been deliberately contaminated with erroneous information ended up making more original connections than the groups that had only been given pure information. The “dissenting” actors prodded the other subjects into exploring new rooms in the adjacent possible, even though they were, technically speaking, adding incorrect data to the environment.

Nemeth has gone on to document the same phenomenon at work in dozens of different environments: mock juries, boardrooms, academic seminars. Her research suggests a paradoxical truth about innovation: good ideas are more likely to emerge in environments that contain a certain amount of noise and error.

The best innovation labs are always a little contaminated.

Recent studies suggest that the mutation rate in human germ cells is roughly one in thirty million base pairs, which means each time parents pass their DNA on to a child, that genetic inheritance comes with roughly 150 mutations. Much of the machinery in our cells is devoted to preserving and reproducing the signal of the genetic code. But evolution has still made room for noise.

When one of our peers calls the blue painting green, or comes to the defense of a suspect who is clearly guilty, he or she is, technically speaking, introducing more inaccurate information to the environment. But that noise makes the rest of us smarter, more innovative, precisely because we’re forced to rethink our biases, to contemplate an alternate model in which the blue paintings are, in fact, green. Being correct is like the phase-lock states of the human brain, all the neurons firing in perfect synchrony. We need the phase-lock state for the same reason we need truth: a world of complete error and chaos would be unmanageable, on a social and a neurochemical level. (Not to mention genetic.) But leaving some room for generative error is important, too. Innovative environments thrive on useful mistakes, and suffer when the demands of quality control overwhelm them. Big organizations like to follow perfectionist regimes like Six Sigma and Total Quality Management, entire systems devoted to eliminating error from the conference room or the assembly line, but it’s no accident that one of the mantras of the Web startup world is fail faster.

The concept of exaptation is crucial in rebutting the classic biblical argument (now often termed “intelligent design”) against Darwinism, one that dates back to the furor surrounding the publication of On the Origin of Species itself: if extraordinary examples of natural engineering like eyes or wings are not the product of an intelligent creator, then how could these traits have survived through what must have been a pronounced developmental state of nonfunctionality? As the wing evolves, by definition it has to go through a long period where it’s utterly useless at flying.

Cities, then, are environments that are ripe for exaptation, because they cultivate specialized skills and interests, and they create a liquid network where information can leak out of those subcultures, and influence their neighbors in surprising ways.

The eighteenth-century English coffeehouse fertilized countless Enlightenment-era innovations; everything from the science of electricity, to the insurance industry, to democracy itself. Freud maintained a celebrated salon Wednesday nights at 19 Berggasse in Vienna, where physicians, philosophers, and scientists gathered to help shape the emerging field of psychoanalysis. Think, too, of the Paris cafés where so much of modernism was born; or the legendary Homebrew Computer Club in the 1970s, where a ragtag assemblage of amateur hobbyists, teenagers, digital entrepreneurs, and academic scientists managed to spark the personal computer revolution.

But encouragement does not necessarily lead to creativity. Collisions do—the collisions that happen when different fields of expertise converge in some shared physical or intellectual space. That’s where the true sparks fly. The modernism of the 1920s exhibited so much cultural innovation in such a short period of time because the writers, poets, artists, and architects were all rubbing shoulders at the same cafés. They weren’t off on separate islands, teaching creative writing seminars or doing design reviews. That physical proximity made the space rich with exaptation: the literary stream of consciousness influencing the dizzying new perspectives of cubism; the futurist embrace of technological speed in poetry shaping new patterns of urban planning.

What Ruef discovered was a ringing endorsement of the coffeehouse model of social networking: the most creative individuals in Ruef’s survey consistently had broad social networks that extended outside their organization and involved people from diverse fields of expertise.

Chance favors the connected mind.

Published several years later as a monograph, Darwin’s theory of atoll formation marked his first significant contribution to science, and it has largely stood the test of time. The idea itself drew on a coffeehouse of different disciplines: to solve the mystery, he had to think like a naturalist, a marine biologist, and a geologist all at once. He had to understand the life cycle of coral colonies, and observe the tiny evidence of organic sculpture on the rocks of the Keeling Islands; he had to think on the immense time scales of volcanic mountains rising and falling into the sea. And, of course, he needed FitzRoy’s technical expertise with the sounding line. To understand the idea in its full complexity required a kind of probing intelligence, willing to think across those different disciplines and scales.

Most hotbeds of innovation have similar physical spaces associated with them: the Homebrew Computing Club in Silicon Valley; Freud’s Wednesday salon at 19 Berggasse; the eighteenth-century English coffeehouse. All these spaces were, in their own smaller-scale fashion, emergent platforms. Coffeehouse proprietors like Edward Lloyd or William Unwin were not trying to invent the modern publishing industry or the insurance business; they weren’t at all interested in fostering scientific advancement or political turmoil. They were just businessmen, trying to make enough sterling to feed their families, just like those beavers constructing lodges to keep their offspring safe. But the spaces Lloyd and Unwin built turned out to have these unusual properties: they made people think differently, because they created an environment where different kinds of thoughts could productively collide and recombine.

An open platform in software is often called an API, which stands for application programming interface. An API is a kind of lingua franca that software applications can use reliably to communicate with each other, a set of standardized rules and definitions that allow programmers to build new tools on top of another platform, or to weave together information from multiple platforms. When Web users make geographic mashups using Google Maps, they write programs that communicate with Google’s geographic data using their mapping API.

pattern Jane Jacobs detected years ago in urban development: innovation thrives in discarded spaces. Emergent platforms derive much of their creativity from the inventive and economical reuse of existing resources, and, as any urbanite will tell you, the most expensive resource in a big city is real estate.

Jacobs’s point was that the frenetic energy of a large city, the urban version of creative destruction, creates a natural supply of older, less-desirable environments that can be imaginatively reoccupied by the small or the eccentric, the subcultures that Fischer found so essential to urban life. Artists, poets, and entrepreneurs are the vibrant fish swimming among the coral of the Keeling Islands: they find it easier to live in an exoskeleton that has long since been abandoned by its original host. As Jacobs observed: As for really new ideas of any kind—no matter how ultimately profitable or otherwise successful some of them might prove to be—there is no leeway for such chancy trial, error and experimentation in the high-overhead economy of new construction. Old ideas can sometimes use new buildings. New ideas must use old buildings.

information industries.

Economists define “efficient markets” as markets where information is evenly distributed among all the buyers and sellers in the space. Efficiency is generally held to be a universal goal for any economy—unless the economy happens to traffic in ideas. If ideas were fully liberated, then entrepreneurs wouldn’t be able to profit from their innovations, because their competitors would immediately adopt them. And so where innovation is concerned, we have deliberately built inefficient markets: environments that protect copyrights and patents and trade secrets and a thousand other barricades we’ve erected to keep promising ideas out of the minds of others.

liquid networks, slow hunches, serendipity, noise, exaptation, emergent platforms—do