Messy: The Power of Disorder to Transform Our Lives

by Tim Harford

T George Harris, a veteran journalist and editor of Psychology Today, put his finger on the problem back in 1977: The office is a highly personal tool shop, often the home of the soul . . . this fact may sound simple, but it eludes most architects, office designers, and thousands of regulations writers in hundreds of giant corporations. They have a mania for uniformity, in space as in furniture, and a horror over how the messy side of human nature clutters up an office landscape that would otherwise be as tidy as a national cemetery.

Resist the urge to tidy up. Leave the mess—and your workers—alone.

After two recording sessions, Miles Davis had a record that would change the course of twentieth-century music—Kind of Blue.13 Quincy Jones, the revered producer of Frank Sinatra and Michael Jackson, said, “I play Kind of Blue every day—it’s my orange juice. It still sounds like it was made yesterday.” Another legend, jazz fusion pioneer Chick Corea, said that while it’s one thing to play a new tune, “it’s another thing to practically create a new language of music, which is what Kind of Blue did.”14 And yet Miles Davis and his band improvised this musical revolution from moment to moment.* That seems little short of a miracle.15 And the most striking fact about Kind of Blue is this: It wasn’t the record that Miles had been hoping to produce.

“It’s magical, but it’s not magic,” says Charles Limb. “It’s a product of the brain.”18

Despite the obvious limitations of this approach, Limb and other neuroscientists19 have been discovering some intriguing hints about what goes on in an improvising brain. When they recruited six professional jazz pianists to improvise short passages of music, comparing them with scales and memorized tunes, Limb and Allen Braun consistently found an intriguing pattern in the prefrontal cortex.20 This is the part of the brain that is most distinctively human when comparing our brains with those of animals. “It’s where the apparent seat of consciousness resides,” says Limb. “Complex memory, sense of self, sense of morality, sense of humor. Higher-order cognitive processing is all implicated in the prefrontal cortex.”21 No surprise, then, that the prefrontal cortex behaves unusually during musical improvisation. But the striking pattern is that rather than being fired up during improvisation, some fairly broad areas of the prefrontal cortex are being shut down. (These are the dorsolateral areas, on either side of the top of your forehead, and the lateral orbital areas, behind your eyes.) At the same time, the medial frontal area, behind the bridge of your nose, becomes more active. Not only was this pattern robust in the work on jazz improv, it also emerges from a separate study of freestyle rappers by Braun and others.22 What does this all mean? It suggests that improvisers are suppressing their conscious control and letting go. Most of us go through our days censoring our ow...

Miles Davis once explained his approach to jazz improv as creating “freedom and space to hear things.”27 The phrase is fascinating: not freedom and space to play things, but to hear things—what the other instruments were doing, even the sound of your own playing, and to respond. Any of us can have that freedom and space if we’re willing to listen. Whether we’re giving a speech, waiting on tables, or sitting in a corporate call center, the messier, improvised response is the one that takes in the entire context: the ambient noise, a customer’s tone of voice, the reaction of an audience, even the weather. Sometimes it’s only when a speaker delivers a line and sees the body language, hears the laughter, or senses the sharp intake of breath that she instinctively understands what she must do next.

Improvising does expose us to new and different risks—but even careful preparation cannot remove risks entirely.

Bezos combined a grandiose vision with the sketchiest understanding of how that vision would be achieved. The name “Amazon” is inspired by the fact that the River Amazon “blows all other rivers away”;8 he wanted his company to be the “everything store,” the largest retailer on the planet, capable of swiftly supplying any consumer with any product they could imagine. And yet despite these big dreams, when the company opened for business selling books in 1995—by which time Microsoft still hadn’t launched its Internet Explorer browser—Bezos was utterly unprepared for the demand.

Amazon Web Services. AWS in particular was a bold stroke—a move into cloud computing in 2006, four years ahead of Microsoft’s Azure and six years ahead of Google Compute.

The OODA loop is military jargon. It was coined by a retired U.S. Air Force colonel named John Boyd, who, two decades after his death, continues to be a cult figure among military thinkers. Boyd’s theory helps explain everything from Rommel to Bezos to Trump. The theory was set out in a much-photocopied document known as “Patterns of Conflict,” a mess of typed and retyped paper, hand-annotated and 196 pages long.

British actions, rigidity of mind and reluctance to change positions as swiftly and readily as situations demanded . . . great fussiness and over-elaboration of detail in orders.”

In other words, the British were Rommel in reverse: rather than quick, deft, heedless of detail, and happy to make things up on the fly, they were slow, clumsy, tidy-minded, and unwilling to improvise.* If you’re trying to win by creating a mess and betting on yourself to figure it out more quickly than your opponent, then it helps if your opponent is schwerfällig.

“I think you might be underestimating the degree to which established brick-and-mortar business, or any company that might be used to doing things a certain way, will find it hard to be nimble or to focus attention on a new channel,” he told the class at Harvard. “I guess we’ll see.”29 Bezos, it turned out, had judged Len Riggio’s perspective perfectly. Barnes & Noble was making good money in its bookstores and didn’t want to spoil the corporate balance sheet by running a loss-making website; its distribution systems were large and well-established and designed to send large shipments to large stores, rather than one book at a time to individual customers. And Riggio didn’t want his best people to be distracted by a website, so Barnes & Noble’s e-commerce operation received too little management attention for too long. Barnes & Noble had developed excellent systems for running retail bookstores. But it hesitated to engage with the chaos of the new world of e-commerce. Amazon thrived on it. To use John Boyd’s jargon, Amazon was inside Barnes & Noble’s OODA loop, and Barnes & Noble’s response was schwerfällig—ponderous.30

This is very much like Christensen's Innovator's Dilemma

Messy improvisation doesn’t guarantee success. It does, however, guarantee that there will be mistakes, recriminations, and stress along the way. Erwin

So if you’re in some kind of contest, and you feel the potential benefits of the messy approach outweigh the risks and costs, then what does it take to get inside your opponent’s OODA loop—to make him seem schwerfällig?

For much the same reason, John Boyd opposed “synchronization,” once a big idea in the U.S. military. Boyd argues that synchronization was for watches, not for people: trying to synchronize activities wasted time and left everyone marching at the pace of the slowest.45

The trouble is that when we start quantifying and measuring the world, we soon begin to change the world to fit the way we measure it.

Dranove and his colleagues concluded that, overall, the report card system was pushing doctors into performing more expensive treatments, but patients were sicker as a result. As Tony Blair might have said, obviously it shouldn’t work like that.

1975 by the title of an article in the Academy of Management Journal: “On the Folly of Rewarding A, While Hoping for B.”10 Apparently it’s easy to make the mistake of setting the wrong target. Why is it so easy to make this mistake, though? Why not simply set the right target?

Gigerenzer has assembled a large library of simple heuristics that rival or even outperform complex decision rules that are widely thought to be theoretically optimal.

Recent research shows that when limited data are available, Markowitz’s rule of thumb—divide assets equally between categories such as stocks, bonds, and property—outperforms Markowitz’s Nobel-worthy theory. What do we mean by “limited” data? Remarkably, anything less than five hundred years’ worth is probably limited enough to tip the balance in favor of the simple rule of thumb.20

The problem here is known as “overfitting”—what happens when a detailed statistical analysis slavishly follows historical data. Imagine a scatterplot graph with a line or a smooth curve drawn through a cloud of dots to pick out the trend. An overfitted line looks more like a dot-to-dot puzzle, trying to pick out a pattern in the incidence of heart attacks or avalanches that isn’t really there. When new data arrive—new dots—they are unlikely to be anywhere near the wriggling curve. The complex rules are like the overfitted line: designed with too much hindsight, but poor foresight. A cruder rule—draw a straight line or a smooth curve—doesn’t fit the old data so well but will often do better when fresh data arrive. Making the targets more complex can’t be the right answer. A complex measure is just as likely to be gamed, and a simple rule of thumb is often an accurate guide to what is happening.

Just as “How much has a bank borrowed?” is a good rule of thumb only when the banks aren’t trying to meet this target, “How many patients can get an appointment in forty-eight hours?” is probably a good rule of thumb for judging the overall quality of doctor surgeries that don’t know they’re being judged on it. “How many ambulances arrive within eight minutes?” is likely an excellent proxy for how well ambulance services are being run, provided they aren’t actually trying to meet that target. But as soon as we formalize a rule of thumb into a target, it becomes a source of distortion.

In September 2015, Volkswagen, one of the largest car manufacturers in the world, was caught cheating on U.S. emissions tests. Public opinion was shocked. Shares in the German car giant plunged; the boss, Martin Winterkorn, resigned; and Germans worried that the reputation of every manufacturer in the nation might be tarnished by association with VW.25 How was such cheating possible? Manufacturers must submit to a program of laboratory tests designed to ensure that the engines emit low levels of nitrogen oxides, gasses that can cause a variety of local pollution problems—acid rain, smog, and soot, damaging agricultural crops and causing breathing difficulties in elderly people and children, sometimes with fatal results. The tests are tough and the standards are high. But the tests are also absurdly predictable—a series of predetermined maneuvers on a treadmill. As a result, VW was able to cheat. The company’s cars—tricked out with computers and sensors as all modern engines are—would recognize the distinctive choreography of a laboratory test. They would switch to a special testing mode, one where a nitrogen oxide trap in the exhaust system would filter out the pollutants, but at the cost of a sluggish, fuel-hungry engine. Outside the laboratory, the engine would be zippier and more efficient, but would spout vastly more nitrogen oxides, perhaps twenty or thirty times more. What VW did is strikingly similar to what regulators observed in banking, with banks buying unprofit...

This problem has a name: the paradox of automation. It applies to a wide variety of contexts, from the operators of a nuclear power station to the crew of a cruise ship to the simple fact that we can’t remember phone numbers anymore because we have them all stored in our cell phones and that we struggle with mental arithmetic because we’re surrounded by electronic calculators. The better the automatic systems, the more out-of-practice human operators will be, and the more unusual will be the situations they face.3 The psychologist James Reason, author of Human Error, wrote: “Manual control is a highly skilled activity, and skills need to be practiced continuously in order to maintain them. Yet an automatic control system that fails only rarely denies operators the opportunity for practicing these basic control skills . . . when manual takeover is necessary something has usually gone wrong; this means that operators need to be more rather than less skilled in order to cope with these atypical conditions.”*4

The paradox of automation, then, has three strands to it. First, automatic systems accommodate incompetence by being easy to operate and by automatically correcting mistakes. Because of this an inexpert operator can function for a long time before his lack of skill becomes apparent—his incompetence is a hidden weakness that can persist almost indefinitely without being detected. Second, even if operators are expert, automatic systems erode their skills by removing the need for them to practice. Third, automatic systems tend to fail either in unusual situations or in ways that produce unusual situations, requiring a particularly skillful human response. For each of these three strands, a more capable and reliable automatic system makes the situation worse.

Earl Wiener, a cult figure in aviation safety who died in 2013, coined what’s known as “Wiener’s Laws” of aviation and human error. One of them was “Digital devices tune out small errors while creating opportunities for large errors.”6 We might rephrase it as: “Automation will routinely tidy up ordinary messes, but occasionally create an extraordinary mess.” It’s an insight that applies far beyond aviation.

psychologists call it automation bias. The problem—in Bradford, on the U.S. no-fly list, everywhere—is that once a computer has made a recommendation, it is all too easy to accept that recommendation unthinkingly.

Gary Klein, a psychologist who specializes in the study of expert and intuitive decision-making, summarizes the problem: “When the algorithms are making the decisions, people often stop working to get better. The algorithms can make it hard to diagnose reasons for failures. As people become more dependent on algorithms, their judgment may erode, making them depend even more on the algorithms. That process sets up a vicious cycle. People get passive and less vigilant when algorithms make the decisions.”

Decision experts such as Klein complain that many software engineers make the problem worse by deliberately designing systems to supplant human expertise by default; if we wish instead to use them to support human expertise, we need to wrestle with the system.

We worry that the robots are taking our jobs, but just as common a problem is that the robots are taking our judgment.

It is possible to resist the siren call of the algorithms. Rebecca Pliske, a psychologist, found that veteran meteorologists would make weather forecasts first by looking at the data and forming an expert judgment; only then would they look at the computerized forecast to see if the computer had spotted anything that they had missed. (Typically, the answer was no.) By making their manual forecast first, these veterans kept their skills sharp, unlike the pilots on the Airbus 330. However, the younger generation of meteorologists are happier to trust the computers. Once the veterans retire, the human expertise to intuit when the computer has screwed up will be lost forever.18

Why, then, do we ask the people to monitor the machines and not the other way around? That is the way the best meteorologists acted when studied by psychologist Rebecca Pliske: the human made the forecast and then asked the machine for a second opinion. Such a solution will not work everywhere, but it is worth exploring.

n 1982, a trainee Australian doctor carried out the most famous piece of self-experimentation since Benjamin Franklin (perhaps) flew a kite in a thunderstorm. Barry Marshall was frustrated by treating stomach ulcers, which were thought to be caused by stress. Ulcers weren’t curable, but managing their symptoms was a fantastically profitable business, producing the first blockbuster drugs, Tagamet and Zantac.1

These discoveries hint at what might be possible as we begin to understand our microbiota (the microbes that live on and inside us) and their microbiome (their genes). The view used to be that the human body was under assault from bacteria, and that antibiotics were an unalloyed good, albeit one to be used with care lest bacteria evolve resistance. But recently medical researchers have realized that our relationship with bacteria is more complex than that. The average human is host to ten thousand bacterial species. These germs outnumber the cells of our own body, they weigh a total of about three pounds, and they play a crucial role in keeping our metabolism ticking. Some bacteria are dangerous, some are harmless passengers, and some are beneficial. Some, such as H. pylori, can be dangerous or beneficial depending on the situation.3 Martin Blaser has become one of the leading champions of the view that our bacterial guests are starting to become less diverse, and that this thinning of the microbiome is doing us harm.

The driving force, the bait that lured Stapel into a life of academic deceit, was simple: journal editors wanted to publish tidy results, too. And that is what Stapel kept providing for them. Stapel, it appears, was not only a fraud but something of a neat-freak himself.

The story of the “broken windows” theory of urban decay is another example of how we instinctively overestimate the benefits of tidying up certain kinds of urban mess. The theory was proposed in an influential article in The Atlantic Monthly in 1982 by criminologist George Kelling and political scientist James Q. Wilson. Kelling and Wilson argued that small signs of disorder led to the breakdown of community norms and, eventually, to serious criminality. Here’s a taste of their argument:

Modern societies continue to flirt with subtler versions of the same desire for homogeneity that underpinned the Nazi purges, and which research shows is self-defeating. Consider the work of two economists, Gianmarco Ottaviano and Giovanni Peri, who asked what impact a stream of immigrants from across the world might have on American cities. In particular they wanted to look at cities with a large number of foreign-born residents, from a range of countries. One might expect that such melting pots might struggle with social cohesion, gangs, and classrooms overstretched by language barriers. Instead, Ottaviano and Peri found that cities that hosted a complex patchwork of nationalities prospered as a result. U.S.-born residents enjoyed higher wages and, if they were landlords, could charge higher rents. If they were tenants, they paid extra for the privilege of living in such vibrant, productive cities.

And we still fear outsiders. Most of us living in rich countries seem content with immigration policies that exclude people from our own societies simply because of where they were born. The idea that immigration is excessive, out of control, and damaging is not just acceptable but the majority view in many countries. That view may be popular, but it is a mistake. Recall that Katherine Phillips and her colleagues found that small student groups disliked having a stranger in their midst, even as the stranger was helping them solve the murder-mystery problem they faced. We suffer the same problem at a societal level: we find it easy to overlook the contributions that immigrants make, and are acutely aware of our own discomfort. Somehow we must get over that: every society needs its outsiders to bring new attitudes, ideas, and perspectives.

here is one bit of everyday wisdom in Franklin’s motto “Let all your things have their places.” As the psychologist Daniel Levitin explains in The Organized Mind, our spatial memory is powerful, so it’s easier to remember things when they are anchored to a particular location.3 Things such as keys and corkscrews have a tendency to wander about in the course of being used. That is why they are easy to lose. And it is no accident that computer filing systems use a spatial metaphor—manila folders nestled inside other manila folders—to help humans keep track of documents that are, in reality, chopped up and scattered across the surface of some hard drive.

Merlin Mann, a productivity expert, has skewered the desire to over-organize a list of tasks in a fast-moving world. Imagine you’re making sandwiches in a deli, says Mann. The first sandwich order comes in and you start spreading the mayonnaise on a slice of rye bread. But wait—the lunchtime rush is coming. Wouldn’t it be a good idea to stop and check whether any more sandwich orders have arrived? Gosh, yes: two more sandwich orders. Now, how best to organize them? In order of arrival, perhaps. Or maybe it’s best to segregate the vegetarian orders from those involving meat? Another possibility is to distinguish the toasted orders from the fresh ones. Hm. Let’s check to see if any more sandwich orders have arrived. Yes! Three more. Now there are six. So many ways to organize them . . .

Mann’s point is not only that we are often too busy to get organized, but that if we focused on practical action, we wouldn’t need to get organized.

right. Archiving is easy: every now and again, you remove the documents on the right. To find any document, simply ask yourself how recently you’ve seen it. It is a filing system that all but organizes itself, and it has won many fans.

this is similar to looking at the dirty pages in the desk catalogs of an auto parts store

David Kirsh, a cognitive scientist at the University of California, San Diego, studies the working styles of “neats” and “scruffies.” For

how do people orient themselves after arriving at the office or finishing a phone call? Kirsh finds that “neats” orient themselves with to-do lists and calendars, while “scruffies” orient themselves using physical cues—the report that they were working on is lying on the desk, as is a letter that needs a reply, and receipts that must be submitted for expenses. A messy desk is full of such cues.*7 A tidy desk conveys no information at all, and it must be bolstered with the prompt of a to-do list. Both systems can work, so we should hesitate before judging other people based on their messy desks.8

Academics who study workplace behavior sometimes make the distinction between “filers” and “pilers”—people who establish a formal organizational structure for their paper documents, versus people who let piles of paperwork accumulate on and around their desks. A few years ago, two researchers at AT&T Labs, Steve Whittaker and Julia Hirschberg, studied how a group of office workers dealt with their paperwork. Who hoarded the most paper? Who used their archives more actively? When an office relocation forced everyone to reduce the size of their archives, who coped best?

approaches to calendar management. One is that a calendar should be used only for noting fixed points—a doctor’s appointment, a flight, a business meeting. Most of the calendar should be left blank to allow us to do whatever seems appropriate in the moment. But an alternative view is that a calendar should be used to plan more carefully, blocking out time to work on different tasks.

If something was important, then it could be done immediately. Otherwise it wasn’t worth signing away a slice of Andreessen’s future. “I’ve been trying this tactic as an experiment,” he wrote in 2007. “And I am so much happier, I can’t even tell you.”14

Similar to Napoleon's mail system

tatters. Daily plans are tidy, but life is messy.15

If you have very specific needs of the sort that can be specified in a database, then Internet dating is a godsend.