Noise: A Flaw in Human Judgment

by Daniel Kahneman

Bias and noise—systematic deviation and random scatter—are different components of error. The targets illustrate the difference.

Some judgments are biased; they are systematically off target. Other judgments are noisy, as people who are expected to agree end up at very different points around the target.

A general property of noise is that you can recognize and measure it while knowing nothing about the target or bias.

We don’t need to know who is right to measure how much the judgments of the same case vary. All we have to do to measure noise is look at the back of the target.

To understand error in judgment, we must understand both bias and noise. Sometimes, as we will see, noise is the more important problem.

In real-world decisions, the amount of noise is often scandalously high. Here are a few examples of the alarming amount of noise in situations in which accuracy matters:

Personnel decisions are noisy. Interviewers of job candidates make widely different assessments of the same people.

Wherever you look at human judgments, you are likely to find noise. To improve the quality of our judgments, we need to overcome noise as well as bias.

wherever there is judgment, there is noise—and more of it than you think.

“Other people view the world much the way I do.” These beliefs, which have been called naive realism, are essential to the sense of a reality we share with other people.

From the perspective of noise reduction, a singular decision is a recurrent decision that happens only once.

“The personal experiences that made you who you are are not truly relevant to this decision.”

Judgment can therefore be described as measurement in which the instrument is a human mind.

Judgment is not a synonym for thinking, and making accurate judgments is not a synonym for having good judgment.

A matter of judgment is one with some uncertainty about the answer and where we allow for the possibility that reasonable and competent people might disagree.

Judges at wine competitions differ greatly on which wines should get medals, but are often unanimous in their contempt for the rejects.

This behavior reflects a cognitive bias known as the gambler’s fallacy: we tend to underestimate the likelihood that streaks will occur by chance.

“As I always suspected, ideas about politics and economics are a lot like movie stars. If people think that other people like them, such ideas can go far.”

“People believe they capture complexity and add subtlety when they make judgments. But the complexity and the subtlety are mostly wasted—usually they do not add to the accuracy of simple models.”

Overconfidence is one of the best-documented cognitive biases.

“The average expert was roughly as accurate as a dart-throwing chimpanzee.”

A more precise statement of the book’s message was that experts who make a living “commenting or offering advice on political and economic trends” were not “better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.”

Tetlock’s findings suggest that detailed long-term predictions about specific events are simply impossible. The world is a messy place, where minor events can have large consequences.

“Wherever there is prediction, there is ignorance, and probably more of it than we think. Have we checked whether the experts we trust are more accurate than dart-throwing chimpanzees?”

“Researchers must reconcile the idea that they understand life trajectories with the fact that none of the predictions were very accurate.”

“We know we have psychological biases, but we should resist the urge to blame every error on unspecified ‘biases.’”

“Prejudgments and other conclusion biases lead people to distort evidence in favor of their initial position.”

In the absence of true values to determine who is right or wrong, we often value the opinion of respect-experts even when they disagree with one another.

“Intelligence is only part of the story, however. How people think is also important. Perhaps we should pick the most thoughtful, open-minded person, rather than the smartest one.”

“Do you know what specific bias you’re fighting and in what direction it affects the outcome? If not, there are probably several biases at work, and it is hard to predict which one will dominate.

“We have more information about this case, but let’s not tell the experts everything we know before they make their judgment, so as not to bias them. In fact, let’s tell them only what they absolutely need to know.” “The second opinion is not independent if the person giving it knows what the first opinion was. And the third one, even less so: there can be a bias cascade.”

“What makes them so good is less what they are than what they do—the hard work of research, the careful thought and self-criticism, the gathering and synthesizing of other perspectives, the granular judgments and relentless updating.”

“It might be costly to remove noise—but the cost is often worth incurring. Noise can be horribly unfair. And if one effort to reduce noise is too crude—if we end up with guidelines or rules that are unacceptably rigid or that inadvertently produce bias—we shouldn’t just give up. We have to try again.”

“If you want to deter misconduct, you should tolerate some noise. If students are left wondering about the penalty for plagiarism, great—they will avoid plagiarizing. A little uncertainty in the form of noise can magnify deterrence.”

“Creative people need space. People aren’t robots. Whatever your job, you deserve some room to maneuver. If you’re hemmed in, you might not be noisy, but you won’t have much fun and you won’t be able to bring your original ideas to bear.”

If a rule is perfect, of course, it will produce no errors. But rules are rarely perfect.

“Rules simplify life, and reduce noise. But standards allow people to adjust to the particulars of the situations.”

The goal of judgment is accuracy, not individual expression.

Think statistically, and take the outside view of the case.

Structure judgments into several independent tasks.

Resist premature intuitions.

Obtain independent judgments from multiple judges, then consider aggregating those judgments.

Favor relative judgments and relative scales.