Noise: A Flaw in Human Judgment

by Daniel Kahneman

Bias and noise—systematic deviation and random scatter—are different components of error. The targets illustrate the difference.

A general property of noise is that you can recognize and measure it while knowing nothing about the target or bias.

as we will see, noise is the more important problem. But in public conversations about human error and in organizations all over the world, noise is rarely recognized. Bias is the star of the show. Noise is a bit player, usually offstage. The topic of bias has been discussed in thousands of scientific articles and dozens of popular books, few of which even mention the issue of noise. This book is our attempt to redress the balance.

Acceptable lotteries are used to allocate “goods,” like courses in some universities, or “bads,” like the draft in the military. They serve a purpose. But the judgment lotteries we talk about allocate nothing.

By our measure, the median difference in underwriting was 55%, about five times as large as was expected by most people, including the company’s executives.

A defining feature of system noise is that it is unwanted, and we should stress right here that variability in judgments is not always unwanted.

But diversity of tastes can help account for errors if a personal taste is mistaken for a professional judgment.

The noise audits suggested that respected professionals—and the organizations that employ them—maintained an illusion of agreement while in fact disagreeing in their daily professional judgments.

We hold a single interpretation of the world around us at any one time, and we normally invest little effort in generating plausible alternatives to it. One interpretation is enough, and we experience it as true. We do not go through life imagining alternative ways of seeing what we see.

Bad judgment is much easier to identify than good judgment. The

Our conclusion is simple: wherever there is judgment, there is noise, and more of it than you think.

Arguably, there is a continuum, not a category difference, between singular and recurrent decisions.

The nature of singular decisions raises an important question for the study of noise. We have defined noise as undesirable variability in judgments of the same problem. Since singular problems are never exactly repeated, this definition does not apply to them. After all, history is only run once.

Our observations here suggest the opposite advice. From the perspective of noise reduction, a singular decision is a recurrent decision that happens only once.

Judgment can therefore be described as measurement in which the instrument is a human mind. Implicit in the notion of measurement is the goal of accuracy

Judgment, like measurement, refers both to the mental activity of making a judgment and to its product.

We will rely extensively on the analogy between judgment and measurement because it helps explain the role of noise in error.

We have contrasted two ways of evaluating a judgment: by comparing it to an outcome and by assessing the quality of the process that led to it.

System noise is inconsistency, and inconsistency damages the credibility of the system.

All we need to measure noise is multiple judgments of the same problem.

Its basic message is straightforward: in professional judgments of all kinds, whenever accuracy is the goal, bias and noise play the same role in the calculation of overall error.

wherever achieving accuracy is the goal. The weighting of errors by their square is central to statistics.

bias and noise play identical roles in the error equation. They are independent of each other and equally weighted in the determination of overall error.

In terms of overall error, noise and bias are independent: the benefit of reducing noise is the same, regardless of the amount of bias.

It provides the rationale for the goal of reducing system noise in predictive judgments, a goal that is in principle as important as the reduction of statistical bias.

Keep your values and your facts separate.”

The variance of biases across cases—some positive, some negative—is an important source of error and unfairness.

Now, add noise by going down each column and changing some numbers here and there—sometimes by adding prison time to the mean sentence, sometimes by subtracting from it. Because the changes you make are not all the same, they create variability within the column. This variability is noise.

The essential result of this study is the large amount of noise observed within the judgments of each case.

For these reasons, we suspect that the amount of noise defendants face in actual courtrooms is even larger than what we see here.

Not surprisingly, conservative ideology was also related to severity of sentences.

noted, “Patterned differences between judges in the influence of offense/offender characteristics” are “an additional form of sentence disparity.”

System noise is undesirable variability in the judgments of the same case by multiple individuals.

Variability is expected, not just between players but within players. The

we do not always produce identical judgments when faced with the same facts on two occasions.

We have described the process that picks an underwriter, a judge, or a doctor as a lottery that creates system noise. Occasion noise is the product of a second lottery. This lottery picks the moment when the professional makes a judgment, the professional’s mood, the sequence of cases that are fresh in mind, and countless other features of the occasion.

When people form a carefully considered professional opinion, they associate it with the reasons that justify their point of view.

Vul and Pashler drew inspiration from the well-known phenomenon known as the wisdom-of-crowds effect: averaging the independent judgments of different people generally improves accuracy.

if you can get independent opinions from others, do it—this real wisdom of crowds is highly likely to improve your judgment. If you cannot, make the same judgment yourself a second time to create an “inner crowd.”

The propensity to find meaning in such statements is a trait known as bullshit receptivity. (Bullshit has become something of a technical term since Harry Frankfurt, a philosopher at Princeton University, published an insightful book, On Bullshit, in which he distinguished bullshit from other types of misrepresentation.)

you are not the same person at all times.

This behavior reflects a cognitive bias known as the gambler’s fallacy: we tend to underestimate the likelihood that streaks will occur by chance.

way. If our mind is a measuring instrument, it will never be a perfect one.

And because of the dynamics among group members—our emphasis here—the level of noise can be high.

If a song benefited from early popularity, it could do really well. If it did not get that benefit, the outcome could be very different.

Very similar groups can end up in divergent places because of social pressures.

deliberating juries were far noisier than statistical juries—a clear reflection of social influence noise. Deliberation had the effect of increasing noise.

compares the accuracy of predictions made by professionals, by machines, and by simple rules.

A measure that captures this intuition is the percent concordant (PC), which answers a more specific question: Suppose you take a pair of employees at random. What is the probability that the one who scored higher on an evaluation of potential also performs better on the

standard measure that social scientists use. The standard measure is the correlation coefficient (r), which varies between 0 and 1 when two variables are positively related. In the preceding example, the correlation between height and foot size is about .60.