Noise: A Flaw in Human Judgment

by Daniel Kahneman

Some judgments are biased; they are systematically off target. Other judgments are noisy, as people who are expected to agree end up at very different points around the target. Many organizations, unfortunately, are afflicted by both bias and noise.

A general property of noise is that you can recognize and measure it while knowing nothing about the target or bias.

Forecasts are noisy. Professional forecasters offer highly variable predictions about likely sales of a new product, likely growth in the unemployment rate, the likelihood of bankruptcy for troubled companies, and just about everything else.

Personnel decisions are noisy. Interviewers of job candidates make widely different assessments of the same people. Performance ratings of the same employees are also highly variable and depend more on the person doing the assessment than on the performance being assessed.

Occasion noise is the variability in judgments of the same case by the same person or group on different occasions. A surprising amount of occasion noise arises in group discussion because of seemingly irrelevant factors, such as who speaks first.

We conclude by offering a system we call the mediating assessments protocol: a general-purpose approach to the evaluation of options that incorporates several key practices of decision hygiene and aims to produce less noisy and more reliable judgments.

wherever there is judgment, there is noise—and more of it than you think.

First, judgment is difficult because the world is a complicated, uncertain place.

Third, noise can be reduced. The approach advocated by Frankel and implemented by the US Sentencing Commission—rules and guidelines—is one of several approaches that successfully reduce noise.

Variability in judgments is also expected and welcome in a competitive situation in which the best judgments will be rewarded.

When several companies (or several teams in the same organization) compete to generate innovative solutions to the same customer problem, we don’t want them to focus on the same approach.

In such settings, variability in ideas and judgments is again welcome, because variation is only the first step. In a second phase, the results of these judgments will be pitted against one another, and the best will triumph.

Most of us, most of the time, live with the unquestioned belief that the world looks as it does because that’s the way it is.

These beliefs, which have been called naive realism, are essential to the sense of a reality we share with other people.

Confidence is nurtured by the subjective experience of judgments that are made with increasing fluency and ease, in part because they resemble judgments made in similar cases in the past.

Most organizations prefer consensus and harmony over dissent and conflict.

While each case is in some sense unique, judgments like these are recurrent decisions.

Historical analyses, like case studies of management successes and failures, aim to understand how an essentially unique judgment was made.

From the perspective of noise reduction, a singular decision is a recurrent decision that happens only once.

Judgment can therefore be described as measurement in which the instrument is a human mind. Implicit in the notion of measurement is the goal of accuracy—to approach truth and minimize error.

Indeed, the word judgment is used mainly where people believe they should agree.

Matters of judgment differ from matters of opinion or taste, in which unresolved differences are entirely acceptable.

Matters of judgment, including professional judgments, occupy a space between questions of fact or computation on the one hand and matters of taste or opinion on the other. They are defined by the expectation of bounded disagreement.

Selective attention and selective recall are a source of variability across people.

You did not construct a plan for answering the question. Without being fully aware of what you were doing, your mind worked to construct a coherent impression of Michael’s strengths and weaknesses and of the challenges he faces.

The Gambardi exercise is an example of a nonverifiable predictive judgment, for two separate reasons: Gambardi is fictitious and the answer is probabilistic.

What made you feel you got the judgment right, or at least right enough to be your answer? We suggest this feeling is an internal signal of judgment completion, unrelated to any outside information.

But the number on which you settled, whatever it is, gave you the sense of coherence you needed. The aim of judgment, as you experienced it, was the achievement of a coherent solution.

System noise is inconsistency, and inconsistency damages the credibility of the system.

in professional judgments of all kinds, whenever accuracy is the goal, bias and noise play the same role in the calculation of overall error.

Gauss proposed a rule for scoring the contribution of individual errors to overall error. His measure of overall error—called mean squared error (MSE)—is the average of the squares of the individual errors of measurement.

The mean contains more information; it is affected by the size of the numbers, while the median is affected only by their order.

Error in a single measurement = Bias + Noisy Error

Overall Error (MSE) = Bias2 + Noise2

As the mathematical expression and its visual representation both suggest, bias and noise play identical roles in the error equation.

A widely accepted maxim of good decision making is that you should not mix your values and your facts.

Good decision making must be based on objective and accurate predictive judgments that are completely unaffected by hopes and fears, or by preferences and values.

Occasion noise is the product of a second lottery. This lottery picks the moment when the professional makes a judgment, the professional’s mood, the sequence of cases that are fresh in mind, and countless other features of the occasion.

wisdom-of-crowds effect: averaging the independent judgments of different people generally improves accuracy.

if you can get independent opinions from others, do it—this real wisdom of crowds is highly likely to improve your judgment. If you cannot, make the same judgment yourself a second time to create an “inner crowd.”

There is at least one source of occasion noise that we have all noticed: mood.

The propensity to find meaning in such statements is a trait known as bullshit receptivity.

Harry Frankfurt, a philosopher at Princeton University, published an insightful book, On Bullshit, in which he distinguished bullshit from other types of misrepresentation.)

When a person is considering a case, the decisions that immediately preceded it serve as an implicit frame of reference.

This behavior reflects a cognitive bias known as the gambler’s fallacy: we tend to underestimate the likelihood that streaks will occur by chance.

But independence is a prerequisite for the wisdom of crowds.

People think that they know what is right or probably right, but they nonetheless go along with the apparent consensus of the group, or the views of early speakers, to stay in the group’s good graces.

Internal discussions often create greater confidence, greater unity, and greater extremism, frequently in the form of increased enthusiasm.

PC is an immediately intuitive measure of covariation, which is a large advantage, but it is not the standard measure that social scientists use. The standard measure is the correlation coefficient (r), which varies between 0 and 1 when two variables are positively related. In the preceding example, the correlation between height and foot size is about .60.

The informal approach you took to this problem is known as clinical judgment. You consider the information, perhaps engage in a quick computation, consult your intuition, and come up with a judgment. In fact, clinical judgment is the process that we have described simply as judgment in this book.