Noise: A Flaw in Human Judgment

by Daniel Kahneman

Bias and noise—systematic deviation and random scatter—are different components of error.

Some judgments are biased; they are systematically off target. Other judgments are noisy, as people who are expected to agree end up at very different points around the target. Many organizations, unfortunately, are afflicted by both bias and noise.

A general property of noise is that you can recognize and measure it while knowing nothing about the target or bias.

To understand error in judgment, we must understand both bias and noise.

in public conversations about human error and in organizations all over the world, noise is rarely recognized. Bias is the star of the show. Noise is a bit player, usually offstage.

In real-world decisions, the amount of noise is often scandalously high.

Wherever you look at human judgments, you are likely to find noise. To improve the quality of our judgments, we need to overcome noise as well as bias.

Occasion noise is the variability in judgments of the same case by the same person or group on different occasions.

A surprising amount of occasion noise arises in group discussion because of seemingly irrelevant factors, such as who speaks first.

the key advantage of rules, formulas, and algorithms over humans when it comes to making predictions: contrary to popular belief, it is not so much the superior insight of rules but their noiselessness.

mediating assessments protocol: a general-purpose approach to the evaluation of options that incorporates several key practices of decision hygiene and aims to produce less noisy and more reliable judgments.

wherever there is judgment, there is noise—and more of it than you think.

The very idea of due process of law seemed, to many, to call for open-ended judicial discretion.

The price of reducing noise was to make decisions unacceptably mechanical.

While few people object to the principle of judicial discretion, almost everyone disapproves of the magnitude of the disparities it produces.

efforts at noise reduction often raise objections and run into serious difficulties.

When traders make different assessments of the value of a stock, some of them will make money, and others will not. Disagreements make markets.

A frequent misconception about unwanted variability in judgments is that it doesn’t matter, because random errors supposedly cancel one another out.

In noisy systems, errors do not cancel out. They add up.

Noise was like a leak in the basement. It was tolerated not because it was thought acceptable but because it had remained unnoticed.

The noise audits suggested that respected professionals—and the organizations that employ them—maintained an illusion of agreement while in fact disagreeing in their daily professional judgments.

Most of us, most of the time, live with the unquestioned belief that the world looks as it does because that’s the way it is. There is one small step from this belief to another: “Other people view the world much the way I do.” These beliefs, which have been called naive realism, are essential to the sense of a reality we share with other people.

One interpretation is enough, and we experience it as true. We do not go through life imagining alternative ways of seeing what we see.

Confidence is nurtured by the subjective experience of judgments that are made with increasing fluency and ease, in part because they resemble judgments made in similar cases in the past.

How had the leaders of the company remained unaware of their noise problem? There are several possible answers here, but one that seems to play a large role in many settings is simply the discomfort of disagreement.

Most organizations prefer consensus and harmony over dissent and conflict. The procedures in place often seem expressly designed to minimize the frequency of exposure to actual disagreements and, when such disagreements happen, to explain them away.

Bad judgment is much easier to identify than good judgment. The calling out of egregious mistakes and the marginalization of bad colleagues will not help professionals become aware of how much they disagree when making broadly acceptable judgments.

On the contrary, the easy consensus about bad judgments may even reinforce the illusion of agreement. The true lesson, about the ubiquity of system noise, will never be learned.

noise is a consequence of the informal nature of judgment.

wherever there is judgment, there is noise, and more of it than you think.

It seems much harder, or perhaps even impossible, to apply the idea of noise to a category of judgments that we call singular decisions.

Important political decisions are often good examples of singular decisions, as are the most fateful choices of military commanders.

Analyses of recurrent decisions have often taken a statistical bent, with social scientists assessing many similar decisions to discern patterns, identify regularities, and measure accuracy.

In contrast, discussions of singular decisions typically adopt a causal view; they are conducted in hindsight and are focused on identifying the causes of what happened.

There is no direct way to observe the presence of noise in singular decisions.

noise in the decision makers and in the decision-making process implies that the singular decision could have been different.

Even when the virus hit them roughly at the same time and in a similar manner, there were wide differences in responses. This variation provides clear evidence of noise in different countries’ decision making.

a singular decision is a recurrent decision that happens only once. Whether you make a decision only once or a hundred times, your goal should be to make it in a way that reduces both bias and noise. And practices that reduce error should be just as effective in your one-of-a-kind decisions as in your repeated ones.

Judgment can therefore be described as measurement in which the instrument is a human mind.

Matters of judgment, including professional judgments, occupy a space between questions of fact or computation on the one hand and matters of taste or opinion on the other. They are defined by the expectation of bounded disagreement.

Many professional judgments are nonverifiable.

Verifiability does not change the experience of judgment.

What made you feel you got the judgment right, or at least right enough to be your answer? We suggest this feeling is an internal signal of judgment completion, unrelated to any outside information.

One approach to the evaluation of the process of judgment is to observe how that process performs when it is applied to a large number of cases.

Another question that can be asked about the process of judgment is whether it conforms to the principles of logic or probability theory.

Focusing on the process of judgment, rather than its outcome, makes it possible to evaluate the quality of judgments that are not verifiable,

two ways of evaluating a judgment: by comparing it to an outcome and by assessing the quality of the process that led to it.

when the judgment is verifiable, the two ways of evaluating it may reach different conclusions in a single case.

Scholars of decision-making offer clear advice to resolve this tension: focus on the process, not on the outcome of a single case.

what people usually claim to strive for in verifiable judgments is a prediction that matches the outcome. What they are effectively trying to achieve, regardless of verifiability, is the internal signal of completion provided by the coherence between the facts of the case and the judgment. And what they should be trying to achieve, normatively speaking, is the judgment process that would produce the best judgment over an ensemble of similar cases.