Noise: A Flaw in Human Judgment

by Daniel Kahneman

In particular, judgments of one’s ability to make precise predictions, even from limited information, are notoriously overconfident. What we said of noise in predictive judgments can also be said of objective ignorance: wherever there is prediction, there is ignorance, and more of it than you think.

“Models do better than people, but not by much. Mostly, we find mediocre human judgments and slightly better models. Still, better is good, and models are better.”

“We may never be comfortable using a model to make these decisions—we just need the internal signal to have enough confidence. So let’s make sure we have the best possible decision process.”

The ability to make a prediction is a measure of whether such a causal chain has indeed been identified. And correlation, the measure of predictive accuracy, is a measure of how much causation we can explain.

There is a psychological explanation for this observation. Some events are surprising: a deadly pandemic, an attack on the Twin Towers, a star hedge fund that turns out to be a Ponzi scheme. In our personal lives as well, there are occasional shocks: falling in love with a stranger, the sudden death of a young sibling, an unexpected inheritance. Other events are actively expected, like a second-grader’s return from school at the appointed time. But most human experience falls between these two extremes. We are sometimes in a state in which we actively expect a specific event, and we are sometimes surprised. But most things take place in the broad valley of the normal, where events are neither entirely expected nor especially surprising. At this moment, for example, you have no specific expectation of what is coming in the next paragraph. You would be surprised to find we suddenly switched to Turkish, but there is a wide

range of things we could say without shocking you.

Causal thinking helps us make sense of a world that is far less predictable than we think. It also explains why we view the world as far more predictable than it really is. In the valley of the normal, there are no surprises and no inconsistencies. The future seems as predictable as the past. And noise is neither heard nor seen.

“We think we understand what is going on here, but could we have predicted it?”

when people forecast how long it will take them to complete a project, the mean of their estimates is usually much lower than the time they will actually need. This familiar psychological bias is known as the planning fallacy.

regardless of the true reasons for your belief, you will be inclined to accept any argument that appears to support it, even when the reasoning is wrong.

“We know we have psychological biases, but we should resist the urge to blame every error on unspecified ‘biases.’” “When we substitute an easier question for the one we should be answering, errors are bound to occur. For instance, we will ignore the base rate when we judge probability by similarity.” “Prejudgments and other conclusion biases lead people to distort evidence in favor of their initial position.” “We form impressions quickly and hold on to them even when contradictory information comes in. This tendency is called excessive coherence.”

Our ability to compare cases is much better than our ability to place them on a scale.

Speaking of Matching

“Both of us say this movie is very good, but you seem to have enjoyed it a lot less than I did. We’re using the same words, but are we using the same scale?” “We thought Season 2 of this series would be just as spectacular as Season 1. We made a matching prediction, and it was wrong.” “It is hard to remain consistent when grading these essays. Should you try ranking them instead?”

ambiguous scales are common, which means that the punitive-damages study holds two general lessons, applicable in business, education, sports, government, and elsewhere. First, the choice of a scale can make a large difference in the amount of noise in judgments, because ambiguous scales are noisy. Second, replacing absolute judgments with relative ones, when feasible, is likely to reduce noise.

Speaking of Scales “There is a lot of noise in our judgments. Could this be because we understand the scale differently?” “Can we agree on an anchor case that will serve as a reference point on the scale?” “To reduce noise, maybe we should replace our judgments with a ranking?”

The main implication of this view of confidence is that subjective confidence in one’s judgment by no means guarantees accuracy. Moreover, the suppression of alternative interpretations—a well-documented process in perception—could induce what we have called the illusion of agreement (see chapter 2). If people cannot imagine possible alternatives to their conclusions, they will naturally assume that other observers must reach the same conclusion, too.

The fact that in these studies level noise is generally not the larger component of system noise is already an important message, because level noise is the only form of noise that organizations can (sometimes) monitor without conducting noise audits.

Causes are natural; statistics are difficult.

As we have also noted, many judgments are not verifiable. Within certain boundaries, we cannot easily know or uncontroversially define the true value at which judgments are aiming. Underwriting and criminal sentencing fall in this category, as do wine tasting, essay grading, book and movie reviewing, and innumerable other judgments. Yet some professionals in these domains come to be called experts. The confidence we have in these experts’ judgment is entirely based on the respect they enjoy from their peers. We call them respect-experts.

More generally, how do people who are themselves respected for the quality of their judgment decide to trust someone as an expert when there is no data to establish expertise objectively? What makes a respect-expert?

Part of the answer is the existence of shared norms, or professional doctrine.

Beyond a knowledge of shared norms, experience is necessary, too.

Another characteristic of respect-experts is their ability to make and explain their judgments with confidence.

We tend to put more trust in people who trust themselves than we do in those who show their doubts.

The confidence heuristic points to the fact that in a group, confident people have more weight than others, even if th...

The conclusion is clear. GMA contributes significantly to the quality of performance in occupations that require judgment, even within a pool of high-ability individuals. The notion that there is a threshold beyond which GMA ceases to make a difference is not supported by the evidence. This

People of high mental ability are more likely than others to make better judgments and to be true experts, but they are also more likely to impress their peers, earn others’ trust, and become respect-experts in the absence of any reality feedback. Medieval astrologers must have been among the highest-GMA people of their time.

cognitive reflection test

(CRT),

Lower CRT scores are associated with many real-world judgments and beliefs, including belief in ghosts, astrology, and extrasensory perception. The scores predict whether people will fall for blatantly inaccurate “fake news.” They are even associated with how much people will use their smartphones.

The CRT is seen by many as one instrument to measure a broader concept: the propensity to use reflective versus impulsive thought processes. Simply put, some people like to engage in careful thought, whereas others, faced with the same problem, tend to trust their first impulses. In our terminology, the CRT can be seen as a measure of people’s propensity to rely on slow, System 2 thinking rather than on fast, System 1 thinking.

The personality of people with excellent judgment may not fit the generally accepted stereotype of a decisive leader. People often tend to trust and like leaders who are firm and clear and who seem to know, immediately and deep in their bones, what is right. Such leaders inspire confidence. But the evidence suggests that if the goal is to reduce error, it is better for leaders (and others) to remain open to counterarguments and to know that they might be wrong. If they end up being decisive, it is at the end of a process, not at the start.

In the United States, federal agencies must compile a formal regulatory impact analysis before they issue expensive regulations designed to clean the air or water, reduce deaths in the workplace, increase food safety, respond to public health crises, reduce greenhouse gas emissions, or increase homeland security. A dense, technical document with an unlovely name (OMB Circular A-4) and spanning nearly fifty pages sets out the requirements of the analysis. The requirements are clearly designed to counteract bias. Agencies must explain why the regulation is needed, consider both more and less stringent alternatives, consider both costs and benefits, present the information in an unbiased manner, and discount the future appropriately.

Noise, on the other hand, is unpredictable error that we cannot easily see or explain. That is why we so often neglect it—even when it causes grave damage. For this reason, strategies for noise reduction are to debiasing what preventive hygiene measures are to medical treatment: the goal is to prevent an unspecified range of potential errors before they occur.

Correcting a well-identified bias may at least give you a tangible sense of achieving something. But the procedures that reduce noise will not. They will, statistically, prevent many errors. Yet you will never know which errors. Noise is an invisible enemy, and preventing the assault of an invisible enemy can yield only an invisible victory.

we will focus on two noise-reduction strategies that have broad applicability. One is an application of the principle we mentioned in chapter 18: selecting better judges produces better judgments. The other is one of the most universally applicable decision hygiene strategies: aggregating multiple independent estimates. The easiest way to aggregate several forecasts is to average them. Averaging is mathematically guaranteed to reduce noise: specifically, it divides it by the square root of the number of judgments averaged. This means that if you average one hundred judgments, you will reduce noise by 90%, and if you average four hundred judgments, you will reduce it by 95%—essentially eliminating it. This statistical law is the engine of the wisdom-of-crowds approach, discussed in chapter 7

Because averaging does nothing to reduce bias, its effect on total error (MSE) depends on the proportions of bias and noise in it.

Straight averaging is not the only way to aggregate forecasts. A select-crowd strategy, which selects the best judges according to the accuracy of their recent judgments and averages the judgments of a small number of judges (e.g., five), can be as effective as straight averaging.

“We should strive to be in perpetual beta, like the superforecasters.

white coat syndrome—your blood pressure goes up in doctors’ offices!”

The medical profession is likely to rely on algorithms more and more in the future; they promise to reduce both bias and noise and to save lives and money in the process.

Speaking of Guidelines in Medicine “Among doctors, the level of noise is far higher than we might have suspected. In diagnosing cancer and heart disease—even in reading X-rays—specialists sometimes disagree. That means that the treatment a patient gets might be a product of a lottery.” “Doctors like to think that they make the same decision whether it’s Monday or Friday or early in the morning or late in the afternoon. But it turns out that what doctors say and do might well depend on how tired they are.” “Medical guidelines can make doctors less likely to blunder at a patient’s expense. Such guidelines can also help the medical profession as a whole, because they reduce variability.”

You might find that the answer lies in how you used the scale—what we have called level noise. Perhaps you thought a 5 requires something truly extraordinary, whereas the other rater thought that it merely requires something unusually good.

This variability is largely the product of pattern noise, the difference in interviewers’ idiosyncratic reactions to a given interviewee.

Why do first impressions end up driving the outcome of a much longer interview? One reason is that in a traditional interview, interviewers are at liberty to steer the interview in the direction they see fit. They are likely to ask questions that confirm an initial impression.

As we can often find an imaginary pattern in random data or imagine a shape in the contours of a cloud, we are capable of finding logic in perfectly meaningless answers.

however much we would like to believe that our judgment about a candidate is based on facts, our interpretation of facts is colored by prior attitudes.

risk-averse or risk-seeking.

Organizations all over the world see bias as a villain. They are right. They do not see noise that way. They should.