Noise: A Flaw in Human Judgment

by Daniel Kahneman

Forecasts are noisy. Professional forecasters offer highly variable predictions about likely sales of a new product, likely growth in the unemployment rate, the likelihood of bankruptcy for troubled companies, and just about everything else. Not only do they disagree with each other, but they also disagree with themselves. For example, when the same software developers were asked on two separate days to estimate the completion time for the same task, the hours they projected differed by 71%, on average.

like a measuring instrument, the human mind is imperfect—it is both biased and noisy.

It can be sensible to place your trust in people who look and sound intelligent and who can articulate a compelling rationale for their judgments, but this strategy is insufficient and may even backfire.

there is some evidence that actively open-minded thinking is a teachable skill.

Everyone agrees that in some contexts, forecasters are biased. For example, official agencies show unrealistic optimism in their budget forecasts.

The research also offers suggestions for reducing noise and bias. We will not review them exhaustively here, but we will focus on two noise-reduction strategies that have broad applicability. One is an application of the principle we mentioned in chapter 18: selecting better judges produces better judgments. The other is one of the most universally applicable decision hygiene strategies: aggregating multiple independent estimates. The easiest way to aggregate several forecasts is to average them. Averaging is mathematically guaranteed to reduce noise:

if you average one hundred judgments, you will reduce noise by 90%, and if you average four hundred judgments, you will reduce it by 95%—essentially eliminating it. This statistical law is the engine of the wisdom-of-crowds approach,

given our objective ignorance of future events, it is much better to formulate probabilistic forecasts.

Tetlock and colleagues did not just ask their forecasters to make one probability estimate about whether an event would happen in, say, twelve months. They gave the participants the opportunity to revise their forecasts continuously in light of new information.

This approach mirrors what is expected of forecasters in business and government, who should also be updating their forecasts frequently on the basis of new information,

The basic message that emerges from this research is a simple one: most ratings of performance have much less to do with the performance of the person being rated than we would wish. As one review summarizes it, “the relationship between job performance and ratings of job performance is likely to be weak or at best uncertain.”

Some evidence suggests that 360-degree feedback is a useful tool in that it predicts objectively measurable performance. Unfortunately, the use of this feedback system has created its own problems.

the fatal flaw of forced ranking is not the “ranking,” but the “forced.” Whenever judgments are forced onto an inappropriate scale, either because a relative scale is used to measure an absolute performance or because judges are forced to distinguish the indistinguishable, the choice of the scale mechanically adds noise.

The large subject of performance evaluation raises many questions, both practical and philosophical. Some people ask, for instance, to what extent the notion of individual performance is meaningful in today’s organizations, where outcomes often depend on how people interact with one another.

if your goal is to bring out the best in people, you can reasonably ask whether measuring individual performance and using that measurement to motivate people through fear and greed is the best approach (or even an effective one).

Almost all companies aggregate the judgments of multiple interviewers on the same candidate. Not to be outdone, Google sometimes had candidates suffer through twenty-five interviews! One of the conclusions of Bock’s review was to reduce that number to four, as he found that additional interviews added almost no predictive validity to what was achieved by the first four.

aggregation works—but only if the judgments are independent.

The three principles—once more, decomposition, independent assessment on each dimension, and delayed holistic judgment—do not necessarily provide a template for all organizations trying to improve their selection processes. But the principles are broadly consistent with the recommendations that organizational psychologists have formulated over the years. In fact, the principles bear some resemblance to the selection method that one of us (Kahneman) implemented in the Israeli army as early as 1956 and described in Thinking, Fast and Slow. That process, like the one Google put in place, formalized an evaluation structure (the list of personality and competence dimensions that had to be evaluated). It required interviewers to elicit objective evidence relevant to each dimension in turn and to score that dimension before moving on to the next. And it allowed recruiters to use judgment and intuition to reach a final decision—but only after the structured evaluation had taken place.

“If you want to deter misconduct, you should tolerate some noise. If students are left wondering about the penalty for plagiarism, great—they will avoid plagiarizing. A little uncertainty in the form of noise can magnify deterrence.”

“Creative people need space. People aren’t robots. Whatever your job, you deserve some room to maneuver. If you’re hemmed in, you might not be noisy, but you won’t have much fun and you won’t be able to bring your original ideas to bear.”