Noise: A Flaw in Human Judgment

by Daniel Kahneman

System noise is undesirable variability in the judgments of the same case by multiple individuals.

Level noise is variability in the average level of judgments by different judges.

Pattern noise is variability in judges’ responses to particular cases.

“Level noise is when judges show different levels of severity. Pattern noise is when they disagree with one another on which defendants deserve more severe or more lenient treatment. And part of pattern noise is occasion noise—when judges disagree with themselves.”

if you can get independent opinions from others, do it—this real wisdom of crowds is highly likely to improve your judgment. If you cannot, make the same judgment yourself a second time to create an “inner crowd.”

actively trying to argue against yourself to find another perspective on the problem.

First, the choice of a scale can make a large difference in the amount of noise in judgments, because ambiguous scales are noisy. Second, replacing absolute judgments with relative ones, when feasible, is likely to reduce noise.

When do you feel confident in a judgment? Two conditions must be satisfied: the story you believe must be comprehensively coherent, and there must be no attractive alternatives.

while the existence of occasion noise is surprising and even disturbing, there is no indication that within-person variability is larger than between-person differences. The most important component of system noise is the one we had initially neglected: stable pattern noise, the variability among judges in their judgments of particular cases.

Noise is mostly a by-product of our uniqueness, of our “judgment personality.”

the average of errors (the bias) and the variability of errors (the noise)

Judgments are both less noisy and less biased when those who make them are well trained, are more intelligent, and have the right cognitive style. In other words: good judgments depend on what you know, how well you think, and how you think.

actively open-minded and willing to learn from new information.

it is wise to recognize the difference between domains in which expertise can be confirmed by comparison with true values (such as weather forecasting) and domains that are the province of respect-experts.

But the evidence suggests that if the goal is to reduce error, it is better for leaders (and others) to remain open to counterarguments and to know that they might be wrong. If they end up being decisive, it is at the end of a process, not at the start.

ex post, by correcting judgments after they have been made, or ex ante, by intervening before a judgment or decision.

The challenge of learning to overcome a bias is to recognize that a new problem is similar to one we have seen elsewhere and that a bias that we have seen in one place is likely to materialize in other places.

checklists have a long history of improving decisions in high-stakes contexts and are particularly well suited to preventing the repetition of past errors.

Instead of offering a gut feeling or some kind of global hunch, they ask and try to answer an assortment of subsidiary questions.

Superforecasters also excel at taking the outside view, and they care a lot about base rates.

the mechanical approach is superior both in general and in the specific case of work performance prediction.

aggregation works—but only if the judgments are independent.

a structured complex judgment is defined by three principles: decomposition, independence, and delayed holistic judgment.

Sensible organizations well understand that the amount of discretion they grant is closely connected with the level of trust they have in their agents.

Whenever numerous decisions must be made, there might well be a lot of noise, and there is a strong argument for clear rules.

Level noise is the variability of the average judgments made by different individuals.

The main source of pattern noise is stable: it is the difference in the personal, idiosyncratic responses of judges to the same case.

This stable pattern noise reflects the uniqueness of judges: their response to cases is as individual as their personality.

occasion noise.

We say that a judge takes the outside view of a case when she considers it as a member of a reference class of similar cases rather than as a unique problem.

People can reduce excessive coherence by breaking down the judgment problem into a series of smaller tasks.

Intuition need not be banned, but it should be informed, disciplined, and delayed.

different aspects of the problem are considered separately and that the formation of a holistic judgment is delayed until the profile of assessments is complete.