Noise: A Flaw in Human Judgment

by Daniel Kahneman

the fact that a fingerprint is found on the murder weapon is sufficient to convict, but the absence of that print is

not sufficient to exonerate a suspect.

“It is easier to bias forensic experts towards the non-committal conclusion of ‘inconclusive’ than to the definitive ‘identification’ conclusion.”

Wherever there is judgment, there is noise, and more of it than you think.

However low the error rate of fingerprint identification may be, it is not zero,

These remarks essentially amount to a denial of the existence of confirmation bias.

bias blind spot: the tendency to acknowledge the presence of bias in others, but not in oneself.

In other words, about half of these forensic professionals believe that their colleagues’ judgments are noisy but that their own are not. Noise can be an invisible problem, even to people whose job is to see the invisible.

redesigning its procedures to minimize the risk of confirmation bias.

decision hygiene strategy that has applicability in many domains: sequencing information to limit the formation of premature intuitions.

by giving the examiners only the information they need,

the approach Dror and colleagues codified is called linear sequential unmasking.

examiners should document their judgments at each step.

sequence of steps helps experts avoid the risk that they see only what they are looking for.

This requirement limits the risk that an early intuition biases the entire process.

When a different examiner is called on to verify the identification made by the first person, the second person should not be aware of the first judgment.

presence of noise

it is also revealing.

shows how our confidence in expert human judgment can sometimes be exaggerated and how a noise audit can reveal an unexpected amount of noise.

The main decision hygiene strategy this case illustrates—sequencing information—has broad applicability as a safeguard against occasion noise.

revisit your judgment at different points in time, when the triggers of occasion noise are likely to be different.

The titles of two Hitchcock movies sum it up: a good decision maker should aim to keep a “shadow of a doubt,” not to be “the man who knew too much.”

there can be a bias cascade.”

Many judgments involve forecasting.

Fundamental choices of private and public institutions often depend on them.

forecasters tend to be overconfident:

Forecasters are also noisy.

“unreliability is a source of error in judgmental forecasting.”

Occasion noise is common; forecasters do not always agree with themselves.

Between-person noise is also...

seen many examples of noisy forecasts, and research on forecasting uncovers many more.

two noise-reduction strategies that have broad applicability.

selecting better judges produces better judgments.

most universally applicable decision hygiene strategies: aggregating multiple...

Averaging is mathematically guaranteed to...

This statistical law is the engine of the wisdom-of-crowds approach,

wisdom of crowds works best when judgments are independent,

unweighted average of a group of forecasters outperforms most and sometimes all individual forecasts.

It is also easier for decision makers who respect expertise to understand and adopt a strategy that relies not only on aggregation but also on selection.

prediction markets,

Many companies in various industries have used prediction markets to aggregate diverse views.

Delphi method.

The method benefits both from aggregation and social learning.

can be challenging to implement.

mini-Delphi, can be deployed within a s...

estimate-talk-e...

The consensus judgment is the average of the individual estimates obtained in that second round.

teamed up to improve our understanding of forecasting and, in particular, why some people are good at it.

wanted to learn whether some people are especially good forecasters.

whether the ability to forecast could be taught or at least improved.