Noise: A Flaw in Human Judgment

by Daniel Kahneman

important not to be exposed to irrelevant information early in the judgment process.

judgment entails the translation of an impression onto a scale, and if different judges use different scales, there will be noise.

structuring, we mean decomposing a judgment into its component parts, managing the process of data collection to ensure the inputs are independent of one another, and delaying the holistic discussion and the final judgment until all these inputs have been collected.

decision maker should aspire to achieve a more precise understanding of the likely gains from each decision hygiene strategy—and of the corresponding costs,

well trained, are more intelligent, and have the right cognitive style.

Highly skilled people are less noisy, and they also show less bias.

superiority over others is verifiable, thanks to the availability of outcome data.

experts’ judgment is entirely based on the respect they enjoy from their peers.

What makes a respect-expert?

Shared norms give professionals a sense of which inputs should be taken into account and how to make and justify their final judgments.

Another characteristic of respect-experts is their ability to make and explain their judgments with confidence.

So do many personality traits—including conscientiousness and grit, defined as perseverance and passion in the pursuit of long-term goals.

occupations but almost no people with below-average GMA among lawyers, chemists, or engineers.

GMA contributes significantly to the quality of performance in occupations that require judgment, even within a pool of high-ability individuals.

CRT questions attempt to measure how likely people are to override the first (and wrong) answer that comes to mind

The CRT is seen by many as one instrument to measure a broader concept: the propensity to use reflective versus impulsive thought processes.

In our terminology, the CRT can be seen as a measure of people’s propensity to rely on slow, System 2 thinking rather than on fast, System 1 thinking.

To be actively open-minded is to actively search for information that contradicts your preexisting hypotheses.

First, it is wise to recognize the difference between domains in which expertise can be confirmed by comparison with true values (such as weather forecasting) and domains that are the province of respect-experts.

Second, some judges are going to be better than their equally qualified and experienced peers.

actively search for new information that could contradict their prior beliefs, who are methodical in integrating that information into their current perspective, and who are willing, even eager, to change their minds as a result.

The challenge of learning to overcome a bias is to recognize that a new problem is similar to one we have seen elsewhere and that a bias that we have seen in one place is likely to materialize in other places.

target a specific bias, which they assume is present. This often-reasonable assumption is sometimes wrong.

it is difficult to know exactly which psychological biases are affecting a judgment.

checklists have a long history of improving decisions in high-stakes contexts and are particularly well suited to preventing the repetition of past errors.

you adopt techniques that reduce noise without ever knowing which underlying errors you are helping to avoid.

occasion noise: the variability between the judgments of the same experts looking at the same evidence twice.

validity requires reliability because, quite simply, it is hard to agree with reality if you cannot agree with yourself.

Wherever there is judgment, there is noise, and more of it than you think.

They illustrate a decision hygiene strategy that has applicability in many domains: sequencing information to limit the formation of premature intuitions.

More information is not always better, especially if it has the potential to bias judgments by leading the judge to form a premature intuition.

examiners should document their judgments at each step.

This sequence of steps helps experts avoid the risk that they see only what they are looking for.

occasion noise is driven by countless triggers, including mood and even outside temperature.

Less obvious is the possibility that your judgment can be altered by another trigger of occasion noise: information—even when it is accurate information.

selecting better judges produces better judgments.

aggregating multiple independent estimates.

Averaging is mathematically guaranteed to reduce noise: specifically, it divides it by the square root of th...

Because averaging does nothing to reduce bias, its effect on total error (MSE) depends on the proportions of bias and noise in it.

A select-crowd strategy, which selects the best judges according to the accuracy of their recent judgments

Delphi method.

method involves multiple rounds during which the participants submit estimates (or votes) to a moderator and remain anonymous to one another.

First, they used a large number of forecasts,

forecasts in terms of probabilities that an event would happen, rather than a binary “it will happen” or “it will not happen.”

Good calibration is one requirement for good forecasting.

opportunity to revise their forecasts continuously in light of new information.

“When the facts change, I change my mind. What do you do?”)

high resolution in addition to good calibration.

To produce a good score, you have not only to be right on average (i.e., well calibrated) but also to be willing to take a stand and differentiate among forecasts (i.e., have high resolution).

ease in thinking analytically and probabilistically.