Noise: A Flaw in Human Judgment

by Daniel Kahneman

The power of first impressions is not the only problematic aspect of interviews. Another is that as interviewers, we want the candidate sitting in front of us to make sense

not a single interviewer realized that the candidates were giving random answers. Worse, when asked to estimate whether they were “able to infer a lot about this person given the amount of time we spent together,” interviewers in this “random” condition were as likely to agree as those who had met candidates responding truthfully. Such is our ability to create coherence. As we can often find an imaginary pattern in random data or imagine a shape in the contours of a cloud, we are capable of finding logic in perfectly meaningless answers.

Google stringently enforces a rule that not all companies observe: the company makes sure that the interviewers rate the candidate separately, before they communicate with one another.

structuring complex judgments.

a structured complex judgment is defined by three principles: decomposition, independence, and delayed holistic judgment.

structured behavioral interviews.

work sample tests are among the best predictors of on-the-job performance.

delayed holistic judgment,

Google allows judgment and intuition in its decision-making process only after all the evidence has been collected and analyzed.

There is overwhelming evidence of the superiority of structured judgment processes (including structured interviews) in hiring.

mettle.

“the persistence of an illusion.”

recruiters and candidates severely underestimate the noise i...

“In traditional, informal interviews, we often have an irresistible, intuitive feeling of understanding the candidate and knowing whether the person fits the bill. We must learn to distrust that feeling.”

“Traditional interviews are dangerous not only because of biases but also because of noise.”

“We must add structure to our interviews and, more broadly, to our selection processes. Let’s start by defining much more clearly and specifically what we are looking for in candidates, and let’s make sure we evaluate...

mediating assessments protocol.

Just like a recruiter in an unstructured interview, we are at risk of using all the debate to confirm our first impressions.

outside view

base rate,

reference class,

the goal was to make evaluations as comparative as possible, because relative judgments are better than absolute ones.

assessments should be as independent of one another as possible, to reduce the risk that one assessment would influence the others. Accordingly, he assigned different analysts to the different assessments, and he instructed them to work independently.

Jane, an experienced member of the team, was charged with two assessments. Jeff chose the two to be as different from each other as possible, and he instructed Jane to complete the first assessment and prepare the report on it before turning to the other.

Each chapter should focus on one assessment and, as requested by Joan, lead to a conclusion in the form of a rating.

if there is information that seems inconsistent or even contradictory with the main rating, don’t sweep anything under the rug. Your job is not to sell your recommendation. It is to represent the truth. If it is complicated, so be it—it often is.”

And if you run into something that really gives you pause—a potential deal breaker—you should, of course, report it immediately.”

estimate-talk-estimate method, which combines the advantages of deliberation and those of averaging independent opinions.

Then Joan asked the board members to use a voting app on their phones to give their own rating on the assessment—either the same as the deal team’s proposed rating or a different one. The distribution of ratings was projected immediately on the screen, without identifying the raters. “This is not a vote,” Joan explained. “We are just taking the temperature of the room on each topic.” By getting an immediate read on each board member’s independent opinion before starting a discussion, Joan reduced the danger of social influence and information cascades.

“we are all reasonable people and we disagree, so this must be a subject on which reasonable people can disagree.”

When the discussion of an assessment drew to a close, Joan asked the board members to vote again on a rating. Most of the time, there was more convergence than in the initial round. The same sequence—a first estimate, a discussion, and a second estimate—was repeated for each assessment.

Finally, it was time to reach a conclusion about the deal. To facilitate the discussion, Jeff showed the list of assessments on the whiteboard, with, for each assessment, the average...

Joan interrupted the discussion. “This is not just about computing a simple combination of the assessment ratings,” she said. “We have delayed intuition, but now is the time to use it. What we need now is your judgment.”

She knew that, particularly with important decisions, people reject schemes that tie their hands and do not let them use their judgment. She had seen how decision makers game the system when they know that a formula will be used.

Joan also knew that letting the board members use their intuition at this stage was very different from having them use it earlier in the process. Now that the assessments were available and known to all, the final decision was safely anchored on these fact-based, thoroughly discussed ratings.

Main steps of the mediating assessments protocol 1. At the beginning of the process, structure the decision into mediating assessments. (For recurring judgments, this is done only once.) 2. Ensure that whenever possible, mediating assessments use an outside view. (For recurring judgments: use relative judgments, with a case scale if possible.) 3. In the analytical phase, keep the assessments as independent of one another as possible. 4. In the decision meeting, review each assessment separately. 5. On each assessment, ensure that participants make their judgments individually; then use the estimate-talk-estimate method. 6. To make the final decision, delay intuition, but don’t ban it.

By implementing these techniques, the mediating assessments protocol aims to change the decision process to introduce as much decision hygiene as possible.

the word process evokes bureaucracy, red tape, and delays.

these concerns are misguided.

adding complexity to the decision-making processes of an organization that is already bureaucratic will not make things better. But decision hygiene need not be slow and certainly doesn’t need to be bureaucratic. On the contrary, it promotes challenge and debate, not the stifling consensus that characterizes bureaucracies.

Handwashing does not prevent all diseases. Likewise, decision hygiene will not prevent all mistakes. It will not make every decision brilliant. But like handwashing, it addresses an invisible yet pervasive and damaging problem. Wherever there is judgment, there is noise, and we propose decision hygiene as a tool to reduce it.

seven major objections to efforts to reduce or eliminate noise.

First, reducing noise can be expensive; it might not be worth the trouble.

Second, some strategies introduced to reduce noise might introduce errors of their own.

Third, if we want people to feel that they have been treated with respect and dignity, we might have to tolerate some noise.

Fourth, noise might be essential to accommodate new values and hence to allow moral and political evolution.

Fifth, some strategies designed to reduce noise might encourage opportunistic behavior, allowing people to game the system or evade prohibitions.

Sixth, a noisy process might be a good deterrent.

Finally, people do not want to be treated as if they are mere things, or cogs in some kind of machine. Some noise-reduction strategies might squelch people’s creativity and prove demoralizing.

Although we will address these objections as sympathetically as we can, we by no means endorse them, at least not if they are taken as reasons to reject the general goal of reducing noise.