Noise: A Flaw in Human Judgment

by Daniel Kahneman

performance rating scales must be anchored on descriptors that are sufficiently specific to be interpreted consistently.

behaviorally anchored rating scales

frame-of-reference training,

help ensure consistency

case scale,

relative judgment.

Frame-of-reference training has been known for decades and provides demonstrably less noisy and more accurate ratings.

complex and time-consuming.

Tellingly, the majority of studies of frame-of-reference rater training have so far been conducted on students, not on actual managers.

whether there exists “star talent” making a hugely disproportionate contribution.

Reducing this noise is a challenge that cannot be solved by simple technological fixes. It requires clear thinking about the judgments that

raters are expected to make.

improve judgments by clarifying the rating scale and training people to use it consistently. This noise-reductio...

recruiting interview

“It is rare, even unthinkable, for someone to be hired without some type of

interview.”

deep-seated belief in the value of judgment when it comes to choosing the people we will work with.

human capacities are after all the chief national resources.”

No complex judgment task has been the focus of so much field research.

extrapolated to many judgments involving a choice among several options.

standard interviews

are not very informative. To put it more starkly, they are often useless.

correlation between the rating an evaluator gives a candidate after an interview and the candidate’s eventual success on the job.

can be assumed to be a good predictor of how candidates will perform.

The definition of success is a nontrivial problem.

However, for the purpose of evaluating the quality of an employer’s judgments when selecting employees, it seems reasonable to use the judgments that the same employer makes when evaluating the employees thus hired. Any analysis of the quality of hiring decisions must make this assumption.

very good correlation by social science standards—but not a very good one on which to base your decisions.

percent concordant (PC)

hardly a fail-safe way to make important decisions.

main purpose of interviews is clearly one of selection. And at that task, they are not exactly a terrific success.

objective ignorance

Job performance depends on many things,

This uncertainty limits the predictive validity of interviews and, indeed, any other personnel selection technique.

Interviews are also a minefield of psychological biases.

For instance, physical appearance plays a large part in the evaluation of candidates, even for positions where it should matter little or not at all.

there is noise as well:

candidate may not behave in exactly the same way with different interviewers.

correlation between their ratings is far from perfect.

same two candidates in the same panel interview,

will still disagree about which of two candidates is better about one-quarter of the time.

variability is largely the product of ...

results aggregated in some way.

consensus must be reached—a procedure that creates its own problems,

presence of much occa...

First impressions turn out to matter—a lot.

we do learn something in the first seconds of interaction with a new acquaintance.

mostly on a candidate’s extraversion and verbal skills.

quality of a handshake

initial impressions have a deep effect on the way the interview proceeds.

we want the candidate sitting in front of us to make sense