Noise: A Flaw in Human Judgment

by Daniel Kahneman

The second problem is that the forced distribution of the ratings is assumed to reflect the distribution of the underlying true performances—typically, something close to a normal distribution. Yet even if the distribution of performances in the population being rated is known, the same distribution may not be reproduced in a smaller group, such as those assessed by a single evaluator. If you randomly pick ten people from a population of several thousand, there is no guarantee that exactly two of them will belong to the top 20% of the general population. (“No guarantee” is an understatement: the probability that this will be the case is just 30%.) In practice, the problem is even worse, because the composition of teams is not random. Some units may be staffed almost entirely with high performers, and others with subpar employees.