Noise: A Flaw in Human Judgment

by Daniel Kahneman

The challenge is that when the formula is applied out of sample—that is, when it is used to predict outcomes in a different data set—the weights will no longer be optimal. Flukes in the original sample are no longer present, precisely because they were flukes; in the new sample, managers with high technical skills are not all superstars. And the new sample has different flukes, which the formula cannot predict. The correct measure of a model’s predictive accuracy is its performance in a new sample, called its cross-validated correlation. In effect, a regression model is too successful in the original sample, and a cross-validated correlation is almost always lower than it was in the original data. Dawes and Corrigan compared equal-weight models to multiple regression models (cross-validated) in several situations. One of their examples involved predictions of the first-year GPA of ninety graduate students in psychology at the University of Illinois, using ten variables related to academic success: aptitude test scores, college grades, various peer ratings (e.g., extroversion), and various self-ratings (e.g., conscientiousness). The standard multiple regression model achieved a correlation of .69, which shrank to .57 (PC = 69%) in cross-validation. The correlation of the equal-weight model with first-year GPA was about the same: .60 (PC = 70%). Similar results have been obtained in many other studies.