In principle at least, level noise—or simple, across-the-board differences between judges—should be a relatively easy problem to measure and address. If there are abnormally “tough” graders, “cautious” child custody officers, or “risk-averse” loan officers, the organizations that employ them could aim to equalize the average level of their judgments. Universities, for instance, address this problem when they require professors to abide by a predetermined distribution of grades within each class. Unfortunately, as we now realize, focusing on level noise misses a large part of what individual
...more
