More on this book
Community
Kindle Notes & Highlights
Judgment can therefore be described as measurement in which the instrument is a human mind.
System noise is inconsistency, and inconsistency damages the credibility of the system.
As any defense lawyer will tell you, judges have reputations, some for being harsh “hanging judges,” who are more severe than the average judge, and others for being “bleeding-heart judges,” who are more lenient than the average judge. We refer to these deviations as level errors.
We use the term level noise for the variability of the judges’ average judgments, which is identical to the variability of level errors.
Looking across a row, you will find that judges are not equally severe in their sentencing of all cases: they are harsher than their personal average in some and more lenient in others. We call these residual deviations pattern errors.
We use the term pattern noise for the variability we just identified, because that variability reflects a complex pattern in the attitudes of judges to particular cases.
If a judge is in a good mood because something nice happened to her daughter, or because a favorite sports team won yesterday, or because it is a beautiful day, her judgment might be more lenient than it would otherwise be. This within-person variability is conceptually distinct from the stable between-person differences that we have just discussed—but it is difficult to tell these sources of variability apart. Our name for the variability that is due to transient effects is occasion noise.
To summarize, we discussed several types of noise. System noise is undesirable variability in the judgments of the same case by multiple individuals. We have identified its two major components, which can be separated when the same individuals evaluate multiple cases: Level noise is variability in the average level of judgments by different judges. Pattern noise is variability in judges’ responses to particular cases.
A person might be approved for a loan if the previous two applications were denied, but the same person might have been rejected if the previous two applications had been granted. This behavior reflects a cognitive bias known as the gambler’s fallacy: we tend to underestimate the likelihood that streaks will occur by chance.
The illusion of validity is found wherever predictive judgments are made, because of a common failure to distinguish between two stages of the prediction task: evaluating cases on the evidence available and predicting actual outcomes.
For every one of the ninety-eight participants, the model did better than the participant did! Decades later, a review of fifty years of research concluded that models of judges consistently outperformed the judges they modeled.
To summarize this short tour of mechanical decision making, we review two reasons for the superiority of rules of all kinds over human judgment. First, as described in chapter 9, all mechanical prediction techniques, not just the most recent and more sophisticated ones, represent significant improvements on human judgment. The combination of personal patterns and occasion noise weighs so heavily on the quality of human judgment that simplicity and noiselessness are sizable advantages. Simple rules that are merely sensible typically do better than human judgment. Second, the data is sometimes
...more
One review of intuition in managerial decision making defines it as “a judgment for a given course of action that comes to mind with an aura or conviction of rightness or plausibility, but without clearly articulated reasons or justifications—essentially ‘knowing’ but without knowing why.”
They also believe in the predictability of events that are in fact unpredictable, implicitly denying the reality of uncertainty. In the terms we have used here, this attitude amounts to a denial of ignorance.
where objective ignorance is severe, we should, after a while, become aware of the futility of crystal balls in human affairs. But that is not our usual experience of the world. Instead, as the previous chapter suggested, we maintain an unchastened willingness to make bold predictions about the future from little useful information.
The confidence we have in these experts’ judgment is entirely based on the respect they enjoy from their peers. We call them respect-experts.
judgment is a form of measurement in which the instrument is a human mind.
Some judgments are predictive, and some predictive judgments are verifiable; we will eventually know whether they were accurate.
But many judgments, including long-term forecasts and answers to fictitious questions, are unverifiable. The quality of such judgments can be assessed only by the quality of the thought process that produces them. Furthermore, many judgments are not predictive but evaluative: the sentence set by a judge or the rank of a painting in a prize competition cannot easily be compared to an objective true value.
We say that bias exists when most errors in a set of judgments are in the same direction. Bias is the average error,
Eliminating bias from a set of judgments will not eliminate all error. The errors that remain when bias is removed are not shared. They are the unwanted divergence of judgments, the unreliability of the measuring instrument we apply to reality. They are noise. Noise is variability in judgments that should be identical. We use the term system noise for the noise observed in organizations that employ interchangeable professionals to make decisions, such as physicians in an emergency room, judges imposing criminal penalties, and underwriters in an insurance company. Much of this book has been
...more
The mean of squared errors (MSE) has been the standard of accuracy in scientific measurement for two hundred years. The main features of MSE are that it yields the sample mean as an unbiased estimate of the population mean, treats positive and negative errors equally, and disproportionately penalizes large errors.
The large role of noise in error contradicts a commonly held belief that random errors do not matter, because they “cancel out.” This belief is wrong. If multiple shots are scattered around the target, it is unhelpful to say that, on average, they hit the bull’s-eye. If one candidate for a job gets a higher rating than she deserves and another gets a lower one, the wrong person may be hired. If one insurance policy is overpriced and another is underpriced, both errors are costly to the insurance company; one makes it lose business, the other makes it lose money.
System noise can be broken down into level noise and pattern noise. Some judges are generally more severe than others, and others are more lenient; some forecasters are generally bullish and others bearish about market prospects; some doctors prescribe more antibiotics than others do. Level noise is the variability of the average judgments made by different individuals. The ambiguity of judgment scales is one of the sources of level noise. Words such as likely or numbers (e.g., “4 on a scale of 0 to 6”) mean different things to different people. Level noise is an important source of error in
...more
Pattern noise also has a transient component, called occasion noise. We detect this kind of noise if a radiologist assigns different diagnoses to the same image on different days or if a fingerprint examiner identifies two prints as a match on one occasion but not on another.
The judges’ cognitive flaws are not the only cause of errors in predictive judgments. Objective ignorance often plays a larger role. Some facts are actually unknowable—how many grandchildren a baby born yesterday will have seventy years from now, or the number of a winning lottery ticket in a drawing to be held next year. Others are perhaps knowable but are not known to the judge. People’s exaggerated confidence in their predictive judgment underestimates their objective ignorance as well as their biases.
Psychological biases are, of course, a source of systematic error, or statistical bias. Less obviously, they are also a source of noise. When biases are not shared by all judges, when they are present to different degrees, and when their effects depend on extraneous circumstances, psychological biases produce noise.
Bias has a kind of explanatory charisma, which noise lacks. If we try to explain, in hindsight, why a particular decision was wrong, we will easily find bias and never find noise. Only a statistical view of the world enables us to see noise, but that view does not come naturally—we prefer causal stories. The absence of statistical thinking from our intuitions is one reason that noise receives so much less attention than bias does.
There is reason to believe that some people make better judgments than others do. Task-specific skill, intelligence, and a certain cognitive style—best described as being actively open-minded—characterize the best judges.
Our main suggestion for reducing noise in judgment is decision hygiene. We chose this term because noise reduction, like health hygiene, is prevention against an unidentified enemy.
The goal of judgment is accuracy, not individual expression.
A radical application of this principle is the replacement of judgment with rules or algorithms. Algorithmic evaluation is guaranteed to eliminate noise—indeed, it is the only approach that can eliminate noise completely.
Think statistically, and take the outside view of the case.
Structure judgments into several independent tasks.
This divide-and-conquer principle is made necessary by the psychological mechanism we have described as excessive coherence, which causes people to distort or ignore information that does not fit a preexisting or emerging story.
Resist premature intuitions.
sequence the information: professionals who make judgments should not be given information that they don’t need and that could bias them, even if that information is accurate.
Obtain independent judgments from multiple judges, then consider aggregating those judgments.
Favor relative judgments and relative scales.
After a successful operation, you like to believe that it is the surgeon’s skill that saved your life—and it did, of course—but if the surgeon and all the personnel in the operating room had not washed their hands, you might be dead. There may not be much glory to be gained in hygiene, but there are results.
