More on this book
Community
Kindle Notes & Highlights
Every day, the news media deliver forecasts without reporting, or even asking, how good the forecasters who made the forecasts really are.
Even fans expect to see player stats on scoreboards and TV screens. And yet when it comes to the forecasters who help us make decisions that matter far more than any baseball game, we’re content to be ignorant.
it turns out that forecasting is not a “you have it or you don’t” talent. It is a skill that can be cultivated.
the average expert was roughly as accurate as a dart-throwing chimpanzee.
A researcher gathered a big group of experts—academics, pundits, and the like—to make thousands of predictions about the economy, stocks, elections, wars, and other issues of the day.
the average expert did about as well as random guessing.
I am that researcher and for a while I didn’t mind the joke. My study was the most comprehensive assessment of expert judgment in the scientific literature. It was a long slog that took about twenty years, from 1984 to 2004,
Old forecasts are like old news—soon forgotten—and pundits are almost never asked to reconcile what they said with what actually happened. The one undeniable talent that talking heads have is their skill at telling a compelling story with conviction, and that is enough.
as word of my work spread, its apparent meaning was mutating.
The message became “all expert forecasts are useless,” which is nonsense.
My research had become a backstop reference for nihilists who see the future as inherently unpredictable and know-nothing populists who insist on preceding “expert” with “so-called.”
There is plenty of room to stake out reasonable positions between the debunkers and the defenders of experts and their forecasts.
This was the Arab Spring—and it started with one poor man, no different from countless others, being harassed by police, as so many have been, before and since, with no apparent ripple effects.
A decade earlier, Lorenz had discovered by accident that tiny data entry variations in computer simulations of weather patterns—like replacing 0.506127 with 0.506—could produce dramatically different long-term forecasts.
Edward Lorenz shifted scientific opinion toward the view that there are hard limits on predictability, a deeply philosophical question.4 For centuries, scientists had supposed that growing knowledge must lead to greater predictability because reality was like a clock—
Laplace called his imaginary entity a “demon.” If it knew everything about the present, Laplace thought, it could predict everything about the future. It would be omniscient.
Lorenz poured cold rainwater on that dream. If the clock symbolizes perfect Laplacean predictability, its opposite is the Lorenzian cloud.
In one of history’s great ironies, scientists today know vastly more than their colleagues a century ago, and possess vastly more data-crunching power, but they are much less confident in the prospects for perfect predictability.
We make mundane predictions like these routinely, while others just as routinely make predictions that shape our lives.
Unpredictability and predictability coexist uneasily in the intricately interlocking systems that make up our bodies, our societies, and the cosmos. How predictable something is depends on what we are trying to predict, how far into the future, and under what circumstances.
In so many other high-stakes endeavors, forecasters are groping in the dark. They have no idea how good their forecasts are in the short, medium, or long term—and no idea how good their forecasts could become.
The consumers of forecasting—governments, business, and the public—don’t demand evidence of accuracy. So there is no measurement. Which means no revision. And without revision, there can be no improvement.
Barbara Mellers and I launched the Good Judgment Project and invited volunteers to sign up and forecast the future.
How good is all this forecasting? That is not easily answered because the intelligence community, like so many major producers of forecasting, has never been keen on spending money to figure that out.
IARPA created a forecasting tournament in which five scientific teams led by top researchers in the field would compete to generate accurate forecasts on the sorts of tough questions intelligence analysts deal with every day. The Good Judgment Project was one of those five teams.
After two years, GJP was doing so much better than its academic competitors that IARPA dropped the other teams.
Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs.
And never forget that even modest improvements in foresight maintained over time add up.
The kind of thinking that produces superior judgment does not come effortlessly. Only the determined can deliver it reasonably consistently, which is why our analyses have consistently found commitment to self-improvement to be the strongest predictor of performance.
in most cases statistical algorithms beat subjective judgment, and in the handful of studies where they don’t, they usually tie. Given that algorithms are quick and cheap, unlike subjective judgment, a tie supports using the algorithm. The point is now indisputable: when you have a well-validated statistical algorithm, use it.
Even with computers making galloping advances, the sort of forecasting that superforecasters do is a long way off.
we will also see more and more syntheses, like “freestyle chess,” in which humans with computers compete as teams, the human drawing on the computer’s indisputable strengths but also occasionally overriding the computer.
To reframe the man-versus-machine dichotomy, combinations of Garry Kasparov and Deep Blue may prove more robust than pure-human or pure-machine approaches.
We have all been too quick to make up our minds and too slow to change them. And if we don’t examine how we make these mistakes, we will keep making them. This stagnation can go on for years. Or a lifetime. It can even last centuries, as the long and wretched history of medicine illustrates.
The standard histories are usually mute on these scores, but when we use modern science to judge the efficacy of historical treatments, it becomes depressingly clear that most of the interventions were useless or worse.
It takes strong evidence and more rigorous experimentation than the “bleed the patient and see if he gets better” variety to overwhelm preconceptions. And that was never done.
“All who drink of this treatment recover in a short time, except those whom it does not help, who all die,” he wrote. “It is obvious, therefore, that it fails only in incurable cases.”
Not until the twentieth century did the idea of randomized trial experiments, careful measurement, and statistical power take hold.
It was the absence of doubt—and scientific rigor—that made medicine unscientific and caused it to stagnate for so long.
Physicians and the institutions they controlled didn’t want to let go of the idea that their judgment alone revealed the truth, so they kept doing what they did because they had always done it that way—and they were backed up by respected authority.
What people didn’t grasp is that the only alternative to a controlled experiment that delivers real insight is an uncontrolled experiment that produces merely the illusion of insight.
introspection can only capture a tiny fraction of the complex processes whirling inside your head—and behind your decisions.
System 2 is the familiar realm of conscious thought. It consists of everything we choose to focus on. By contrast, System 1 is largely a stranger to us. It is the realm of automatic perceptual and cognitive operations—
an ingenious psychological measure, the Cognitive Reflection Test, which has shown that most people—including very smart people—aren’t very reflective.
That is normal human behavior. We tend to go with strong hunches. System 1 follows a primitive psycho-logic: if it feels true, it is.
As Daniel Kahneman puts it, “System 1 is designed to jump to conclusions from little evidence.”
A defining feature of intuitive judgment is its insensitivity to the quality of the evidence on which the judgment is based.
These tacit assumptions are so vital to System 1 that Kahneman gave them an ungainly but oddly memorable label: WYSIATI (What You See Is All There Is).
The human brain demands order. The world must make sense, which means we must be able to explain what we see and think. And we usually can—because we are creative confabulators hardwired to invent stories that impose coherence on the world.
These people are remarkably normal, but their condition allows researchers to communicate directly with only one hemisphere of their brain—by showing an image to only the left or right field of vision—without sharing the communication with the other hemisphere.
