Black Box Thinking: Why Some People Never Learn from Their Mistakes - But Some Do

by Matthew Syed

Mechanisms designed to learn from mistakes are impotent in many contexts if people won’t admit to them. It was only when the mindset of the organization changed that the system started to deliver amazing results.

Science has a structure that is self-correcting. By making testable predictions, scientists are able to see when their theories are going wrong, which, in turn, hands them the impetus to create new theories. But if scientists as a community ignored inconvenient evidence, or spun it, or covered it up, they would achieve nothing. Science is not just about a method, then; it is also about a mindset. At its best, it is driven forward by a restless spirit, an intellectual courage, a willingness to face up to failures and to be honest about key data, even when it undermines cherished beliefs. It is about method and mindset.

“The fundamental problem with the quality of American medicine is that we have failed to view the delivery of health care as a science. You find genes, you find therapies, but how you deliver them is up to you . . . That has been a disaster. It is why we have so many people being harmed.”

One particular problem in health care is not just the capacity to learn from mistakes, but also that even when mistakes are detected, the learning opportunities do not flow throughout the system. This is sometimes called the “adoption rate.” Aviation, as we have seen, has protocols that enable every airline, pilot, and regulator to access every new piece of information in almost real time. Data is universally accessible and rapidly absorbed around the world. The adoption rate is almost instantaneous.

an autopsy allows his colleagues to look inside a body and actually determine the precise cause of death. It is the medical equivalent of a black box. This has rather obvious implications for progress. After all, if the doctor turns out to be wrong in his diagnosis of the cause of death, he may also have been wrong in his choice of treatment in the days, perhaps months, leading up to death. That might enable him to reassess his reasoning, providing learning opportunities for him and his colleagues. It could save the lives of future patients.

In the civilian world around 80 percent of families give permission for autopsies to be performed when asked, largely because it provides them with answers as to why a loved one died.32 But despite this willingness, autopsies are hardly ever performed. Data in the United States indicate that less than 10 percent of deaths are followed by an autopsy.

doctors are reluctant to access the data: it hinges on the prevailing attitude toward failure. After all, why conduct an investigation if it might demonstrate that you made a mistake?

The underlying problem is not psychological or motivational. It is largely conceptual. And until we change the way we think about failure, the ambition of high performance will often remain a mirage, not just in health care but elsewhere, too.

hinged

“It is better that ten guilty persons escape than that one innocent suffer.”1 But miscarriages of justice have a quite different significance: they also represent precious learning opportunities.

acquitting

If you wanted to eliminate wrongful convictions altogether you could, say, increase the burden of proof required by the prosecution to 100 percent. But this outcome would come at a hefty price. It would mean that many more criminals would walk free.

when a doctor diagnoses a tumor that isn’t actually there. This is sometimes called a Type One error: an error of commission. The second kind is when a doctor fails to diagnose a tumor that is there. This is called a Type Two error: an error of omission. It is possible to reduce one kind of error while simultaneously increasing the other kind by altering the “evidence threshold,” as in the criminal justice system. But this trade-off should not obscure the fact that it is possible to reduce both kinds of error at the same time. That is what progress is ultimately about.

people don’t like to admit to failure. How are the police going to feel when they are told that all their hard work to find a brutal killer has served only to put an innocent man in jail? How will prosecutors, who often make the decisive difference in court, feel when all those efforts have ruined the life of an innocent man?

We cannot learn if we close our eyes to inconvenient truths,

DNA also has profound implications for cases that have already been tried: the power to exonerate. After all, if the DNA from the sperm in a rape victim has been stored, and if it does not match the DNA of the person serving time in prison, the conclusion is difficult to deny: it came from a different man, the real criminal. “DNA testing is to justice what the telescope is for the stars: not a lesson in biochemistry, not a display of the wonders of magnifying optical glass, but a way to see things as they really are,”

In situations where evidence had been stored, clients of the Innocence Project (a nonprofit group that helps prisoners protesting their innocence) were exonerated in almost half the cases. These exonerations raised dozens of questions. Why were police pursuing the wrong suspects? Why were eyewitnesses misidentifying criminals? Why were interrogation techniques used by the police leading to false conclusions? Why were the courts failing? And what could be done about it? There was a wider question, too: What about the system more generally? DNA is relevant in only a small number of cases (rapes, murders, etc., where human tissue had been found and stored). What about all the other cases, where convicted criminals had no recourse to DNA fingerprinting to establish their innocence? How many innocent people were behind bars in total?

“If we reviewed prison sentences with the same level of care that we devote to death sentences, there would have been over 28,500 non-death-row exonerations [in the United States] in the past 15 years rather than the 255 that have in fact occurred.”12

In 2005 the lawyers representing Juan Rivera applied for a DNA test. At the time, he had been in jail for almost thirteen years.

It showed that Rivera was not the source of the semen found inside the corpse of Holly Staker.

Rivera would spend another six years in jail. Why? Think back to the police. Were they going to accept their mistake? Were the prosecutors going to hold up their hands and admit they had gotten it wrong? Was the wider system going to accept what the DNA evidence was revealing about its defects?

When we are confronted with evidence that challenges our deeply held beliefs we are more likely to reframe the evidence than we are to alter our beliefs. We simply invent new reasons, new justifications, new explanations. Sometimes we ignore the evidence altogether.

“Cognitive dissonance” is the term Festinger coined to describe the inner tension we feel when, among other things, our beliefs are challenged by evidence. Most of us like to think of ourselves as rational and smart. We reckon we are pretty good at reaching sound judgments. We don’t like to think of ourselves as dupes.

In these circumstances we have two choices. The first is to accept that our original judgments may have been at fault. We question whether it was quite such a good idea to put our faith in a cult leader whose prophecies didn’t even materialize. We pause to reflect on whether the Iraq War was quite such a good idea given that Saddam didn’t pose the threat we imagined. The difficulty with this option is simple: it is threatening. It requires us to accept that we are not as smart as we like to think. It forces us to acknowledge that we can sometimes be wrong, even on issues on which we have staked a great deal. So, here’s the second option: denial. We reframe the evidence. We filter it, we spin it, or ignore it altogether. That way, we can carry on under the comforting assumption that we were right all along.

Think about it in terms of cognitive dissonance. If I have put up with a lot to become a member of a group, if I have voluntarily subjected myself to acute embarrassment, I would have to be pretty stupid if the group turned out to be anything less than wonderful. To protect my self-esteem I will want to convince myself that the group is pretty damn good. Hence the necessity to talk it up, to reframe my perceptions in a positive direction.

It is only when we have staked our ego that our mistakes of judgment become threatening.

cognitive dissonance is a deeply ingrained human trait. The more we have riding on our judgments, the more we are likely to manipulate any new evidence that calls them into question.

concocted

The reframing exercise often took a distinctive path. First the prosecutors would try to deny access to DNA evidence in the first place. When that strategy was batted away by judges, and the test had excluded the convict as the source of the DNA, they would claim that it had not been carried out correctly. This didn’t last long, either, because when the test was redone it would invariably come back with the same result. The next stage was for the prosecutor to argue that the semen belonged to a different man who was not the murderer. In other words, the victim had had consensual sex with another man, but had subsequently been raped by the prisoner, who had used a condom.22 This is the domino effect of cognitive dissonance: the reframing process takes on a life of its own. The presence of an entirely new man, not mentioned at the initial trial, for whom there were no eyewitnesses, and whom the victim often couldn’t remember having sex with, may seem like a desperate ploy to evade the evidence. But it has been used so often that it has been given a name by defense lawyers: “the unindicted co-ejaculator.”

Sometimes the system itself seems designed not to learn from mistakes but to bury them.

Defective systems create errors even when procedures are followed. Think of United Airlines 173, where the pilots followed procedure but the plane crashed. It was precisely because of the evidence provided by the crash that procedures were altered (the introduction of Crew Resource Management, for example). That is one of the key ways in which progress happens.

progress in most human activities depends, in large part, on our willingness to learn from failure. If we edit out failure, if we reframe our mistakes, we are effectively destroying one of the most precious learning opportunities that exists.

a subtle difference between external and internal deception. A deliberate deception (misleading one’s colleagues, or a patient, or a boss) has at least one clear benefit. The person doing the deceiving will, by definition, recognize the deceit and will inwardly acknowledge the failure. Perhaps he will amend the way he does his job to avoid such a failure in the future. Self-justification is more insidious. Lying to oneself destroys the very possibility of learning. How can one learn from failure if one has convinced oneself—through the endlessly subtle means of self-justification, narrative manipulation, and the wider psychological arsenal of dissonance-reduction—that a failure didn’t actually occur?

Psychologists often point out that self-justification is not entirely without benefits. It stops us from agonizing over every decision, questioning every judgment, staying awake at night wondering if getting married/taking that job/going on that course was the right thing to do. The problem, however, is when this morphs into mindless self-justification: when we spin automatically; when we reframe wantonly; when failure is so threatening we can no longer learn from it.

If nurses and doctors were fully aware of the fatal errors they were making, nondisclosure would add to their emotional anguish. They would know that they had harmed a patient,

the most effective cover-ups are perpetrated not by those who are covering their backs, but by those who don’t even realize that they have anything to hide.

Self-justification, the desire to protect one’s self-image, has the potential to afflict us all. The health care and criminal justice systems are but two strands in a wider story that represents a clear and present danger to our future progress.

classic response predicted by cognitive dissonance: we tend to become more entrenched in our beliefs

Twelve months later, when the Iraq survey group, Blair’s inspectors of choice, couldn’t find the weapons either, he changed tack again. Speaking to the House of Commons Liaison Committee, he said: “I have to accept we haven’t found them and we may never find them, we don’t know what has happened to them . . . They could have been removed, they could have been hidden, they could have been destroyed.” The evidential dance was now at full tilt. The lack of evidence for WMD in Iraq, according to Blair, was no longer because troops had not had enough time to find them, or because of the inadequacy of the inspectors: rather, it was because the Iraqi troops had spirited them out of existence. But this stance, within a few months, became untenable too. As the search continued in a state of near desperation, it became crystal clear that not only were there no WMD, but there were no remnants of them, either. Iraqi troops could not have spirited them away. So Blair parried again. In a set-piece speech at the Labour Party Conference, he finally accepted that Saddam did not have chemical or biological weapons, but argued that the decision to go to war was right anyway. “The problem is that I can apologize for the information that turned out to be wrong, but I can’t, sincerely at least, apologize for removing Saddam,” he said. “The world is a better place with Saddam in prison.”

If it is intolerable to change your mind, if no conceivable evidence will permit you to admit your mistake, if the threat to ego is so severe that the reframing process has taken on a life of its own, you are effectively in a closed loop. If there are lessons to be learned, it has become impossible to acknowledge them, let alone engage with them.

we often suppose that bright people are the most likely to reach the soundest judgments. We associate intelligence, however defined, as the best way of reaching truth. In reality, however, intelligence is often deployed in the service of dissonance-reduction. Indeed, sometimes the most prestigious thinkers are the most adept at deploying the techniques of reframing,

Dissonance is not just about Tony Blair, or doctors, or lawyers, or members of religious cults, it is also about world-famous business leaders, historians, and economists. Ultimately, it concerns how our culture’s stigmatizing attitude toward error undermines our capacity to see evidence in a clear-eyed way.

the intellectual energy of some of the world’s most formidable thinkers is directed, not at creating new, richer, more explanatory theories, but at coming up with ever-more-tortuous rationalizations as to why they were right all along.

And these are also the kinds of people (or institutions) who often have the capacity to employ expensive PR firms to bolster their post hoc justifications. They have the financial means, in addition to a powerful subconscious urge, to bridge the gap between beliefs and evidence, not by learning, but by spinning.

the learning advantage of adapting to a mistake is outweighed by the reputational disadvantage of admitting to it.

A rational person should keep those shares most likely to appreciate in the future while selling those likely to depreciate. Indeed, this is what you must do if you are attempting to maximize your financial return. The stock market rewards those who buy low and sell high. But we are actually more likely to keep the shares that have lost money, regardless of their future prospects. Why? Because we hate to crystallize a loss. The moment a losing stock is sold, a paper loss becomes a real loss.

confirmation bias in action, and it is eerily reminiscent of early medicine (where doctors interpreted any outcome in their patients as an affirmation of bloodletting).

Intelligence and seniority when allied to cognitive dissonance and ego is one of the most formidable barriers to progress in the world today.

in 1958, peasants were ordered to plant 6.5 million per 2.5 acres.” Too late, it was discovered that the seeds did indeed compete with each other, stunting growth and damaging yields. It contributed to one of the worst disasters in Chinese history, a tragedy that even now has not been fully revealed. Historians estimate that between 20 and 43 million people died during one of the most devastating famines in human history.

even practicing scientists are suckers for the seemingly inviolable power of memory. When we remember seeing something, it feels as if we are accessing a videotape of a real, tangible, rock-solid event.