The Field Guide to Understanding 'Human Error'

by Sidney Dekker

Practitioners are not all exposed to the same kind and level of accident risk. This makes it impossible to compare their accident rates and say that some, because of personal characteristics, are more accident-prone than others.

There is no evidence that a system approach dilutes personal accountability. In fact, second victims show just how much responsibility practitioners take for things that go wrong.

If you hold somebody accountable, that does not have to mean exposing that person to liability or punishment. • You can hold people accountable by letting them tell their story, literally “giving their account.” • Storytelling is a powerful mechanism for others to learn vicariously from trouble.

Don’t ask who is responsible, ask what is responsible.

Link knowledge of the messy details with the creation of justice.

Explore the potential for restorative justice.

Go from backward to forward-looking accountability.

Put second victim support in place.

To understand failure, you first have to understand your own reactions to failure. Reactions to failure are typically: • Retrospective. They arise from your ability to look back on a sequence of events. • Counterfactual. They lay out what people could or should have done to avoid the outcome that you now know about. • Judgmental. They judge people for not doing what you believe they should have done, or for not paying enough attention to what you now know is important. • Proximal. They focus on those people closest in time and place to (preventing) the mishap.

If you look, for a moment, at the psychological research underlying all this, there are actually two ways in which your understanding of a past situation gets influenced: • The hindsight bias. Finding out about an outcome increases the estimate we make about its likelihood. In other words, as a retrospective reviewer who knows the outcome of an event, you exaggerate your own ability to predict and prevent the outcome—while not even being aware of that bias.2 • The outcome bias. Once you know the outcome, it changes your evaluation of decisions that led up to it. If the outcome is bad, then you are not only more willing to judge the decisions, but also more likely to judge them more harshly.3

How can you avoid hindsight? Figure 2.2 shows two different perspectives on a pathway to failure: • The perspective from the outside and hindsight (typically your perspective). From here you can oversee the entire sequence of events—the triggering conditions, its various twists and turns, the outcome, and the true nature of circumstances surrounding the route to trouble. • The perspective from the inside of the tunnel. This is the point of view of people in the unfolding situation. To them, the outcome was not known (or they would have done something else). They contributed to the sequence of events because of what they saw on the inside of the unfolding situation. To understand ‘human error,’ you have to take this perspective.

In order to understand error, you have to examine the larger system in which these people worked. You can divide an operational system into a sharp end and a blunt end: • At the sharp end (for example the train cab, the cockpit, the surgical operating table), people are in direct contact with the safety-critical process. • The blunt end is the organization or set of organizations that both supports and constrains activities at the sharp end (for example, the airline or hospital; equipment vendors and regulators).

Out of context I: Micro-matching One of the most popular ways you can assess performance after the fact is to hold it up against a world you now know to be true. This can be: • A procedure or collection of rules: People’s behavior was not in accordance with standard operating procedures that were found to be applicable for that situation afterward. • A set of cues: People missed cues or data that turned out to be critical for understanding the true nature of the situation. • Standards of good practice that people’s behavior falls short of. The problem is that these after-the-fact-worlds may have very little in common with the actual world that produced the behavior. They contrast people’s behavior against your reality, not the reality that surrounded the behavior in question. Thus, micro-matching fragments of behavior with these various standards explains nothing—it only judges.

Out of context II: Cherry-picking The second broad way in which you can take data out of context, in which you give them meaning from the outside, is by grouping and labeling behavior fragments that, in hindsight, appear to represent a common condition.

Out of context III: The shopping bag With the benefit of hindsight, it is easy to sweep together all the evidence that people should have seen. If they had, they would have recognized the situation for what we now know it turned out to be. But that doesn’t mean the evidence presented itself that way to people at the time.

There is no “root” cause So what is the cause of the accident? This question is just as bizarre as asking what the cause is of not having an accident. There is no single cause—neither for failure, nor for success. In order to push a well-defended system over the edge (or make it work safely), a large number of contributory factors are necessary and only jointly sufficient. How is it that a mishap gives you so many causes to choose from? Part of the story is the sheer complexity of the systems that we have put together, and that we have protected our systems so well against failure. A lot needs to go wrong for an incident or accident to occur. So you can really construct “causes” from everywhere. The causal web quickly multiplies and fans out, like cracks in a window. What you call “root cause” is simply the place where you stop looking any further. That means that any policy or standard that gets you to identify root causes, or probable causes, is of necessity: Cause is not something you find. Cause is something you construct. • Selective. There are only so many things you can label “causal” before the word “causal” becomes meaningless. • Exclusive. They leave out factors that were also necessary and only jointly sufficient to “cause” the failure. • Oversimplified. They highlight only a few hotspots in a long, twisted and highly interconnected, much finer-grained web that spreads far outside the causes you come up with. If you find a root or primary or...

Systems models focus on the whole, not the parts (like the accident models above do). The interesting properties of systems (the ones that give rise to system accidents) can only be studied and understood when you treat them in their entirety. System models build on two fundamental ideas:9 • Emergence: Safety is an emergent property that arises when system components and processes interact with each other and their environment. Safety can be determined only by seeing how parts or processes work together in a larger system; • Control imposes constraints on the degrees of freedom (for example, through procedures, design requirements) of components, so as to control their interaction. Such control is based on prior (and possibly false) ideas about how components and processes interact. Control can be imperfect and even erode over time. System accidents result not from component failures, but from an erosion of control of safety-related constraints on the development, design and operation of the system. People and organizations may have thought that their control was adequate—based on their knowledge, goals and focus of attention at the time. But systems are not static designs; they are dynamic processes, continually adapting to achieve goals in a changing environment. What was adequate control before may now have eroded constraints and pushed the system closer to the edge, operating with smaller margins. Such erosion typically happens because these systems are t...

To explain failure, system models do not need a component to break or a human to err. In fact, they do not have to rely on anything “going wrong,” or anything being out of the ordinary. Even though an accident happened, nothing really might have gone wrong—in the sense that nothing that happened was out of the ordinary. Accidents, in other words, are typically the by-product of the normal functioning of the system, not the result of something breaking down or failing inside of that system. In that sense: • There is not much difference (if at all) between studying a successful or a failed system, since the organizational processes that lead to both are very similar. • System models, in other words, are hardly “accident models.” They are models of a system’s or organization’s normal functioning. It is this normal functioning that contains the seeds both of success and failure.

System models do not rely on linear cause–effect relationships to explain how factors interact or relate to one another. This means that they can stay closer to the complexity and goal conflicts behind system success and failure. It also means that they, as models, are more complex.

The management of goal conflicts under uncertainty gets pushed down into local operating units—control rooms, cockpits, operating theaters and the like. There the conflicts are to be negotiated and resolved in the form of thousands of little and larger daily decisions and trade-offs. These are no longer decisions and trade-offs made by the organization, but by individual operators or crews. What they accept as risky or normal will shift over time: • as a result of pressures and expectations put on them by the organization; • and as a result of continued success, even under those pressures. This takes us to another family of models about risk and accidents: Drift into failure.

This gave birth in the 1970s to “man-made disaster theory.” It was really the first to see accidents as the result of a drift into failure, and to focus on the organizational blunt end to explain that drift.11 This theory was a call to understand accidents not as sudden phenomena where energy was not contained, but as phenomena over time, where people and whole organizations subtly changed their idea of what was risky in the first place.

A safety culture is a culture that allows the boss to hear bad news.

Resilience and Safety I versus Safety II If the major risk is success—that is, things going right—instead of failures, then why do we spend the majority of our safety resources on investigating what goes wrong? This is exactly the question Erik Hollnagel raises. Managing safety on the basis of incidents is only one way—and in a sense a very limited way. It focuses, after all, on the few occasional times when things go (almost) wrong, rather than on the many times that things go right. It is a reactive, lagging kind of safety management that might turn into firefighting instead of a proactive, coordinated improvement of overall organizational performance. This is why Erik Hollnagel has made the distinction between:16 • Safety I. Safety is the absence of negative events. A system is safe if there are no incidents or accidents. The purpose of safety management is to ensure that as little as possible goes wrong. The focus is on negative events and reducing their severity and number. This often translates into trying to reduce the variability and diversity of people’s behavior—to constrain them and get them to adhere to standards. • Safety II. Safety is the presence of positive capacities, capabilities and competencies that make things go right. This is resilience: the ability of a system to adjust its functioning before, during or after changes and disturbances, so that it can sustain required operations under both expected and unexpected conditions. This needs t...

Is safety making sure those few things don’t go wrong, or that as many things as possible go right?

Safety has increasingly morphed from operational value into bureaucratic accountability. Those concerned with safety are more and more removed—organizationally, culturally, psychologically—from those who do safety-critical work at the sharp end.

Safety as Responsibility Down, Not Accountability Up

Life at the thin edge of the wedge Many industries have become considerably safer over the past century. For sure, there are activities that lie at or near the unsafe system definition (certain forms of surgery, for example, or recreational activities such as base jumping). But many organizations are now ultra-safe or near-zero. They live at the thin edge of the wedge. Creating even more progress on safety there is quite difficult. It is easy to see, however, that doing more of the same does not lead to something different: it will maintain the status quo. In other words, emphasizing standardization and compliance, and investing in incident reporting, will help the organization maintain its safety level. It will do little, however, to improve safety even further. Let’s focus on ultra-safe or near-zero organizations for the moment. This is important because: • Safety improvements over the last decades, including technologies, regulation, training, reporting systems, social awareness and more, have brought many industries into or toward the ultra-safe band. Parts of your industry may be close to, or in it, as well. • Making an ultra-safe or near-zero organization even safer is one of the most difficult challenges. There is little theory to guide you. Safety progress in such organizations has gone asymptotic: that is, the curve of progress is plateauing. • At the same time, ultra-safe or near-zero organizations still suffer surprisingly big accidents from...

Decoy phenomena What can offer some insight into managing risk at the thin edge of the wedge? One is “decoy phenomena.” Barry Turner identified these in the 1970s in his studies of industrial disasters and accidents (he was, as you might recall, the founder of man-made disaster theory). Decoy phenomena are apparent safety issues that take up a lot of the organization’s attention. They may have been singled out as a safety priority. Your organization may have labeled them as one of the “top five” or “top three” risk priorities that needs to be controlled in order to prevent an accident. Such decoy phenomena are strong in pulling both attention and organizational resources away from other possible sources of risk. These may as yet be unknown. Or they may be known but not believed to be very dangerous. Decoy phenomena are typical for organizations at the thin edge of the wedge. Pursuing what they see as the last frontiers in improving their safety performance, they may devote so much time to one or a few issues that they lose sight of the rest.

At the thin edge of the wedge, holes in layers of defense and formally reported incidents are no longer the herald of accidents or fatalities. Normal work is.

Hard fixes change something fundamental about, or in, the organization. This makes them hard. But it also makes them real fixes.

Your organization and ‘human error’ • Your organization is not basically or inherently safe. People have to create safety by putting tools and technologies to use while negotiating multiple system goals at all levels of your organization. • The priorities and preferences that people express through their practice may be a logical reproduction of what the entire organization finds important. • Human error is the inevitable by-product of the pursuit of success in an imperfect, unstable, resource-constrained world. The occasional human contribution to failure occurs because complex systems need an overwhelming human contribution for their safety. • So ‘human error’ is never at the root of your safety problems. ‘Human error’ is the effect of trouble deeper inside your system. • It also means that ‘human error’ is not random. ‘Human error’ is systematically connected to features of people’s tools, tasks and operating environment.

What to think of when investigating ‘human error’ • As far as the people involved were concerned, the outcome was not going to happen. If they knew it was going to, they would have done something else. • Nobody comes to work to do a bad job. This is the local rationality principle. People do what makes sense to them at the time given their focus of attention, their knowledge and their goals (which may well be the organization’s goals, stated or unstated). • Human error is not the cause of failure, but the effect. So ‘human error,’ under whatever label (“loss of situation awareness,” complacency, inadequate CRM) can never be the conclusion of your investigation. It is the starting point. • Explaining one error (for example, operator error) by pointing to another (inadequate supervision, deficient management, bad design) does not explain anything. It only judges other people for not doing what you, in hindsight, think they should have done. • To understand what went on in somebody’s mind, you have to reconstruct the situation in which that mind found itself. • There is no such thing as the cause of a mishap. This is like looking for the cause of not having a mishap. What you deem causal depends on your accident model.

Reprimanding “Bad Apples” is like peeing in your pants. You did something about the problem and feel relieved. But then it gets cold and u...

Creating progress on safety with the new view • To create safety, you don’t need to rid your system of ‘human errors’. Instead, you need to realize how people at all levels in the organization contribute to the creation of safety and risk through goal trade-offs that are legitimate and desirable in their setting. • Rather than trying to reduce “violations,” New View strategies will find out more about the gap between work-as-imagined and work-as-done—why it exists, what keeps it in place and how it relates to priorities among organizational goals (both stated and unstated). • New View thinking wants to learn about authority–responsibility mismatches—places where you expect responsibility of your people, but where their situation is not giving them requisite authority to live up to that responsibility. • You know your organization is maturing toward the New View once it actively tries to learn how it is learning about safety. This means your organization is calibrating whether its strategies for managing safety and risk are up-to-date. • Every organization has room to improve its safety. What separates a strong safety culture from a weak one is not how large this room is. What matters is the organization’s willingness to explore this space, to find leverage points to learn and improve.

Speaking for the dead speaks as loudly for them as it does for everyone who could have been in their shoes. It speaks for the past as much as for the future. Of course, we might throw up our hands and admit defeat; say that we cannot crawl into the skull of a dead man; concede that time is irreversible and that reconstructing somebody’s mindset is impossible. But that does not mean that there is nothing systematic about what people do in interaction with complex systems, about why they do what they do. It does not mean that everything about their mind and the world in which its understanding unfolded is unintelligible, inexplicable. It does not mean that there is nothing predictable about how things go right or wrong. And it does not release us from the ethical responsibility to at least try. If all there is in the past is dead people, whose intentions and machinations are forever closed off to our analysis and attempts at understanding, then we have little to say about the future too.