Sources of Power: How People Make Decisions

by Gary Klein

Here, the difficulty is created by the lack of information rather than the closeness of the two outcomes. We can think of this as a zone of ignorance:

I think it's still essentially the same thing because with the given, limited information, the options are still close to each other. Saying they are not can only be said outside of the said sitution with more context and information than the agents in the situation have. This perspective can easily develop when looking at a situation in hindsight. I think this tendency to ignore the agent's perspective of limited information and time is a common but a dangerous one. Discussed quiye a bit in Dekker's second book that I read but whose name I forget.

Because the key to effective decision making is to build up expertise, one temptation is to develop training to teach people to think like experts. But in most settings, this can be too time-consuming and expensive. However, if we cannot teach people to think like experts, perhaps we can teach them to learn like experts. After reviewing the literature, I identified a number of ways that experts in different fields learn Klein (1997): They engage in deliberate practice, so that each opportunity for practice has a goal and evaluation criteria. They compile an extensive experience bank. They obtain feedback that is accurate, diagnostic, and reasonably timely. They enrich their experiences by reviewing prior experiences to derive new insights and lessons from mistakes.

Important as a whole but the last note about comparisons is very interesting in itself as well. Many points are about getting more out of every experience and so is the last point.

In domains that are not marked by opportunities for effective feedback (e.g., clinical psychology), mere accumulation of experience does not appear to result in growth of decision expertise.

This suggests that an important challenge in software design and leadership are the long feedback geedback loops. A simple example is writing tests in code. We know the savings on maintenance costs make it worthwhile but it can be very difficult for an individual to develop that understanding in isolation. Luckily feedback can also come from experienced peers rather than the work itself which can address this problem in effective teams. This shows in how some teams can stay in the "not sure if tests are worth it" stage for essentially forever while people joining effective orgs learn othe practices very quickly. If feedback comes from a peer instead of from the work itself then that requires that the recipient respects and believes the feedback giver. Otherwise it may not have that effect. Ties into the general understanding of how culture (as shared knowledge built up ober time) affects us. How physically we may not be very different from our ancestors but we know hell of a lot more.

During a tournament, a grand master will be working against the clock and will not be reflecting on the implications of the game. But afterward, there is time to go over the game record to look for opportunities that were missed, early signals that were not noticed, or assessments and assumptions that were incorrect. In this way, an experience (even a single game) can be recycled and reused. In many field settings where there are limited opportunities for experience, developing the discipline of reviewing the decision-making processes for each incident can be valuable.

This highlights that the value of an incident postmortem or review practice is not just to share lessons but also to actually learn from the incident yourself. The incentive to explain what happened and why forces you to investigate and develop an understanding. So the different goals of a postmortem process are: forcing investigation to ensure available lessons are learned; share those lessons to spread them across the org; communicate internally and externally to mainatain and build trust. Distinguishing between these goals allows developing different means to address each.

Reviews generally occur after mistakes, and not as a routine. And when they do occur, they seem to be directed at what happened, whereas the cognitive critiques are aimed at the thought processes of the key decision makers. Moreover, reviews are conducted in a forum that is usually not conducive to reflection.

Story based; in a safe environment; focused on understanding each perspective and not criticising or offering counter factuals; running regularly as chaos exercises with an integral review process instead of waiting for things to happen. All important aspects packed into this bit.

in most settings there are not always enough real incidents to build expertise quickly enough. That is why it is important to make effective use the incidents that occur.

Importance of learning from incidents and sharing the lessons.

A sixth application is to use decision requirements for designing software systems. For a given task, the decision requirements are the key decisions and how they are made.

.c1

When design engineers are given a new project, they are usually told what the system is supposed to accomplish, that is, what it will do if it works the way it is supposed to. Yet they are rarely told what the key decisions are that the system must help the operator make or the types of strategies or rules of thumb that the operator is likely to use. Left without any way to visualize the operator, designers do the best job they can to pack information onto screens so that it will all be there when needed. All the relevant information was available to the crew during the Vincennes shootdown, but the format of the information was difficult to interpret. The crucial altitude information was presented as a static four-digit number on a separate monitor.

.c2 The static number was problematic because what was important in the moment was information about how the number is changing over time not so much what the number is. This emphasizes use case and user focused design and strong product management work. Not just a dashboard with all the data the system has.

Skillful problem solving is impresive because after the fact, the solution seems obvious, yet we know that, without any guidance, most people would miss the answer. They would not even know that an answer was possible.

For example, for miniature visual displays, color would be better than black and white. However, color displays have lower resolution than black and white because it takes a mixture of three primary colors to produce a single pixel. Dave Post, a specialist in visual perception, wondered if he could use a subtractive method rather than an additive method, to get the same resolution as black and white.

Imtersting what that means and how it works. LCD? Should look up.

Experts know how the official records are compiled, whether they are maps, computer manuals, diagnostic tests, or air crew checklists. They know when the steps have to be followed and when to make exceptions.

This approach to problem solving can be traced back to the German research psychologist Karl Duncker, one of the central figures in the Gestalt psychology school in Europe. The Gestalt school emphasized perceptual approaches to thought. Rather than treating thought as calculating ways of manipulating symbols, the Gestaltists viewed thought as learning to see better, using skills such as pattern recognition.

How does this relate to the field of sensemaking research? Do they overlap? What are the differences?

Duncker found that as his subjects worked on these problems, they simultaneously changed their understanding of the goal and assessed solutions. A subject might think of a solution, try it out, realize it would not work, realize what was missing, and then add to the definition of the goal. This new definition would suggest new approaches, and when these approaches failed, they helped to clarify the goal even further.

This.highlights a challenge with goal-setting in general and goal based delegation in particular. If unconstrained, the goals frequently change to account for new information and understanding. If goals are fixed, then this active iteration on the goal itself may be hindered.

Problem representation is another output, sometimes the necessary and sufficient output for determining how to proceed.

Link to note aboit solutions following an understanding of the problem

There are medical diagnosticians whose responsibility is primarily to provide skillful problem representations. Generating forecasts is itself a professional specialty in many fields.

Examples of problem presentation as a crucial part of decision making. Often ignored as a separate and important part.

The goals affect the way we evaluate courses of action, and the evaluation can help us learn to set better goals. The goals determine how we assess the situation, and the things we learn about the situation change the nature of the goals.

We build.a mental model of the situation and that mental model affects our sensibilities. I.e. if we explicitly work to reduce cycle time, then people become more conscious of how different aspects of their work affect cycle time and where everage points could lie.

When a gap or opportunity is identified, often we will try to diagnose it.

.c1

We may also want to project trends based on the diagnosis, to see how the situation may change. This is the forecasting process. In many situations, the primary need is to build a reliable forecast in order to determine whether the difficulty will disappear on its own, or will get worse and require action. The problem representation and diagnosis processes are linked to forecasting, which usually requires mental simulation.

.c2 Problem presentation can but need not include simulations of the past and the present. These help us better understand the problem and if it is worth doing somethomg about it.

De Groot (1945) and Isenberg (1984) have suggested that what triggers active problem solving is the ability to recognize when a goal is reachable.

This is significant. Inexperienced people may not ne able to start the problem solving process because they're unable to imagine a plausible solution. Seems like this has an impact on procrastination as well.

Experience lets us recognize the existence of opportunities.

This is highly impactful because without this recognition of opportunities and leverage points, the person and others would continue spending more effort for a worse result..This is why it is so important to reward this in a career framework.

For some problems, clarifying the initial state is the most important outcome. For example, diagnosing the disease causing a mysterious set of symptoms will usually enable a physician to determine the appropriate course of treatment. For other problems, such as rescuing people from a burning building, the diagnosis of how they got into the building is irrelevant.

Re action follows sensemaking

One of the most unexpected things I learned from reviewing the Apollo 13 episodes was the importance of forecasting. In my tally, fifteen of the episodes required forecasting. In four of these, the forecasts resulted in a revised problem representation, and in another three episodes the forecasting produced problem detection.

Forecasts are a form of mental (or otherwise) simulation that check the viability of current plans. But also very importantly, they are a way to test our assumptions and our understanding of the situation. If there is deviation from the forecast, then we are either incorrect, have missed some crucial information, or something has changed. All compel us to revise our understanding of the situation.

Forming an accurate problem representation was the most common activity for this incident.

Re action follows sensemaking

One application is to be less enthusiastic about rational planning approaches. Certainly there is value in trying to envision goals more clearly in planning and preparation. Nevertheless, we must accept the limitations of our ability to make plans for complex situations. We can prepare to improvise as we redefine the goals midway through a project.

This all is not to say that more formal methods could not be useful. Rather that we shouldn't give these formal methods undue credibility not to be misled by them.

Of course, it is a good idea to try to define the goal as clearly as possible before proceeding.

.c1

My skepticism about rational problem-solving methods is that they do not prepare you to improvise, act without all of the relevant information, or cope with unreliable data or shifting conditions. They do not prepare you to learn about the goals throughout the problem-solving process.

.c2

Even brainstorming, a method that has been around for decades, seems primarily a social activity. If the participants generate their ideas individually, the resulting set of suggestions is usually longer and more varied than when everyone works together.

Experienced problem solvers can distinguish genuine anomalies from transients.

The nonlinear model of problem solving shares a common perspective with the work of Isenberg (1984), Mintzberg, Raisinghani, and Theoret (1976), and Weick (1983).

The reference to Weick here implies a relation to sensemaking as well in addition to gestalt.

There are many things experts can see that are invisible to everyone else: Patterns that novices do not notice. Anomalies—events that did not happen and other violations of expectancies. The big picture (situation awareness). The way things work. Opportunities and improvisations. Events that either already happened (the past) or are going to happen (the future). Differences that are too small for novices to detect. Their own limitations. These aspects of expertise can be tied to the two primary sources of power we have been examining: pattern matching and mental simulation.

In contrast, office managers who have been in charge of large secretary pools for a long time would not have to interview many people. They are experienced secretary selectors. They might interview the first candidate on a list and recognize that he or she is much better than average. They do not need to look any further.

Step beyond batch processing

The paramedics recognized expertise when they saw it. Nine of the ten picked the actual paramedic. When asked why, they could not point to any one thing.

It takes an expert to spot one. Important re interviews to have the board consist of people who are at least a level above your level in skills being assessed.

Novices are confused by much that happens to them because they have so much trouble forming expectancies.

Have to make a prediction to calibrate your understnding aginst reality of the situation

One critical type of cue that surprises experts, but not novices, is the absence of a key event. Since novices do not know what is supposed to happen, they are slow to appreciate the significance of something’s not happening. Experts pick this up right away.

Only through expectancies can someone notice that something did not happen.

Since the experts have a mental model of the task, they know how the subtasks fit together and can adapt the way they perform individual subtasks to blend in with the others.

Clear process is essentially a crutch. Experts who understand the dynamics of the situation can successfully improvise outside of standard process. This is particularly important in novel situations where standard process could be harmful or destructive. Also re building process from experts' methods or intuition.

If they have to explain what they are doing to novices, they may have to stop and artificially break it down into subtasks. Often they feel uncomfortable teaching the separate steps because they know they are teaching some bad habits. They are teaching the novices to do the task in a choppy way. In the short run, though, this task decomposition makes it easier for the novices since they do not have to worry about the big picture. They just have to remember the steps. As part of their mental model of the task, experts know various tricks of the trade, along with the conditions for using them.

Again re process being a crutch and why it's hard to describe process in a complet, conditional, nuanced manner. I know I've felt uncomfortable simplifying a process, having to leave out the many but rare edge cases and contingencies.

Probing for counterfactual thinking, we asked what sequence of events might occur that would result in the ceiling lifting earlier than that, by noon. He was unable to imagine such a possibility. He had followed a set of rules to generate his prediction and could not conceive of a different world. To us, this signaled the fact that he was not an expert.

Interesting. Could perhaps be useful for interviewing. An expert in a skill should have an advanced mental model able to account for and imagine many different variations of a particular situation. Whereas novices may only consider one a single understanding of a situation. E.g. re performance management, being able to come up with dofferent potential reasons and different approaches, given these reasons.

This is one way to distinguish true experts from people who pretend to be experts. The pretenders have mastered many procedures and tricks of the trade; their actions are smooth. They show many of the characteristics of expertise. However, if they are pushed outside the standard patterns, they cannot improvise. They lack a sense of the dynamics of the situation.

Difference between mastering the process (the crutch, the map) vs the situation/skill (terrain).

The data confirmed the prediction made by the RPD model: skilled decision makers generate feasible options as the first ones they think of. Therefore, there is little to be gained by generating and then evaluating lots of options.

The most typical perspective is that experts know more; they have more facts and rules at their disposal. In this chapter I have taken a different perspective: expertise is learning how to perceive. The knowledge and rules are incidental.

Solutions follow sensemaking

novices follow rules, whereas the experts do not.

.c1

As adults, we do not believe we learned to use those training wheels so well that they became an ingrained part of our bicycle riding perspective. We outgrew the need for training wheels. We developed a sense of bicycle dynamics.

.c2 Process is a crutch

We usually find that there is not one expert. Rather, different people know worthwhile things in different areas. The job of cognitive task analysis is to focus on the expertise, not the experts. The aim is to find individuals whose experience is respected in the organization, in order to learn how they see their job.

Cognitive task analysis is a method for capturing expertise and making it accessible for training and system design.

Experts can perceive things that are invisible to novices: fine discriminations, patterns, alternate perspectives, missing events, the past and the future, and the process of managing decision-making activities.

Individual differences in abilities, strength, or other factors did not have the same impact as sheer amount of time spent practicing.

Volume best predictor of success

Drama, empathy, and wisdom are key. Stories are remembered because they are dramatic. They are used because we can identify with one or more of the actors. They are told and retold because of the wisdom they contain—the lessons that keep emerging with each telling.

The method we have found most powerful for eliciting knowledge is to use stories. If you ask experts what makes them so good, they are likely to give general answers that do not reveal much. But if you can get them to tell you about tough cases, nonroutine events where their skills made the difference, then you have a pathway into their perspective, into the way they are seeing the world. We call this the critical decision method, because it focuses attention on the key judgments and decisions that were made during the incident being described.

According to Kolodner, the stripped-down version of a case would be a description of a problem plus a solution. Next is the problem solution, and success or failure. Then add some additional factors involved. Most useful is a case with a problem, solution, success or failure, factors involved, and the reason the factors resulted in the outcome.