More on this book
Community
Kindle Notes & Highlights
Started reading
September 12, 2019
In other words, management needs a method to analyze options for reducing uncertainty about decisions.
The fact is that the superiority of even simple quantitative models for decision making has been established for many areas normally thought to be the preserve of expert intuition, a point this book will spend some time supporting with citations of several published studies.
So don’t confuse the proposition that anything can be measured with everything should be measured.
Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations.
Once managers figure out what they mean and why it matters, the issue in question starts to look a lot more measurable. This is usually my first level of analysis when I conduct what I’ve called “clarification workshops.” It’s simply a matter of clients stating a particular, but initially ambiguous, item they want to measure. I then follow up by asking “What do you mean by <fill in the blank>?” and “Why do you care?”
Rule of Five There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population.
The Single Sample Majority Rule (i.e., The Urn of Mystery Rule) Given maximum uncertainty about a population proportion—such that you believe the proportion could be anything between 0% and 100% with all values being equally likely—there is a 75% chance that a single randomly selected sample is from the majority of the population.
I’ll say more about this later, but a proven formula from the field of decision theory allows us to compute a monetary value for a given amount of uncertainty reduction. I put this formula in an Excel macro and, for years, I’ve been computing the economic value of measurements on every variable in scores of various large business decisions. I found some fascinating patterns through this calculation but, for now, I’ll mention this: Most of the variables in a business case had an information value of zero.
what makes a measurement of high value is a lot of uncertainty combined with a high cost of being wrong.
If managers can’t identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value.
The data on the dashboard was usually not selected with specific decisions in mind based on specific conditions for action. It was often merely hoped that when the right conditions arose in the data, the manager would recognize a need to act and already know what action is required in sufficient detail that they could react without any delay.
A decision has two or more realistic alternatives. It could be to fund a new product-development project or not. It could be to build a dam or not. It could be to expend resources to reduce exposure to some risk. But it cannot be a false-dichotomy—as we saw is sometimes the case when managers want to know the value of IT or the value of clean drinking water.
A decision has uncertainty. If there is no uncertainty about a decision, then it’s not really much of a dilemma.
A decision has potentially negative consequences if it turns out you took the wrong position. Just as a lack of uncertainty makes for no real dilemma in a decision, the lack of any consequence also means there is no dilemma.
They reasoned that this variance among different professions shows that putting odds on uncertain things must be a learned skill. Researchers learned how experts can measure whether they are systematically underconfident, overconfident, or have other biases about their estimates. Once people conduct this self-assessment, they can learn several techniques for improving estimates and measuring the improvement. In short, researchers discovered that assessing uncertainty is a general skill that can be taught with a measurable improvement.
If you said you were 100% confident on any answer, you must get it right. Getting even one 100% confident answer wrong is sufficient evidence that you are overconfident. Remarkably, just over 15% of responses where the stated confidence was 100% turned out to be wrong and some individuals (we will see shortly) would get a third or more of their “certain” answers wrong.
Simply practicing with multiple tests may improve calibration scores but, for most people, becoming well calibrated also requires learning some specific techniques. One particularly powerful tactic for becoming more calibrated is to pretend to bet money.
Research indicates that even just pretending to bet money significantly improves a person’s ability to assess odds.7 In fact, actually betting money turns out to be only slightly better than pretending to bet.
If they are testing another participant’s 90% CI for some quantity, I expect them to ask, “What would you prefer: (1) to win $1,000 if the correct answer is within your bounds or (2) to spin a dial that gives a 90% chance of paying off $1,000?” But for some reason this seems to confuse most people who first attempt this.
Another calibration training method involves asking people to identify potential problems for each of their estimates. Just assume your answer is wrong and then explain to yourself why you were wrong.
This means that estimators must be 95% sure that the true value is less than the upper bound. If they are not that certain, they should increase the upper bound until they are 95% certain.
A shortcut can apply in some situations. If we had all normal distributions and we simply wanted to add or subtract ranges—such as a simple list of costs and benefits—we might not have to run a Monte Carlo simulation. If we just wanted to add up the three types of savings in our example, we can use a simple calculation. Use these six steps to produce a range: Subtract the midpoint from the upper bound for each of the three cost savings ranges: in this example, $20 – $15 = $5 for maintenance savings; we also get $5 for labor savings and $3 for materials savings. Square each of the values from
...more
In summary, the range interval of the total is equal to the square root of the sum of the squares of the range intervals.
Using all optimistic values for the optimistic case and all pessimistic values for the pessimistic case is a common error and no doubt has resulted in a large number of misinformed decisions.
But we don’t just want to add these up; we want to multiply them by the production level, which is also a range. The simple range addition method doesn’t work with anything other than subtraction or addition so we would need to use a Monte Carlo simulation. A Monte Carlo simulation is also required if these were not all normal distributions.
