More on this book
Community
Kindle Notes & Highlights
Often an important decision requires better knowledge of the alleged intangible, but when an executive believes something to be immeasurable, attempts to measure it will not even be considered.
Anything can be measured. If something can be observed in any way at all, it lends itself to some type of measurement method. No matter how “fuzzy” the measurement is, it’s still a measurement if it tells you more than you knew before.
As far as the propositions of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality. —Albert Einstein
Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations.
For all practical decision-making purposes, we need to treat measurement as observations that quantitatively reduce uncertainty. A mere reduction, not necessarily elimination, of uncertainty will suffice for a measurement.
Clarification Chain If it matters at all, it is detectable/observable. If it is detectable, it can be detected as an amount (or range of possible amounts). If it can be detected as a range of possible amounts, it can be measured.
the purpose of the measurement gives us clues about what the measure really means and how to measure it.
sometimes even small samples can tell you something that improves the odds of making a better bet in real decisions.
“If you don’t know what to measure, measure anyway. You’ll learn what to
what makes a measurement of high value is a lot of uncertainty combined with a high cost of being wrong.
Four Useful Measurement Assumptions It’s been measured before. You have far more data than you think. You need far less data than you think. Useful, new observations are more accessible than you think.
when you know almost nothing, almost anything will tell you something
An unidentified decision is no better than having no decision in mind at all.
The data on the dashboard was usually not selected with specific decisions in mind based on specific conditions for action.
If someone is having a hard time getting their measurement problem to fit a specific decision according to these rules, they may be making some unnecessary presumptions about what constitutes a decision.
the great statistician George Box put it, “Essentially, all models are wrong, but some are
Uncertainty: The lack of complete certainty, that is, the existence of more than one possibility. The “true” outcome/state/result/value is not known. Measurement of Uncertainty: A set of probabilities assigned to a set of possibilities. For example: “There is a 60% chance this market will more than double in five years, a 30% chance it will grow at a slower rate, and a 10% chance the market will shrink in the same period.” Risk: A state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. Measurement of Risk: A set of possibilities each with
...more
Calibrated probability assessments are the key to measuring your current state of uncertainty about anything.
if we can’t put a probability on some event, then the event is considered uncertain. If you can put a probability on an event, then it is considered a risk—without regard to whether the probability is for a potential loss.
Knowing what you know now about something actually has an important and often surprising impact on how you should measure it or even whether you should measure it.
assessing uncertainty is a general skill that can be taught with a measurable improvement.
People generally fare slightly better on the true/false tests, but, on average, they still tend to be overconfident—and overconfident by enough that even a small sample of 10 can usually detect it.
In my calibration training classes, I’ve been calling this the “equivalent bet test.”
“absurdity test.” It reframes the question from “What do I think this value could be?” to “What values do I know to be ridiculous?” We look for answers that are obviously absurd and then eliminate them until we get to answers that are still unlikely but not entirely implausible. This is the edge of our knowledge about that quantity.
An assumption is a statement we treat as true for the sake of argument, regardless of whether it is true.
if you are allowed to model your uncertainty with ranges and probabilities, you do not have to state something you don’t know for a fact. If you are uncertain, your ranges and assigned probabilities should reflect that.
studies have shown that it is quite possible to experience an increase in confidence about decisions and forecasts without actually improving things—or even by making them worse.
all risk in any project investment ultimately can be expressed by one method: the ranges of uncertainty on the costs and benefits and probabilities on events that might affect them.
The McNamara Fallacy “The first step is to measure whatever can be easily measured. This is okay as far as it goes. The second step is to disregard that which can’t easily be measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can’t be measured easily isn’t important. This is blindness. The fourth step is to say that what can’t easily be measured really doesn’t exist. This is suicide.” —Charles Handy, The Empty Raincoat (1995), describing the Vietnam-era measurement policies of Secretary of Defense Robert McNamara
there are really only three basic reasons why information ever has value to a business: Information reduces uncertainty about decisions that have economic consequences. Information affects the behavior of others, which has economic consequences. Information sometimes has its own market value.
The difference between the EOL before a measurement (perhaps based only on initial calibrated estimates) and the EOL after a measurement is called the “Expected Value of Information” (EVI). In other words, the value of information is equal to the value of the reduction in risk.
Value of Information
