More on this book
Community
Kindle Notes & Highlights
Read between
October 7 - November 20, 2019
When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science. —Lord Kelvin (1824–1907), British physicist and member of the House of Lords
In other words, management needs a method to analyze options for reducing uncertainty about decisions.
So don’t confuse the proposition that anything can be measured with everything should be measured.
Applied Information Economics: A Universal Approach to Measurement Define the decision. Determine what you know now. Compute the value of additional information. (If none, go to step 5.) Measure where information value is high. (Return to steps 2 and 3 until further measurement is not needed.) Make a decision and act on it. (Return to step 1 and repeat as each action creates new decisions.) Each of these steps will be explained in more detail in chapters to come. But, in short: measure what matters, make better decisions. My hope is that as we raise the curtain on each of these steps in the
...more
Success is a function of persistence and doggedness and the willingness to work hard for twenty-two minutes to make sense of something that most people would give up on after thirty seconds. —Malcolm Gladwell, Outliers: The Story of Success
best-known example of such a “Fermi question” was Fermi asking his students to estimate the number of piano tuners in Chicago. His students—science and engineering majors—would begin by saying that they could not possibly know anything about such a quantity. Of course, some solutions would be to simply do a count of every piano tuner perhaps by looking up advertisements, checking with a licensing agency of some sort, and so on. But Fermi was trying to teach his students how to solve problems where the ability to confirm the results would not be so easy. He wanted them to figure out that they
...more
This approach to solving a Fermi question is known as a Fermi decomposition or Fermi solution. This method helped to estimate the uncertain quantity but also gave the estimator a basis for seeing where uncertainty about the quantity came from. Was the big uncertainty about the share of households that had tuned pianos, how often a piano needed to be tuned, how many pianos a tuner can tune in a day, or something else? The biggest source of uncertainty would point toward a measurement that would reduce the uncertainty the most.
The lesson for business is to avoid the quagmire that uncertainty is impenetrable and beyond analysis.
Emily’s example provides more than one lesson for business. First, even touchy-feely-sounding things like “employee empowerment,” “creativity,” or “strategic alignment” must have observable consequences if they matter at all. I’m not saying that such things are “paranormal,” but the same rules apply.
Fermi might say, “Yes, there are a lot of things you don’t know, but what do you know?”
The concept of measurement as “uncertainty reduction” and not necessarily the elimination of uncertainty is a central theme of this book.
Concept of measurement. The definition of measurement itself is widely misunderstood. If one understands what “measurement” actually means, a lot more things become measurable. Object of measurement. The thing being measured is not well defined. Sloppy and ambiguous language gets in the way of measurement. Methods of measurement. Many procedures of empirical observation are not well known. If people were familiar with some of these basic methods, it would become apparent that many things thought to be immeasurable are not only measurable but may already have been measured. A good way
Implicit or explicit in all of these answers is that measurement is certainty—an exact quantity with no room for error. If that was really what the term means, then, indeed, very few things would be measurable.
Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations.
A problem well stated is a problem half solved. —Charles Kettering (1876–1958), American inventor, holder of 300 patents, including electrical ignition for automobiles
So, what do you mean by ‘mentorship’?” The person almost immediately responded, “I don’t think I know,” to which I said, “Well, then maybe that’s why you believe it is hard to measure. You haven’t figured out what it is.”
f a thing exists, it exists in some amount, if it exists in some amount, it can be measured”
They may barely recall, but misunderstand, concepts like “statistical significance.”
we see there is a 93.75% chance that the median of the entire population of employees is between those two numbers. I call this the “Rule of Five.” The Rule of Five is simple, it works, and it can be proven to be statistically valid for a wide variety of problems. With a sample this small, the range might be very wide, but if it is significantly narrower than your previous range, then it counts as a measurement.
psychologists Daniel Kahneman and Amos Tversky showed that people will routinely overestimate the probability of extreme sample results.
what makes a measurement of high value is a lot of uncertainty combined with a high cost of being wrong.
A well-known example of this is the so-called Houston Miracle of the Texas school system in the 1990s. Public schools were under a new set of performance metrics to hold educators accountable for results. It is now known that the net effect of this “miracle” was that schools were incentivized to find ways to drop low-achieving students from the rolls. This is hardly the outcome most taxpayers thought they were funding.
“How premature is the death?” Should the death of a very old person be considered equal to that of a younger person, when limited resources force us to make choices? At one point, the EPA considered using what it called a “senior death discount.” A death of a person over 70 was valued about 38% less than a person under 70.
As ingrained as measurements-for-the-sake-of-measurements seemed to be with many of the scientists, they did understand the need for measurements to support specific decisions. Dr. Shepherd added, “We realized that forecasting intervention impacts with AIE not only implicitly laid out the impact pathway, but also pointed us to which metrics had highest value.”
The Office of Management and Budget already required the EPA to produce economic arguments for any proposed environmental policy. The EPA had to compute costs of compliance and benefits to the public for each policy it wanted to enforce. Several such studies showed the economic impact of different types of the most common drinking water contamination. The EPA often resorted to a willingness-to-pay (WTP) argument, but sometimes it used only workdays lost in calculating the cost of contamination. By focusing on how SDWIS
Even though the ranges in all the variables expressed a lot of uncertainty, only one variable merited measurement: the average health effects of new safe drinking water policies.
On further review, it turns out that this particular economic analysis looked only at extremely conservative economic impacts of water contamination—basically, just workdays lost and the economic impact of the loss. However, most people would agree that being sick is worse than just losing a couple of days of wages.
is an example of how an intangible like public health is quantified for an IT project. I’ve seen many IT projects dismiss much more easily measured benefits as “immeasurable” and exclude them from the ROI calculation.
This example is about what didn’t have to be measured. Only 1 variable out of 99 turned out to require uncertainty reduction. The initial calibrated estimates were sufficient for the other 98.
“What surprised me was the convoy model that showed most fuel was burned on logistics routes. The study even uncovered that tank operators would not turn tanks off if they didn’t think they could get replacement starters. That’s something that a logistician in 100 years probably wouldn’t have thought of.” The more “abstract” benefits of an everything-is-measurable
like typical management consulting: heavy on high-minded concepts and visions, no measurements and no new information.
This information might seem overwhelming. But, as with almost everything else in business or life, it’s often just a matter of getting started on a few examples, working through a problem, and seeing the results.
He preached that if you don’t have a measurement program, you don’t have a quality program. To Deming, quality was the consistency with which expectations were met. The lack of meeting defined expectations
It might also be helpful to remember the distinction between stated and revealed preferences. In a survey, customers state their preferences. When they are making (or not making) purchases, they reveal their preferences. The ultimate expression of quality is the premium customers are willing to pay for a product.
how to measure the value of information technology. I ask, “Why, are you considering getting rid of it?”
If it’s really that important, it’s something you can define. If it’s something you think exists at all, it’s something you’ve already observed somehow.
can quantify your current uncertainty with calibrated estimates.
