How to Measure Anything: Finding the Value of Intangibles in Business

by Douglas W. Hubbard

usually you want to measure something because it supports some decision. And these decisions tend to have thresholds where one action is required if a value is above it, and another is required if the value is below it.

I call this an instinctive Bayesian approach because when people update their prior uncertainties with new information, as you did with the jelly beans, there is evidence to believe that those people update their knowledge in a way that is mostly Bayesian.

I found a book with the inclusive-sounding title Objective Measurement.10 The title might lead you to believe such a book would be a comprehensive treatment of the issues of measurement that might be interesting to any astronomer, chemical engineer, or economist. However, this five-volume work is only about human performance and education testing. It’s as if you saw an old map titled “Map of the World” that was really a map of a single, remote Pacific island, made by people unaware that they were on just one part of a larger planet.

suppose you wanted to evaluate the proficiency of project managers based on their performance when assigned to various projects. If you have a large number of project managers, you probably have to have several judges, each assigned to a single project manager. The judges, in fact, might be the project managers’ immediate superiors (as others are not familiar with the project). The assigned projects, also, probably vary greatly in difficulty. But now suppose all project managers, regardless of their project or whom they reported to, had to compete for the same limited pool of promotions or bonuses. Those assigned to a “hard grader” or a difficult project would be at a disadvantage that had nothing to do with their performance. The comparison of different project managers would not be invariant (i.e., independent) of who judged them or the projects they were judged on.

A very different behavior occurs when the task is to generate exact values for a business case, especially one where the estimator has a stake in the outcome, as opposed to a calibrated estimator providing an initial 90% confidence interval (CI). Sitting in a room, one or more people working on the business case will play a game with each estimate. Forced to choose exact values, no matter how uncertain or arbitrary, the estimators ask: “How much should this value be to be agreeable to others and still be sufficient to prove my (predetermined) point?” It is almost as if the terms “consensus” and “fact” are used as synonyms.

A different and disturbing trend in management decision making is to develop a type of weighted score where the score and the weight are both subjective scales with arbitrary point values, not z-scores like those Dawes used. Like the simple linear models discussed previously, these methods might ask a project portfolio manager to rate a proposed project in categories such as “strategic alignment,” “organizational risk,” and so on. Most of these methods have between 4 and 12 categories of evaluation, but some have over 100. The proposed project is typically given a score of, say, 1 to 5 in each of these categories. The scores in each category are then multiplied by a weighting factor—perhaps also a scale of 1 to 5—which is meant to account for the relative importance of each of the scores categorized. The weighting factors are usually standardized for a given company so that all projects are evaluated by comparable criteria. The adjusted scores are then totaled to give an overall score for the proposed project. Scores are methods of attempting to express relative worth, preference, and so on, without employing a real unit of measure. Although scoring is fairly called one type of ordinal measurement system we discussed in Chapter 3, I’ve always considered an arbitrary score to be a sort of measurement wannabe. The problems with this method are one of the key criticisms I make about popular risk assessment methods in my second book The Failure of Risk Management: Why It’s Bro...