More on this book
Community
Kindle Notes & Highlights
“Those who believe that what you cannot quantify does not exist also believe that what you can quantify, does.” —AARON HASPEL
There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive.
Accountability ought to mean being held responsible for one’s actions. But by a sort of linguistic sleight of hand, accountability has come to mean demonstrating success through standardized measurement, as if only that which can be counted really counts.
If what is actually measured is a reasonable proxy for what is intended to be measured, and if it is combined with judgment, then measurement can help practitioners to assess their own performance, both for individuals and for organizations. But problems arise when such measures become the criteria used to reward and punish—when metrics become the basis of pay-for-performance or ratings.
Whenever reward is tied to measured performance, metric fixation invites gaming.
The demand for measured accountability and transparency waxes as trust wanes.
Practical or tacit knowledge is the product of experience: it can be learned, but cannot be conveyed in general formulas.
In Oakeshott’s famous example, there is the sort of abstract, recipe knowledge conveyed by cookbooks; but actually knowing how to make use of such knowledge (“beat an egg,” “whisk the mixture”) requires practical knowledge, based upon experience, that cannot be learned from books. Oakeshott criticized “rationalists” for assuming that the conduct of human affairs is a matter of applying the right formulas or recipes.
Or in the formulation of Alfie Kohn, a long-time critic of pay-for-performance, metrics “inhibits risk-taking, an inevitable concomitant of exploration and creativity. We are less likely to take chances, to play with possibilities, and to follow hunches, which may, after all, not pay off.”
For potential employers, degrees act as signals: they serve as a shorthand that allows employers to rank initial applicants for a job.
distinguish training, which is oriented to production and survival, from education, which is oriented to making survival meaningful,
One role is informational and diagnostic: the process of keeping track of various methods and procedures, and then comparing the outcomes, makes it possible to determine which are most successful.
health expenditure is what economists call a “luxury good”—the richer people are, the more they are willing to spend on it.
Thus, developing valid metrics of success and failure requires a good deal of local knowledge, knowledge that may be of no use in other circumstances—to the chagrin of those who look for universal templates and formulae. The hard part is knowing what to count, and what the numbers you have counted actually mean in context.
the Sarbanes-Oxley Act of 2002, which sought to strengthen corporate accountability, in part by holding the members of the board of directors of public corporations legally liable for the accuracy of financial statements.
economist Frank Knight noted almost a century ago, entrepreneurship entails “immeasurable uncertainty,” which is not susceptible to metric calculation.
One conception of the purpose of social science was articulated in the nineteenth century by Auguste Comte: Savoir pour prévoir, prévoir pour prévenir (Know in order to predict, predict in order to avert [the previously unanticipated consequences of our actions]).
Economists Bengt Holmström and Paul Milgrom have described it in more formal terms as a problem of misaligned incentives: workers who are rewarded for the accomplishment of measurable tasks reduce the effort devoted to other tasks.
Measured performance encourages what Robert K. Merton called “the imperious immediacy of interests … where the actor’s paramount concern with the foreseen immediate consequences excludes consideration of further or other consequences.”
THE CHECKLIST
What kind of information are you thinking of measuring?
Measurement becomes much less reliable the more its object is human activity, since the objects—people—are self-conscious, and are capable of reacting to the process of being measured.
How useful is the information?
ask yourself, is what you are measuring a proxy for what you really want to know?
How useful are more metrics?
the more you measure, the greater the likelihood that the marginal costs of measuring will exceed the benefits.
What are the costs of not relying upon standardized measurement?
To what purposes will the measurement be put, or to put it another way, to whom will the information be made transparent?
What are the costs of acquiring the metrics?
every moment you or your colleagues or employees are devoting to the production of metrics is time not devoted to the activities being measured.
Ask why the people at the top of the organization are demanding performance metrics.
How and by whom are the measures of performance developed?
Remember that a system of measured performance will work to the extent that the people being measured believe in its worth.
Remember that even the best measures are subject to corruption or goal diversion.
Remember that sometimes, recognizing the limits of the possible is the beginning of wisdom. Not all problems are soluble, and even fewer are soluble by metrics.
