The Tyranny of Metrics

by Jerry Z. Muller

Metric fixation leads to a diversion of resources away from frontline producers toward managers, administrators, and those who gather and manipulate data. When metrics are used by managers

“Not everything that can be counted counts, and not everything that counts can be counted.”7)

Campbell’s Law,” named for the American social psychologist Donald T. Campbell, holds that “[t]he more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”9 In a variation named for the British economist who formulated it, we have Goodhart’s Law, which states, “Any measure used for control is unreliable.”10 To put it another way, anything that can be measured and rewarded will be gamed. We will see many variations on this theme.

Then there are the cultural peculiarities of some American bureaucracies (corporate, governmental, and nonprofit), which assume that each person can and should be rotated through an ascending hierarchy of posts, both within an organization and among organizations. This militates against developing a depth of expertise that would allow for meaningful evaluation of the significance and qualitative importance of work done by subordinates. Hence the attractiveness of relying on measurable, quantitative, criteria.

People learn just enough to be dangerous.

A hallmark of practical, local knowledge, as James Scott has noted, is that “it is as economical and accurate as it needs to be, no more and no less, for addressing the problem at hand.”6 By contrast, the degree of numerical precision promised by metrics may be far greater than is required by actual practitioners, and attaining that precision requires an expenditure of time and effort that may not be worthwhile. The quest for precision may therefore be wasteful, and resented for that reason by those required to sacrifice their time and ingenuity.

A feel for the whole and a sense for the unique are precisely what numerical metrics cannot supply.

Reward for measured performance in higher education is touted by its boosters as making universities “more like a business.” But businesses have a built-in restraint on devoting too much time and money to measurement—at some point, it cuts into profits. Ironically, since universities and other nonprofit institutions have no such bottom line, government or accrediting agencies or the university’s administrative leadership can extend metrics endlessly.25 The effect is to increase costs or to divert spending from the doers to the administrators—which usually suits the latter just fine. It is hard to find a university where the ratio of administrators to professors and of administrators to students has not risen astronomically in recent decades.26 And the same holds true on the national level.

In academia as elsewhere, that which gets measured gets gamed.

As Niall Ferguson has put it, “those whom the gods want to destroy they first teach math.”

We live in a world in which privacy is being eroded both through technology (the Internet) and a culture that proclaims the virtue of candor while dismissing the need for shame. In such a post-privacy society, people are inclined to overlook the value of secrecy.8 Thus, the power of “transparency” as a magic formula is such that its counterproductive effects are often ignored. “Sunlight is the best disinfectant” has become the credo of the new faith of Wikileakism: the belief that making public the internal deliberations of all organizations and governments will make the world a better place. But more often, the result is paralysis. Politicians forced to reveal their every action are unable to arrive at compromises that make legislation possible. Officials who need to fear that their internal deliberations will be made public are less positioned to make effective public policy. Intelligence agencies that require secrecy to gather information on the nation’s enemies are thwarted. In each case, transparency becomes the enemy of performance.

Goal displacement through diversion of effort to what gets measured.

workers who are rewarded for the accomplishment of measurable tasks reduce the effort devoted to other tasks.2 The result is that the metric means comes to replace the organizational ends that those means ought to serve.

Promoting short-termism.

Costs in employee time.

Diminishing utility.

benefits.

Rule cascades.

further slows down the institution’s functioning

Rewarding luck.

Discouraging risk-taking.

Discouraging innovation.

Discouraging cooperation and common purpose.

Degradation of work.

Costs to productivity.

As we’ve seen time and again, measurement is not an alternative to judgment: measurement demands judgment: judgment about whether to measure, what to measure, how to evaluate the significance of what’s been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available.

the ease of measuring may be inversely proportional to the significance of what is measured. To

Measurement instruments, such as tests, are invaluable, but they are most useful for internal analysis by practitioners rather than for external evaluation by public audiences who may fail to understand their limits.

8. How and by whom are the measures of performance developed? Accountability metrics are less likely to be effective when they are imposed from above, using standardized formulas developed by those far from active engagement with the activity being measured.

Remember that a system of measured performance will work to the extent that the people being measured believe in its worth.

even the best measures are subject to corruption or goal diversion.