The Tyranny of Metrics

by Jerry Z. Muller

The most characteristic feature of metric fixation is the aspiration to replace judgment based on experience with standardized measurement. For judgment is understood as personal, subjective, and self-interested. Metrics, by contrast, are supposed to provide information that is hard and objective.

Metric fixation leads to a diversion of resources away from frontline producers toward managers, administrators, and those who gather and manipulate data.

When metrics are used by managers as a tool to control professionals, it often creates a tension between the managers who seek to measure and reward performance, and the ethos of the professionals (doctors, nurses, policemen, teachers, professors, etc.). The professional ethos is based on mastery of a body of specialized knowledge acquired through an extended process of education and training; autonomy and control over work; an identification with one’s professional group and a sense of responsibility toward colleagues; a high valuation of intrinsic rewards; and a commitment to the interests of clients above considerations of cost.

A key premise of metric fixation concerns the relationship between measurement and improvement.

The key components of metric fixation are ■ the belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data (metrics); ■ the belief that making such metrics public (transparent) assures that institutions are actually carrying out their purposes (accountability); ■ the belief that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay-for-performance) or reputational (rankings). Metric fixation is the persistence of these beliefs despite their unintended negative consequences when they are put into practice.

“Not everything that can be counted counts, and not everything that counts can be counted.”7) Most organizations have multiple purposes, and that which is measured and rewarded tends to become the focus of attention, at the expense of other essential goals. Similarly, many jobs have multiple facets, and measuring only a few aspects creates incentives to neglect the rest.

Whenever reward is tied to measured performance, metric fixation invites gaming.

What has come to be called “Campbell’s Law,” named for the American social psychologist Donald T. Campbell, holds that “[t]he more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”9 In a variation named for the British economist who formulated it, we have Goodhart’s Law, which states, “Any measure used for control is unreliable.”10 To put it another way, anything that can be measured and rewarded will be gamed.

Gaming through creaming. This takes place when practitioners find simpler targets or prefer clients with less challenging circumstances, making it easier to reach the metric goal, but excluding cases where success is more difficult to achieve.

Other would-be reformers sought to bring to the school system the fruits of the industrial efficiency movement, founded by Frederick Winslow Taylor, an American engineer who coined the term “scientific management” in 1911.

The demand for measured accountability and transparency waxes as trust wanes.

“Every profession is a conspiracy against the laity,” wrote George Bernard Shaw in his play, The Doctor’s Dilemma.

gerontocracy.

Yet by the 1990s, despite a number of studies indicating that patient empowerment did nothing to contain costs or improve quality of care, the model of the patient as consumer in the marketplace of medical services became ever more popular among politicians and policymakers on both sides of the political spectrum10

The twin tasks of organizations thus becomes how to provide information to organizational superiors about the activities of their subordinates, and how to create systems of reward to align the interests of the agents with those of the principals. The quest for information leads to performance metrics: standardized numbers that will efficiently convey to the principals how well their agents are carrying out the principals’ goals.

The professional management literature derived its own conclusions from principal-agent theory: that management is a matter of setting clear goals, and then of monitoring and incentivizing. It relies upon information and reporting systems on the one hand, and cleverly structured rewards on the other.

[I]t is a characteristic of practical knowledge that it is not susceptible of formulation of this kind. Its normal expression is in a customary or traditional way of doing things, or, simply, in practice. And this gives it the appearance of imprecision and consequently of uncertainty, of being a matter of opinion, of probability rather than truth.

The error of rationalism, for Oakeshott, is its failure to appreciate the necessity of practical knowledge and of knowledge of the peculiarity of circumstances.2

judgment is a sort of skill at grasping the unique particularities of a situation, and it entails a talent for synthesis rather than analysis, “a capacity for taking in the total pattern of a human situation, of the way in which things hang together.”7 A feel for the whole and a sense for the unique are precisely what numerical metrics cannot supply.

In academia as elsewhere, that which gets measured gets gamed.

Nowhere are metrics in greater vogue than in the field of medicine. Nowhere, perhaps, are they more promising. And the stakes are high.

A great deal of corporate dysfunction comes from pay-for-performance schemes that are narrowly tailored to measure a single outcome. Problems occur both at the top and at the bottom of the corporate ladder.

Performance indicators can certainly aid, but not replace, the key functions of management: thinking ahead, judging, and deciding.

Goal displacement through diversion of effort to what gets measured.

Promoting short-termism.

Costs in employee time.

Diminishing utility.

Rule cascades.

Rewarding luck.

Discouraging risk-taking.

Discouraging innovation.

Discouraging cooperation and common purpose.

Degradation of work.

Costs to productivity.