The Tyranny of Metrics

by Jerry Z. Muller

Pay-for-performance schemes try to compensate for this by what is known as “risk adjustment.” But calculations of the degree of risk are at least as prone to mismeasurement and manipulation as other metrics.

But metrics tend to be most successful for those interventions and outcomes that are almost entirely controlled by and within the organization’s medical system, as in the case of checklists of procedures to minimize central line–induced infections.

There are indeed circumstances when pay for measured performance fulfills that promise: when the work to be done is repetitive, uncreative, and involves the production or sale of standardized commodities or services; when there is little possibility of exercising choice over what one does; when there is little intrinsic satisfaction in it; when performance is based almost exclusively on individual effort, rather than that of a team; and when aiding, encouraging, and mentoring others is not an important part of the job.

“Extrinsic rewards become an important determinant of job satisfaction only among workers for whom intrinsic rewards are relatively unavailable.”

People do want to be rewarded for their performance, both in terms of recognition and remuneration. But there is a difference between promotions (and raises) based on a range of qualities, and direct remuneration based on measured quantities of output. For most workers, contributions to their company include many activities that are intangible but no less real: coming up with new ideas and better ways to do things, exchanging ideas and resources with colleagues, engaging in teamwork, mentoring subordinates, relating to suppliers or customers, and more. It’s appropriate to reward such activities through promotions and bonuses—even if it is more difficult to document and requires a greater degree of judgment by those who decide on the rewards.

Dan Cable and Freek Vermeulen of the London Business School recall many of the problems we have explored: the depressive effect of performance pay on creativity; the propensity to cook the books; the inevitable imperfections of the measurement instruments; the difficulty of defining long-term performance; and the tendency for extrinsic motivation to crowd out intrinsic motivation.

“even though over half of the companies used forced ranking, the respondents reported that this approach resulted in lower productivity, inequity, skepticism, decreased employee engagement, reduced collaboration, damage to morale, and mistrust in leadership.”

Yet other companies are dropping annual ratings in favor of “crowdsourced” continuous performance data, by which supervisors, colleagues, and internal customers provide ongoing online feedback about employee performance. That may be substituting the frying pan for the fire, as employees constantly game for compliments, while resenting the omnipresent surveillance of their activities

Sometimes, the way in which managers and employees are addressed by their company actually influences the way they think, so that they come to act in the narrowly self-interested way posited by the most reductive versions of principal-agent theory, with deceit and guile.

Gaming the metrics often takes the form of diverting resources away from their best long-term uses to achieve measured short-term goals.

A focus on measurable performance indicators can lead managers to neglect tasks for which no clear measures of performance are available, as the organizational scholars Nelson Repenning and Rebecca Henderson have recently noted.

Unable to count intangible assets such as reputation, employee satisfaction, motivation, loyalty, trust, and cooperation, those enamored of performance metrics squeeze assets in the short term at the expense of long-term consequences. For all these reasons, reliance upon measurable metrics is conducive to short-termism, a besetting malady of contemporary American corporations.

Managers and employees learn to lie, to massage, embellish, or disguise the numbers that are used to calculate their pay. But since these are the very numbers that executives use to coordinate the activities of the organization and decide on the allocation of future resources, the productivity and efficiency of the organization is damaged as resources are misallocated.

The attempt to substitute precise measurement for informed judgment also limits innovation, which necessarily entails guesswork and risk.

Most companies are wedded to highly analytical methods for evaluating investment opportunities. Still, it remains enormously hard to assess long-term R&D programs with quantitative techniques…. Usually, the data, or even reasonable estimates, are simply not available. Nonetheless, all too often these tools become the ultimate arbiter of what gets funded and what does not. So short-term projects with more predictable outcomes beat out the long-term investments needed to replenish technical and operating capabilities.

Performance metrics as a measure of accountability help to allocate blame when things go badly, but do little to encourage success,28 especially when success requires imagination, innovation, and risk.

Performance indicators can certainly aid, but not replace, the key functions of management: thinking ahead, judging, and deciding.

Goal displacement through diversion of effort to what gets measured. Goal displacement comes in many varieties. When performance is judged by a few measures, and the stakes are high (keeping one’s job, getting a raise, raising the stock price at the time that stock options are vested), people will focus on satisfying those measures—often at the expense of other, more important organizational goals that are not measured.1 Economists Bengt Holmström and Paul Milgrom have described it in more formal terms as a problem of misaligned incentives: workers who are rewarded for the accomplishment of measurable tasks reduce the effort devoted to other tasks.2 The result is that the metric means comes to replace the organizational ends that those means ought to serve.

Promoting short-termism. Measured performance encourages what Robert K. Merton called “the imperious immediacy of interests … where the actor’s paramount concern with the foreseen immediate consequences excludes consideration of further or other consequences.”3 In short, advancing short-term goals at the expense of long-range considerations.

Costs in employee time. To the debit side of the ledger must also be added the transactional costs of metrics: the expenditure of employee time by those tasked with compiling and processing the metrics—not to speak of the time required to actually read them. That is exacerbated by the “reporting imperative”—the perceived need to constantly generate information, even when nothing significant is going on. Sometimes the metric of success is the number and size of the reports generated, as if nothing is accomplished unless it is extensively documented. Those within the organization end up spending more and more time compiling data, writing reports, and attending meetings at which the data and reports are coordinated. So, as the heterodox management ...

Diminishing utility. Sometimes, newly introduced performance metrics will have immediate benefits in discovering poorly performing outliers.5 Having gleaned the low-hanging fruit, there is tendency to expect a continuingly bountiful harvest. The problem is that the metrics continue to get collected from everyone. And soon the marginal costs of assembling and analyzing the metrics exceed the marginal benefits.

Rule cascades. In an attempt to staunch the flow of faulty metrics through gaming, cheating, and goal diversion, organizations institute a cascade of rules. Complying with them further slows down the institution’s functioning and diminishes its efficiency.

Rewarding luck. Measuring outcomes when the people involved have little control over the results is tantamount to rewarding luck. It means that people are rewarded or penalized for outcomes that are actually independent of their efforts. ...

Discouraging risk-taking. Attempts to measure productivity through performance metrics have other, more subtle effects: they not only promote short-termism, as noted earlier, but also discourage initiative and risk-taking. The intelligence analysts who ultimately located Bin Laden worked on the problem for years. If measured at any point, their productivity would have seemed to be zero. Month after month, their failure rate was 100 percent, until they achieved success. From the perspective of their superiors, allowing the analysts to work on the project for years involved a high degree of risk: the investment in time...

Discouraging innovation. When people are judged by performance metrics, they are incentivized to do what the metrics measure, and what the metrics measure will be some established goal. But that impedes innovation, which means doing something that is not yet established, indeed hasn’t been tried out. Innovation involves experimentation. Trying out something new entails risk, including the possibility, perhap...

Discouraging cooperation and common purpose. Rewarding individuals for measured performance diminishes the sense of common purpose as well as the social relationships that provide the unmeasureable motivation for cooperation and institutional effectiveness.7 Reward based on measured performance tends to promote not cooperation but competition. If the individuals or units respond to the incentives created, rather than aiding, assisting, and advising one another, they strive to maximize their own metrics, ignoring, or even sabotaging, their fellows. As Donald Berwick, a leading medical reformer, has recounted, One hospital CEO described to me his system of profit-center management, in which middle management bonuses depended on local budget performance...

Degradation of work. Compelling the people in an organization to focus their efforts on the narrow range of what gets measured leads to a degradation of the experience of work. Edmund Phelps, a Nobel Prize winning economist, claims in his book Mass Flourishing: How Grassroots Innovation Created Jobs, Challenge, and Change that one of the virtues of capitalism is its ability to provide “the experience of mental stimulation, the challenge of new problems to solve, the chance to try the new, and the excitement of venturing into the unknown.”9 That is indeed a possibility under capitalism. But those subject to performance metrics are forced to focus their efforts on limited goals, imposed by others, who may not understand the work that they do. For the workers under scrutiny, mental stimulation is dulled, they decide neither the problems to be solved nor how to solve them, and there is no excitement of venturing into the unknown because the unknown is beyond the measureable. In short, the entrepreneurial element of human nature—which extends beyond the owners of enterprises—may be stifled by metric fixation.10 One result is to motivate those with greater initiative and enterprise to move out of mainstream, large-scale organizations where the culture of accountable performance prevails. Teachers move out of public schools to private schools and charter schools. Engineers move out of large corporations to boutique firms. Enterprising government employees become consultants. Ther...

Costs to productivity. Economists who specialize in measuring economic productivity report that in recent years the only increase in total factor productivity in the American economy has been in the information-technology-producing industries.11 A question that ought to be asked is to what extent the culture of metrics—with its costs in employee time, morale, and initiative, and its promotion of short-termism—has itself contributed to economic stagnation?

In medicine, Peter Pronovost’s Keystone project demonstrates how effective diagnostic metrics can be in lowering the incidence of medical errors, when what is measured accords with the professional values of practitioners. The success of the Geisinger medical system illustrates the remarkable improvements made possible by computerized measurement when integrated into an institutional culture based on cooperation, where the setting of measurement criteria and the evaluation of performance are done by teams that include physicians as well as administrators. In both cases, metrics were used in ways that appealed to intrinsic motivation and to professionalism. But elsewhere in the medical system, as we’ve seen, the use of reward for measured performance sometimes proved fruitless or led to perverse outcomes.

As we’ve seen time and again, measurement is not an alternative to judgment: measurement demands judgment: judgment about whether to measure, what to measure, how to evaluate the significance of what’s been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available.

1. What kind of information are you thinking of measuring? The more the object to be measured resembles inanimate matter, the more likely it is to be measureable: that is why measurement is indispensable in the natural sciences and in engineering. When the objects to be measured are influenced by the process of measurement, measurement becomes less reliable. Measurement becomes much less reliable the more its object is human activity, since the objects—people—are self-conscious, and are capable of reacting to the process of being measured. And if rewards and punishments are involved, they are more likely to react in a way that skews the measurement’s validity. By contrast, the more they agree with the goals of those rewards, the more likely they are to react in a way that enhances the measurement’s validity.

2. How useful is the information? Always begin by reminding yourself that the fact that some activity is measureable does not make it worth measuring, indeed, the ease of measuring may be inversely proportional to the significance of what is measured. To put it another way, ask yourself, is what you are measuring a proxy for what you really want to know? If the information is not very useful or not a good proxy for what you’re really aiming at, you’re probably better off not measuring it.

3. How useful are more metrics? Remember that measured performance, when useful, is more effective in identifying outliers, especially poor performers or true misconduct. It is likely to be less useful in distinguishing between those in the middle or near the top of the ladder of performance. Plus, the more you measure, the greater the likelihood that the marginal costs of measuring will exceed the benefits. So, the fact that metrics is helpful doesn’t mean that more metrics is more helpful.

4. What are the costs of not relying upon standardized measurement? Are there other sources of information about performance, based on the judgment and experience of clients, patients, or parents of students? In a school setting, for example, the degree to which parents request a particular teacher for their children is probably a useful indicator that the teacher is doing something right, whether or not the results show up on standardized tests. In the case of charities, it may be most useful to allow the beneficiaries to judge the results.

5. To what purposes will the measurement be put, or to put it another way, to whom will the information be made transparent? Here a key distinction is between data to be used for purposes of internal monitoring of performance by the practitioners themselves versus data to be used by external parties for reward and punishment. For example, is crime data being used to discover where the police ought to deploy more squad cars or to decide whether the precinct commander will get a promotion? Or is a surgical team using data to discover which procedures have worked best or are administrators using that same data to decide whether the hospital will be financially rewarded or penalized for its scores? Measurement instruments, such as tests, are invaluable, but they are most useful for internal analysis by practitioners rather than for external evaluation by public audiences who may fail to understand their limits. Such measurement can be used to inform practitioners of their performance relative to their peers, offering recognition to those who have excelled and offering assistance to those who have fallen behind. To the extent that they are used to determine continuing employment and pay, they will be subject to gaming the statistics or to outright fraud. Remember that, as we’ve seen, performance metrics that link reward and punishment may actually help reinforce intrinsic motivation when the goals to be rewarded accord with the professional goals of the practitioners.1 If, on ...

6. What are the costs of acquiring the metrics? Information is never free, and often it is expensive in ways that rarely occur to those who demand more of it. Collecting data, processing it, analyzing it—all of these take time, and their expense is in the opportunity costs of the time put into them. To put it another way, every moment you or your colleagues or employees are devoting to the production of metrics is time not devoted to the activities being measured. If you’re a data analyst, of course, producing metrics is your primary activity. For everyone else, it’s a distraction. So, even if the performance measurements are worth having, their worth may be less than the costs of obtaining them. Remember, too, that those costs in human time and effort are themselves almost impossible to calculate—another reason to err on the side of caution.

7. Ask why the people at the top of the organization are demanding performance metrics. As we’ve noted, the demand for performance measures sometimes flows from the ignorance of executives about the institutions they’ve been hired to manage, and that ignorance is often a result of parachuting into an organization with which one has little experience. Since experience and local knowledge matter, lean toward hiring from within. Even if there is someone smarter and more successful elsewhere, their lack of particular knowledge of your company, university, government agency, or other organization may not outweigh the benefits of hiring from within.

8. How and by whom are the measures of performance developed? Accountability metrics are less likely to be effective when they are imposed from above, using standardized formulas developed by those far from active engagement with the activity being measured. Measurements are more likely to be meaningful when they are developed from the bottom up, with input from teachers, nurses, and the cop on the beat. That means asking those with the tacit knowledge that comes from direct experience to provide suggestions about how to develop appropriate performance standards.2 Try to involve a representative group of those who will have a stake in the outcomes.3 In the best of cases, they should continue to be part of the process of evaluating the measured data. Remember that a system of measured performance will work to the extent that the people being measured believe in its worth. So far, in this chapter, we’ve taken the perspective of those in a position to decide whether and how to institute metrics. But what if you are not in such a position, if you’re further down in the organizational hierarchy, where you are expected to execute metrics—a mid-level manager, say, or the head of an academic department? Then, you face a choice. If you believe in the goals for which the information is being collected, then your challenge is to provide accurate data in the most efficient way possible, one that demands the least time of you and those you manage. If, by contrast, you believe that the...

9. Remember that even the best measures are subject to corruption or goal diversion. Insofar as individuals are agents out to maximize their own interests, there are inevitable drawbacks to all schemes of measured reward. If, as is currently still the case, doctors are remunerated based on the procedures they perform, that creates an incentive for them to perform too many procedures that have high costs but produce low benefits. But pay doctors based on the number of patients they see, and they have an incentive to see as many patients as possible, and to skimp on procedures that are time-consuming but potentially useful. Compensate them based on successful patient outcomes, and they are more likely to cream, avoiding the most problematic patients.5 That doesn’t mean that performance measures should be abandoned just because they have some negative outcomes. Such metrics may still be worth using, despite their anticipatable problems: it’s a matter of trade-offs. And that too is a matter of judgment.

10. Remember that sometimes, recognizing the limits of the possible is the beginning of wisdom. Not all problems are soluble, and even fewer are soluble by metrics. It’s not true that everything can be improved by measurement, or that everything that can be measured can be improved. Nor is making a problem more transparent necessarily a step to its solution. Transparency may make a troubling situation more salient, without making it more soluble.