The Tyranny of Metrics

by Jerry Z. Muller

For potential employers, degrees act as signals: they serve as a shorthand that allows employers to rank initial applicants for a job. Having completed high school signals a certain, modest level of intellectual competence as well as personality traits such as persistence. Finishing college is a signal of a somewhat higher level of each of these. In a society where a small minority successfully completes college, having a B.A. signals a certain measure of superiority. But the higher the percentage of people with a B.A., the lower its value as a sorting device. What happens instead is that jobs that once required only a high school diploma now require a B.A. That is not because the jobs have become more cognitively demanding or require a higher level of skill, but because employers can afford to choose from among the many applicants who hold a B.A., while excluding the rest. The result is both to depress the wages of those who lack a college degree, and to place many college graduates in jobs that don’t actually make use of the substance of their college education.4 That leads to a positional arms race: as word spreads that a college diploma is the entry ticket to even modest jobs, more and more people seek degrees.

A consequence of students entering college without the ability to do college-level work is the ever larger number of students who enroll but do not complete their degrees—a widespread and growing phenomenon that has substantial costs for the students who do so, in tuition, living expenses, and earnings foregone.10 High dropout rates seem to indicate that too many students are attempting college, not too few.11 And those who do obtain degrees find that a generic B.A. is of diminishing economic value, because it signals less and less to potential employers about real ability and achievement.12 Recognizing this, prospective college students and their parents seek admission not just to any college, but to a highly ranked one.

Reward for measured performance in higher education is touted by its boosters as making universities “more like a business.” But businesses have a built-in restraint on devoting too much time and money to measurement—at some point, it cuts into profits. Ironically, since universities and other nonprofit institutions have no such bottom line, government or accrediting agencies or the university’s administrative leadership can extend metrics endlessly.25 The effect is to increase costs or to divert spending from the doers to the administrators—which usually suits the latter just fine. It is hard to find a university where the ratio of administrators to professors and of administrators to students has not risen astronomically in recent decades.

The hazard of metrics so purely focused on monetary return on investment is that like so many metrics, they influence behavior. Already, universities at the very top of the rankings send a huge portion of their graduates into investment banking, consulting, and high-end law firms—all highly lucrative pursuits.48 These are honorable professions, but is it really in the best interests of the nation to encourage the best and the brightest to choose these careers? One predictable effect of the weight attributed to future income in college rankings will be to incentivize institutions to channel their students into the most high-paying fields. Those whose graduates go on to careers in less remunerative fields, such as teaching or public service, will be penalized.49 A capitalist society depends for its flourishing on a variety of institutions that provide a counterweight to the market, with its focus on monetary gain. To prepare pupils and university students for their roles as citizens, as friends, as spouses, and above all to equip them for a life of intellectual richness—those are among the proper roles of college. Conveying marketable skills is a proper role as well. But to subordinate higher education entirely to the capacity for future earnings is to measure with a very crooked yardstick.

It is no wonder, then, that teachers (encouraged by their principals) divert class time toward the subjects tested—mathematic and English—and away from other subjects, such as history, social studies, art, music, and physical education. Instruction in math and English is narrowly focused on the sorts of skills required by the test, rather than broader cognitive processes: that is, students too often learn test-taking strategies rather than substantive knowledge. As depicted in the HBO series The Wire, a great deal of class time is devoted to practicing for tests—hardly a source of stimulation for pupils. Because students in English are taught to answer multiple choice and short-answer questions based on brief passages, the students are worse at reading extended texts and writing extended essays—much as Mathew Arnold had predicted a century and a half earlier.

But value-added tests work best when they are “low stakes.”10 It is the emphasis placed on these tests as the major criterion for evaluating schools that creates perverse incentives, including focusing on the tests themselves at the expense of the broader goals of the institution.

Yet it is striking that after decades of gathering and publicizing these metrics, the outcome has remained more or less unchanged. The positions of blacks and Hispanics relative to whites are remarkably stable. While there have been some minor fluctuations when students are measured in grades 4 and 8, there is almost no change in the ultimate result—the metrics in grade 12, that is, at the end of high school.

General improvements in schooling do not therefore lead to greater equality of outcomes. As the political scientist Edward Banfield noted a generation ago, “All education favors the middle- and upper-class child, because to be middle- or upper-class is to have qualities that make one particularly educable.” Improvements in the quality of schools may elevate overall educational outcomes, but they tend to increase, rather than diminish, the gap in achievement between children from families with different levels of human capital.26 Such outcomes might lead one to conclude that the achievement gap cannot in fact be closed by education—and that the reasons lie beyond the schoolhouse door. Yet measuring continues unabated. That is perhaps because, as Banfield noted, the idea that some problems are insoluble is morally unacceptable to a substantial portion of educated Americans.27 When it comes to gaps in school achievement, it seems that in the absence of discernable progress in results, the resources devoted to ongoing measurement becomes itself a sign of moral earnestness.

there is much more to school than the learning of English and mathematics: not only other academic subjects but also the stimulation of interest in the world, and the cultivation of habits of behavior (self-control, perseverance, ability to cooperate with others) that increase the likelihood of success in the adult world. Development of these noncognitive qualities may well be going on in classrooms and schools without being reflected in performance metrics based on test scores.

though exposing students to better teachers may lead to gains in academic achievement, those gains tend to fade away over time. The noncognitive gains, however, appear to persist.30 Character development matters—which has led some legislatures to try to incorporate measurement of character into their accountability systems!

The costs of trying to use metrics to turn schools into gap-closing factories are therefore not only monetary. The broader mission of schools to instruct in history and in civics is neglected as attention is focused on attempting to improve the reading and math scores of lower-performing groups. Pedagogic strategies that may be effective for lower-achieving students (such as longer school days and shorter summer vacations) are extended to students for whom these strategies are counterproductive. And resources are diverted away from maximizing learning on the part of the more gifted and talented—who may in fact hold the key to national economic performance.32 The emphasis on measuring the achievement gap and the pressure to close it has other troubling effects. One is the blame heaped upon teachers and schools for their failure to accomplish what may be beyond their reach, and for reasons that have little to do with their own limitations. The logic of NCLB, “Race to the Top,” and similar programs, places the responsibility for closing achievement gaps on those who may have neither the power nor the ability to do so. That itself is a recipe for the demoralization of teachers. Add to that the dilemma presented to teachers: pursuing the multiple aims of education versus teaching to the test; following their broad vocational mission versus adhering to the narrow criteria upon which they are to be remunerated. Whichever course they choose, they lose. In addition, many teachers p...

is the notion of narrowly self-interested agents a fact of life, or is it exacerbated by a managerial ideology that uses extrinsic rewards based upon simple models of human behavior that then become self-fulfilling prophecies? Sometimes, the way in which managers and employees are addressed by their company actually influences the way they think, so that they come to act in the narrowly self-interested way posited by the most reductive versions of principal-agent theory, with deceit and guile.13 In fact, it may create a situation in which the managers and employees most knowledgeable about the workings of the performance indicators are best positioned to manipulate those indicators for their own benefit, and most likely to do so.

A focus on measurable performance indicators can lead managers to neglect tasks for which no clear measures of performance are available, as the organizational scholars Nelson Repenning and Rebecca Henderson have recently noted.25 Unable to count intangible assets such as reputation, employee satisfaction, motivation, loyalty, trust, and cooperation, those enamored of performance metrics squeeze assets in the short term at the expense of long-term consequences. For all these reasons, reliance upon measurable metrics is conducive to short-termism, a besetting malady of contemporary American corporations.

Performance metrics as a measure of accountability help to allocate blame when things go badly, but do little to encourage success,28 especially when success requires imagination, innovation, and risk. Indeed, as the economist Frank Knight noted almost a century ago, entrepreneurship entails “immeasurable uncertainty,” which is not susceptible to metric calculation.29 Thus, even in business and finance, metric fixation takes its toll. Businesses must be judged by more than one indicator of performance. Profit surely matters. But so, in the long run, does reputation, market share, customer satisfaction, and employee morale, which makes it possible to adapt and to find solutions to the new problems that will inevitably arise in the marketplace. In an economic world characterized by unpredictable change, there is a need for ongoing innovation, small and large, that is not readily reducible to a single performance target. Performance indicators can certainly aid, but not replace, the key functions of management: thinking ahead, judging, and deciding.

For most charities, equating low overhead with higher productivity is not only deceptive but downright counterproductive. In order to be successful, charitable organizations need competent, trained staff. They need adequate computer and information systems. They need functional offices. And yes, the ability to keep raising funds. But the assumption that the effectiveness of charities is inversely proportional to their overhead expenses leads to underspending on overhead and the degradation of organizational capacities: instead of high-quality and well-trained staff, too many novices and too much staff turnover; computer systems that are out of date and inefficient; and as a result, less effectiveness in raising funds for ongoing activities or new programs. To make matters worse, the funders impose growing demands for reports, so that staff time devoted to documentation eats up an ever larger portion of the grant. In response, the leaders of charitable organizations often end up trying to game the figures: by reporting that the time of leading staff members is devoted almost entirely to programs, or that there is no spending on fundraising. That response is understandable. But it feeds the expectations of funders that low overhead is the measure they should be looking at to hold charities accountable.1 Thus the snake of accountability eats its own tail.

And it is characteristic of our culture that we tend to assume that performance and transparency rise and fall together. But that is a fallacy, or at least a misleading generalization. For just as there are limits to the efficacy of measured performance, there are limits to the efficacy of transparency. In some cases, how well our institutions perform depends on not making them transparent. At issue here is not the question of metrics, but of performance in the broadest sense: success in what we’re supposed to be doing. To appreciate the dark side of transparency, let us begin not with organizations but with interpersonal relations.

As the contemporary philosopher Moshe Halbertal puts it, If a person’s thoughts were written on his forehead, exposed before all, the distinction between interior and exterior would vanish, and with it also individuation. Privacy, expressed through the possibility of concealment, thus protects the very ability of a person to define himself as an individual. Furthermore, the self may create special relationships by displaying differential measures of exposure and intimacy. He moves through social space by allotting revelation and concealment and establishing differential measures of distance and closeness.

That is why on sensitive matters, the negotiating process is most effective when it takes place behind closed doors. As Tom Daschle, the Democratic former majority leader of the Senate, has recently observed, the “idea that Washington would work better if there were TV cameras monitoring every conversation gets it exactly wrong…. The lack of opportunities for honest dialogue and creative give-and-take lies at the root of today’s dysfunction.”2 That is also why effective politicians must to some degree be two-faced, pursuing more flexibility in closed negotiations than in their public advocacy. Only when multiple compromises have been made and a deal has been reached can it be subjected to public scrutiny, that is, made transparent.

We need to distinguish between those elements of government that ought to be made public and those that should not be. Cass R. Sunstein, a wide-ranging academic who has also served in government, makes a useful distinction between government inputs and outputs. Outputs include data that the government produces on social and economic trends, as well as the results of government actions, such as regulatory rules. Outputs, he argues, ought to be made as publicly accessible as possible. Inputs, by contrast, are the discussions that go into government decision-making: discussions between policymakers and civil servants. There are increasing pressures to make those publicly available as well: whether through legal means such as Freedom of Information Act requests; or congressional demands, as in the case of congressional committees demanding the email correspondence of former Secretary of State Hillary Clinton in the case of the Benghazi investigations; or illegal means such as the electronic theft and dissemination of internal government documents by organizations such as Wikileaks. Making internal deliberations open to public disclosure—that is, transparent—is counterproductive, Sunstein argues, since if government officials know that all of their ideas and positions may be made public, it inhibits openness, candor, and trust in communications. The predictable result will be for government officials to commit ever less information to writing, either in print or in the form of ...

A degree of legitimate concealment is necessary to maintain the state and its democratic institutions. Military secrets, techniques for fighting crime, intelligence gathering, and even diplomatic negotiations that will fall apart if they become exposed—all these domains have to stay shrouded in secrecy in order to allow the functioning of ordinary transparency in the other institutions of the state. Our transparent open conversation rests upon a rather extensive dark and hidden domain that insures its flourishing.

Goal displacement through diversion of effort to what gets measured. Goal displacement comes in many varieties. When performance is judged by a few measures, and the stakes are high (keeping one’s job, getting a raise, raising the stock price at the time that stock options are vested), people will focus on satisfying those measures—often at the expense of other, more important organizational goals that are not measured.1 Economists Bengt Holmström and Paul Milgrom have described it in more formal terms as a problem of misaligned incentives: workers who are rewarded for the accomplishment of measurable tasks reduce the effort devoted to other tasks.2 The result is that the metric means comes to replace the organizational ends that those means ought to serve. Promoting short-termism. Measured performance encourages what Robert K. Merton called “the imperious immediacy of interests … where the actor’s paramount concern with the foreseen immediate consequences excludes consideration of further or other consequences.”3 In short, advancing short-term goals at the expense of long-range considerations. Costs in employee time. To the debit side of the ledger must also be added the transactional costs of metrics: the expenditure of employee time by those tasked with compiling and processing the metrics—not to speak of the time required to actually read them. That is exacerbated by the “reporting imperative”—the perceived need to constantly generate information, even when nothing sig...