More on this book
Community
Kindle Notes & Highlights
unlike most business books—which tend to be upbeat and chirpy, promising some new technique that will revolutionize one’s organization—this book is more critical in tone, promising less of a cure-all than an antidote to the snake oil sold under the labels of measurement, accountability and transparency.
under Congressional pressure for measurable results, and performance-based funding cycles, USAID was re-oriented toward the measurable rather than the important.
When numbers, standardized measurement of performance, and big data are seen as the wave of the future, professional judgment based upon experience and talent are seen as retrograde, almost anachronistic. Human judgment—based on talent and experience—has become unfashionable. Measurement is in.
Accountability ought to mean being held responsible for one’s actions. But by a sort of linguistic sleight of hand, accountability has come to mean demonstrating success through standardized measurement, as if only that which can be counted really counts.
Schemes of measured performance are deceptively attractive because they often “prove” themselves by spotting the most egregious cases of error or neglect, but are then applied to all cases. Tools appropriate for discovering real misconduct become tools for measuring all performance.
Deming on being in statistical control or outside of it
Concrete interests of power, money, and status are at stake. Metric fixation leads to a diversion of resources away from frontline producers toward managers, administrators, and those who gather and manipulate data.
That tension is sometimes necessary and desirable, for the professional ethos tends to discount issues of cost and opportunity cost. That is, the professional is inclined to see only the advantages of providing more of his or her services, without much attention to the limits of resources, or their alternate uses. Professionals don’t like to think about costs. Metrics folks do. When the two groups work together, the result can be greater satisfaction for both. When they are pitted against one another, the result is conflict and declining morale.
That is why I wrote this book. Little of what this book has to say is entirely new—it is based on synthesizing research and insights drawn from many other authors.
Professionals tend to resent the impositions of goals that may conflict with their vocational ethos and judgment, and thus morale is lowered.
In situations where there are no real feasible solutions to a problem, the gathering and publication of performance data serves as a form of virtue signaling. There is no real progress to show, but the effort demonstrated in gathering and publicizing the data satisfies a sense of moral earnestness. In lieu of real progress, the progress of measurement becomes a simulacrum of success.
Taylorism was based on trying to replace the implicit knowledge of the workmen with mass-production methods developed, planned, monitored, and controlled by managers. “Under scientific management,” he wrote, “the managers assume … the burden of gathering together all of the traditional knowledge which in the past has been possessed by the workmen and then of classifying, tabulating, and reducing this knowledge to rules, laws, formulae…. Thus all of the planning which under the old system was done by the workmen, must of necessity under the new system be done by management in accordance with
...more
The demand for measured accountability and transparency waxes as trust wanes.
numbers convey the air of objectivity; they imply the exclusion of subjective judgment.2 Numbers are regarded as “hard,” and thus a safer bet for those disposed to doubt their own judgments.
In a vicious circle, a lack of social trust leads to the apotheosis of metrics, and faith in metrics contributes to a declining reliance upon judgment. In a series of books, Philip K. Howard has argued that the decline of trust leads to a new mindset in which “[a]voiding human choice in public decisions is not just a theory … but a kind of theology…. Human choice is considered too dangerous.” As a consequence, “Officials no longer are allowed to act on their best judgment”4 or to exercise discretion, which is judgment about what the particular situation requires.5 The result is overregulation:
...more
When institutions are particularly large, complex, and made up of dissimilar parts, that comprehension is simply impossible. Those at the top face to a greater degree than most of us a cognitive constraint that confronts all of us: making decisions despite having limited time and ability to deal with information overload. Metrics are a tempting means of dealing with this “bounded rationality,” and engaging with matters beyond one’s comprehension.
Or empowering those at lower levels and providing enough context. Or maybe that is naive
The problem is that management’s quest to get a handle on a complex organization often leads to what Yves Morieux and Peter Tollman have dubbed “complicatedness”: the expansion of procedures for reporting and decision-making, requiring ever more coordination bodies, meetings, and report-writing. With all that time spent reporting, meeting, and coordinating, there is little time left for actual doing.
with an eye on their eventual exit to some better job with another organization, mobile managers are on the lookout for metrics of performance that can be deployed when the headhunter calls.
Because spreadsheets can do so many important things, those who use them tend to lose sight of the crucial fact that the imaginary businesses that they can create on their computers are just that—imaginary. You can’t really duplicate a business inside a computer, just aspects of a business. And since numbers are the strength of spreadsheets, the aspects that get emphasized are the ones easily embodied in numbers. Intangible factors aren’t so easily quantified.
Gibbons, a professor of organizational economics at MIT, pointed out that in fact the principal (the owner of the firm, for example) profits from a variety of outputs from the agent (the employee), and that many of these outputs are not highly visible or measureable in any numerical sense. Organizations depend on employees engaging in mentoring and in team work, for example, which are often at odds with what the employees would do if their only interests were to maximize their measured performance for purposes of compensation.
It does not have to be at odds if focusing on the long term
In Oakeshott’s famous example, there is the sort of abstract, recipe knowledge conveyed by cookbooks; but actually knowing how to make use of such knowledge (“beat an egg,” “whisk the mixture”) requires practical knowledge, based upon experience, that cannot be learned from books.
So for Standard Work in knowledge work it is also okay to describe the high level steps that need to be carried through, while the actual execution of these steps requires experience and tacit knowledge. Similarly to surgeon checklists.
The rationalist believes in the sovereignty of technique in which the only form of authentic knowledge is technical knowledge, for it alone satisfies the standard of certainty that marks real knowledge. The error of rationalism, for Oakeshott, is its failure to appreciate the necessity of practical knowledge and of knowledge of the peculiarity of circumstances.
By setting out in advance a limited and purportedly measurable set of goals, metric fixation truncates the range of actual goals of a business or organization. It also precludes entrepreneurship within organizations, as there may be new goals and purposes worth pursuing that are not part of the metric.
“To demand or preach mechanical precision, even in principle, in a field incapable of it is to be blind and to mislead others,” as the British liberal philosopher Isaiah Berlin noted in an essay on political judgment. Indeed what Berlin says of political judgment applies more broadly: judgment is a sort of skill at grasping the unique particularities of a situation, and it entails a talent for synthesis rather than analysis, “a capacity for taking in the total pattern of a human situation, of the way in which things hang together.”7 A feel for the whole and a sense for the unique are precisely
...more
Once we become fixated on measurement, we easily slip into believing that more is better.
The search for more data means more data managers, more bureaucracy, more expensive software systems. Ironically, in the name of controlling costs, expenditures wax.
The numbers gathered from citation databases may be of some use in that process, but numbers too require judgment grounded in experience to evaluate their worth.
Rankings create incentives for universities to become more like what the rankings measure. What gets measured is what gets attention. That leads to homogenization as they abandon their distinctive missions and become more like their competitors.
What the advocates of greater government accountability metrics overlook is that the very real problem of the increasing costs of college and university education is due in part to the expanding cadres of administrators, many of whom are required in order to comply with government mandates.
The culprit was presumed to be a lack of professionalism among public school teachers.
If you're blaming people, especially a large group of people, then it's clear that big systems problems are being overlooked
Under the NCLB act, enacted early in Bush’s presidency, states were to test every student in grades 3–8 each year in math, reading, and science. The act was meant to bring all students to “academic proficiency” by 2014, and to ensure that each group of students—including blacks and Hispanics, who were singled out for comparative evaluation—within each school made “adequate yearly progress” toward proficiency each year. It imposed an escalating series of penalties and sanctions for schools in which the designated groups of students did not make adequate progress.
If you already know that there is a problem, which you must know to start a program like this, then adding measurements and forcing numerical improvements doesn't really do anything. Real leadership is identifying highest leverage points of intervention and acting on them.
Asians tended to outscore whites, who in turn tended to outscore blacks and Hispanics. Most salient was the ongoing deficiency of African American students. Eight years after the introduction of NCLB, their relative scores had not changed.
Is the average the best indicator? What about the variability and the whole distribution?
The unintended consequences of NCLB’s testing-and-accountability regime are more tangible, and exemplify many of the characteristic pitfalls of metric fixation.
National education is undoubtedly a complex system. It should only be changed through experiments and the scaling up and down of the same. Yet in politics the best looking fully fledged plan is what wins but can never be truly effective because reality is too complex to comprehend to that degree
value-added tests work best when they are “low stakes.”10 It is the emphasis placed on these tests as the major criterion for evaluating schools that creates perverse incentives, including focusing on the tests themselves at the expense of the broader goals of the institution.
Tying performance metrics to salary vs using them as part of the signal
Such outcomes might lead one to conclude that the achievement gap cannot in fact be closed by education—and that the reasons lie beyond the schoolhouse door. Yet measuring continues unabated. That is perhaps because, as Banfield noted, the idea that some problems are insoluble is morally unacceptable to a substantial portion of educated Americans.27 When it comes to gaps in school achievement, it seems that in the absence of discernable progress in results, the resources devoted to ongoing measurement becomes itself a sign of moral earnestness.
The emphasis on measuring the achievement gap and the pressure to close it has other troubling effects. One is the blame heaped upon teachers and schools for their failure to accomplish what may be beyond their reach, and for reasons that have little to do with their own limitations. The logic of NCLB, “Race to the Top,” and similar programs, places the responsibility for closing achievement gaps on those who may have neither the power nor the ability to do so.
Through slogans and exhortations. Not through management
Metrics at Geisinger are effective because of the way in which they are embedded in a larger system. Crucially, the establishment of measurement criteria and the evaluation of performance are done by teams that include physicians as well as administrators. The metrics of performance, therefore, are neither imposed nor evaluated from above by administrators devoid of firsthand knowledge. They are based on collaboration and peer review.
To be sure, there are some real advantages to publicly available metrics of surgeon success and of hospital mortality rates. Their publication can point out very poor performers, who may then cease practicing, in the case of surgeons—a sifting process all the more valuable in a profession in which practitioners are reluctant to dismiss incompetent fellow members of the guild. Or the lower-level performers can take steps to improve their measured performance, in the case of hospitals.
Should only be acted on in cases outside of Deming's statistical control
Attaining the goal of reduced admissions depends not only on the steps that the hospital takes to educate the patient and provide necessary medications, but also on many factors over which the hospital has little control: the patient’s underlying physical and mental health, social support system, and behavior.
Punishing and rewarding for factors outside of the agent's or group's control.
metrics tend to be most successful for those interventions and outcomes that are almost entirely controlled by and within the organization’s medical system, as in the case of checklists of procedures to minimize central line–induced infections. When the outcomes are dependent upon more wide-ranging factors (such as patient behavior outside the doctor’s office and the hospital), they become more difficult to attribute to the efforts or failures of the medical system.
Such problems preceded the rise of Compstat and exist independent of it. In 1976 the social psychologist Donald T. Campbell (of Campbell’s Law, see chapter 1) noted that President Richard Nixon’s declared crackdown on crime “had as its main effect the corruption of crime-rate indicators, achieved through underrecording and downgrading the crimes to less serious classifications.”
As a detective in the narcotics division, Burns sought to meticulously build a case against top drug lords. But his superiors were uninterested in that prospect, which was consuming manpower and would take years to produce an arrest. They were interested in enhancing the metrics, and since arresting five teenagers a day selling drugs on street corners yielded better statistics than arresting a drug king-pin after a multiyear investigation, they favored the course that quickly produced the higher numbers. From their point of view—and from the point of view of the politicians to whom they
...more
Interpretation of indicators is critically important, and requires informed expert judgment. It is not enough merely to count incidents or conduct quantitative or statistical analysis—interpretation is a qualitative activity based on familiarity with the environment, and it needs to be conducted by experienced personnel who have worked in that environment for long enough to detect trends by comparison with previous conditions.
These concerns apply well beyond the military realm: to the extent that we try to develop performance metrics for any complex environment or organization that is either unique or substantially different from other environments or organizations, standardized measures of performance will be inaccurate and deceptive. Yet the desire to create performance metrics that are “transparent” in the interests of “accountability” usually translates into using metrics that are standardized and centralized, since such metrics are more easily grasped by superiors and by publics far from the field of
...more
People do want to be rewarded for their performance, both in terms of recognition and remuneration. But there is a difference between promotions (and raises) based on a range of qualities, and direct remuneration based on measured quantities of output. For most workers, contributions to their company include many activities that are intangible but no less real: coming up with new ideas and better ways to do things, exchanging ideas and resources with colleagues, engaging in teamwork, mentoring subordinates, relating to suppliers or customers, and more. It’s appropriate to reward such
...more
Increasing numbers of technology companies, conscious of the demotivating effect of performance rankings on the majority of their staff, are moving away from performance bonuses. They are replacing them with higher base salaries combined with shares or share options, to give employees a tangible interest in the long-term flourishing of the company (while paying special rewards to particularly high performers).
reliance upon measurable metrics is conducive to short-termism, a besetting malady of contemporary American corporations.
When rewards such as pay, bonuses, and promotions are tied to meeting budget targets, there is yet another danger: distorting the information system of the organization. Managers and employees learn to lie, to massage, embellish, or disguise the numbers that are used to calculate their pay. But since these are the very numbers that executives use to coordinate the activities of the organization and decide on the allocation of future resources, the productivity and efficiency of the organization is damaged as resources are misallocated.
Gary Pisano and Willy Shih have argued, Most companies are wedded to highly analytical methods for evaluating investment opportunities. Still, it remains enormously hard to assess long-term R&D programs with quantitative techniques…. Usually, the data, or even reasonable estimates, are simply not available. Nonetheless, all too often these tools become the ultimate arbiter of what gets funded and what does not. So short-term projects with more predictable outcomes beat out the long-term investments needed to replenish technical and operating capabilities.
Performance indicators can certainly aid, but not replace, the key functions of management: thinking ahead, judging, and deciding.
For most charities, equating low overhead with higher productivity is not only deceptive but downright counterproductive. In order to be successful, charitable organizations need competent, trained staff. They need adequate computer and information systems. They need functional offices. And yes, the ability to keep raising funds. But the assumption that the effectiveness of charities is inversely proportional to their overhead expenses leads to underspending on overhead and the degradation of organizational capacities:
