The Tyranny of Metrics by Jerry Z. Muller

Nov 06, 2018 BlackOxford rated it did not like it

Shelves: science, measurement, technology

Ayn Rand Lives

This is an important book about an important subject. It’s primary importance lies in the fact that it is completely wrong about what’s at issue and how to fix it. It is so wrong that it makes the case for its antithesis. This too is wrong, if only slightly less 0bviously so.

Here’s the thesis: “We live in the age of measured accountability, of reward for measured performance, and belief in the virtues of publicizing those metrics through ‘transparency.’ But the identification of Ayn Rand Lives

This is an important book about an important subject. It’s primary importance lies in the fact that it is completely wrong about what’s at issue and how to fix it. It is so wrong that it makes the case for its antithesis. This too is wrong, if only slightly less 0bviously so.

Here’s the thesis: “We live in the age of measured accountability, of reward for measured performance, and belief in the virtues of publicizing those metrics through ‘transparency.’ But the identification of accountability with metrics and with transparency is deceptive. Accountability ought to mean being held responsible for one’s actions. But by a sort of linguistic sleight of hand, accountability has come to mean demonstrating success through standardized measurement,... The most characteristic feature of metric fixation is the aspiration to replace judgment based on experience with standardized measurement.”

Here’s the problem: what Muller means by standardized metrics is what some group of people - managers, politicians, scientists - have agreed among themselves as to what constitutes ‘success’. The entire book is then devoted to examples of how this sort of agreement results in stupid actions and consequences. My experience, like his, is that many metrics are misconceived, obsolete, counter-productive, and well... stupid. But rather than investigate how to improve the way these metrics are arrived at by any group, Muller plumps for replacing them with a vague concept of ‘individual experience.’ But what is the content of this individual experience? It can only be another more or less (usually less) articulate metric, some different criterion of correct choice, or it’s nothing but fantasy. And if this criterion has not been accepted by the group as superior to the existing ‘standardized’ criterion, how has ‘accountability’ been served?

An answer might be that the decision on the criterion used by an individual in defiance of the established metric is the precise action for which he or she is to be held accountable. Is the metric of individual experience better than the established metric? This would be quite reasonable - disobeying orders for a good reason should be acceptable when the orders are obviously destructive, illegal, or have unexpected collateral effects. But this answer would require some elaboration about the process by which an individual’s experience, when in conflict with established criteria of decision, can be reconciled with, incorporated into, or modify the established view of correct action. A military courts-martial for example is usually not convened to determine whether insubordination has taken place but whether it is justified in the circumstances.

But Muller doesn’t provide the least hint how the experience of an individual, or new experiences at all, should be used to modify the ‘standing orders’ of established metrics. He seems to believe that there are real but inherently inarticulate criteria living in the nervous systems of decision-makers which, if forced into some level of operational literacy, would lose their real import. He also believes that in general terms these inarticulate instincts are superior to any metric formulated by a group. If the individual is in conflict with the group, the group is simply wrong. Whatever else he has to say about accountability is therefore nonsense. There is none. Disagreement about ‘what counts’ is down to some sort of genetically-instilled spiritual preferences (or perhaps hedonistic utility) that can’t be questioned lest it inhibit the exercise of individual sovereignty.

I understand Muller’s frustration with the idiotic metrics which are employed in business and social policy. I have my own thesaurus of war stories about how such metrics have destroyed organisations and damaged individuals. And Muller is fighting against a real intellectual enemy - the idea that measurement of anything is an activity devoid of political judgment. This ideology, and it is just that, is the antithesis to Muller’s thesis. It seeks to establish that what we measure, principally in our choice of a metric of success, as some objective property of the thing measured. This is the foundation stone of scientism which the belief that individuals must submit to the rationality of reality, a reality determined by a consensus of experts. These experts may include a diverse set of skills - accountancy, economics, sociology, management or finance for example - but their aim is the same: to impose a standard, universal metric of success wherever they ply their influence. And Muller is right to resist this sort of intellectual totalitarianism.

But he is not right to substitute the arbitrariness of some group of experts with the arbitrariness of some individual’s experience. This is just another ideology. It strikes me that Muller takes the ideas of economic neo-liberalism to their philosophical outer limits. Margret Thatcher and Ronald Reagan didn’t believe that society existed; only individuals interacting did. Muller extends this to the realm of practical ideas from moral calculus to commercial decision-making. Discussion, mutual discovery, agreement and group learning are not things he can even consider in his ideological cage. How conflicts about what is important and therefore what should be measured in business, politics, and any cooperative effort are simply not his concern. What he calls ‘metric fixation’ is no different from ‘social awareness’. I am quite sure that Ayn Rand is alive and well in Muller’s garden shed.

Postscript: Muller implicitly presumes that social groups are fictional and should be given no status within his ideology of measurement. The opposing ideology, that only social conventions have meaning in measurement, is also prevalent. See here for a typical example: https://www.goodreads.com/review/show.... There are interesting philosophical implications of both views. For example ‘measurement socialists’ believe that the measurements they make are objective properties of the thing measured. While ‘measurement individualists’ like Muller believe that what is measured is entirely in the head of the one doing the measuring. Debates between the two groups are rarely edifying. For an alternative theory which avoids both the Scylla of collectivism and the Charybdis of individualism in measurement, see here: https://www.goodreads.com/review/show... ...more

flag 71 likes · Like · see review

May 15, 2018 Jan Rice rated it it was amazing

Shelves: politics, history, philosophy, morality-justice

As I was reading the midsection of this book, I was thinking it was a chore--where "chore" means taking some notes--but an absolutely necessary one, given the revelatory nature. Then, the ending hit me with an emotional wallop. That could not have happened if I hadn't read on through the material.

The wallop was this: we have made an idol out of transparency. And this: our whole society is like a bad marriage. We don't know how to fix it, but we can't just divorce everybody.

Before I read this ...more

flag 25 likes · Like · see review

Apr 18, 2018 Mal Warwick rated it it was amazing

Shelves: nonfiction

I was not surprised by historian Jerry Z. Muller's comments about "metric fixation" in his illuminating new book, The Tyranny of Metrics.

Some years ago the chairman and CEO of a Fortune 500 company remarked to me that nobody, not even Jack Welch, the then-idolized leader of General Electric, could possibly turn in solid revenue and profit increases steadily, quarter-after-quarter, year-after-year, through ups and downs in the market. It was clear to him that somebody was cooking the books at GE. I was not surprised by historian Jerry Z. Muller's comments about "metric fixation" in his illuminating new book, The Tyranny of Metrics.

Some years ago the chairman and CEO of a Fortune 500 company remarked to me that nobody, not even Jack Welch, the then-idolized leader of General Electric, could possibly turn in solid revenue and profit increases steadily, quarter-after-quarter, year-after-year, through ups and downs in the market. It was clear to him that somebody was cooking the books at GE. I had been in business long enough then to understand how easy it is to shift sales and revenue from one quarter or year to the next and how other steps could be taken to fudge the numbers. (I was also aware of Welch's reputation as "Neutron Jack" for his ruthless practice of "downsizing" to increase profits, a practice that can also be timed to give the appearance of steadily increasing profits.) It had been clear to me for many years that numbers can lie.

In his new book, Muller tackles our society's obsession with metrics and accountability. However, "[t]his book is not about the evils of measuring," he writes. "It is about the unintended consequences of trying to substitute standardized measures of performance for personal judgment based on experience. The problem is not measurement, but excessive measurement and inappropriate measurement—not metrics but metric fixation." In just 200 pages, Muller assesses the use and misuse of metrics through case studies drawn from a wide range of fields: colleges and universities, K-12 education, medicine, policing, the military, business and finance, and philanthropy and foreign aid. The research he cites, and the examples he chooses, are compelling.

I've had my own intimate experience with the misuse of metrics in both philanthropy and foreign aid. More than thirty years ago, when I was running a consulting agency that raised money for nonprofit organizations, I wrote and spoke to whoever would listen about the absurdity of measuring nonprofit performance on the basis of its fundraising costs. Years later, when I became involved in consulting with NGOs in developing countries, I saw for myself the folly of the metrics obsession that had seized hold of the international development community after Bill Gates began proselytizing on the subject. That fixation on the numbers forced far too many charities and government agencies to funnel money toward easily measured but trivial or even irrelevant programs while ignoring others that might actually have some positive impact in the field.

Muller sums up the problem nicely. "There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive." Amen.

Some of the examples Muller cites are familiar to the public at large. "Teaching to the test," for example. The tendency of some surgeons to decline to operate in difficult cases because failure would lower their success ratios. And the tendency of police under pressure from politicians to make the numbers look better by classifying felonies as misdemeanors or altogether refusing to write up crimes. These are just a few of the many sad ways that our obsession with accountability distorts our understanding of the world around us.

Among the many unintended consequences of the misuse of metrics that Muller cites are the following:

** Inducing people whose performance is measured to divert their efforts to what gets measured;

** Promoting short-termism (as in Wall Street's obsessive preoccupation with quarterly earnings reports at the expense of companies' long-term health);

** Discouraging innovation and risk-taking;

** Sidetracking nonprofit staff members (or corporate employees, for that matter) from focusing on the mission that motivates them; and

** Forcing employees to spend time logging data instead of doing their jobs (a requirement that was a major factor in convincing my brother to close his psychiatric practice many years ago).

Muller concludes The Tyranny of Metrics with a useful checklist of ten questions that any manager should ask when considering the application of metrics at work. ...more

flag 16 likes · Like · see review

Jul 25, 2018 Tom Mackay rated it it was ok

A sustained critique of what Muller calls "metric fixation" is necessary and long-overdue. Muller captures how the overuse and dependency on metrics and quantification, particularly for evaluating performance, is hollowing out and damaging key social, cultural, political, commercial, and philanthropic institutions. Spelling this out in a succinct and accessible manner for an audience broader than academia is vital. For that, Muller should be commended.

However, the book is not as hard-hitting as A sustained critique of what Muller calls "metric fixation" is necessary and long-overdue. Muller captures how the overuse and dependency on metrics and quantification, particularly for evaluating performance, is hollowing out and damaging key social, cultural, political, commercial, and philanthropic institutions. Spelling this out in a succinct and accessible manner for an audience broader than academia is vital. For that, Muller should be commended.

However, the book is not as hard-hitting as it needs to be, it reads as a rushed effort, and its broader analysis is superficial, at least in terms of its contextualisation. Worse, it overlooks or downplays key structural and ideological factors. Nowhere, for instance, is neoliberalism mentioned or really even alluded to. Muller is right to highlight that there's no single cause, but some causes are more significant and more influential than others. Economic transformations and the social and political dominance of the fiscal right should be given much more weight than they are. Moreover, the critique of transparency is wrongheaded and the examples used are lazy. It is here that Muller's ideological convictions start to become abundantly clear. The problem, according to Muller, isn't neoliberal capitalism but an obsession with transparency coming from both the left and the right. This isn't just an aside - this is the foundation from which the entire argument is built.

So while the book nicely captures some of the consequences of "metric fixation", it fails to adequately account for its underlying causes. True, Muller suggests that we can redress our overuse of metrics and offers the reader a guide on how and when to use them appropriately. But without pointing readers to the broader structures that are actually encouraging this fixation in the first place, the tyranny will persist. Again, a sustained critique is needed more than ever and it is great that work has begun. Regardless, we need much more than a conservative exposition of a problem stemming from conservatism.

For more critical insights, read Mark Fisher's Capitalist Realism, especially Chapter Six ('All that is solid melts into PR') and Annie McClanahan's critique of credit scoring in chapter two of Dead Pledges: Debt, Crisis, and Twenty-First Century Culture. ...more

flag 7 likes · Like · see review

May 23, 2018 Mehrsa rated it liked it

It's good for what it is and I think it's right, but there is nothing new here. Read Cathy O'Neills Weapons of Math Destruction, which is better and more interesting.

flag 5 likes · Like · see review

Jan 20, 2018 David Wineberg rated it really liked it

Too many metrics are killing productivity

During the great famine in China, local officials yanked plants out of the ground and raced ahead of Mao’s itinerary to plant them at his next stop, thus fooling the chairman into thinking bumper crops were everywhere. They kept their jobs and their heads, but as a result, Mao declared there was no famine and refused to release government stores of rice to the starving. Fifty million died. This is the poster child for metrics.

The Tyranny of Metrics is an Too many metrics are killing productivity

During the great famine in China, local officials yanked plants out of the ground and raced ahead of Mao’s itinerary to plant them at his next stop, thus fooling the chairman into thinking bumper crops were everywhere. They kept their jobs and their heads, but as a result, Mao declared there was no famine and refused to release government stores of rice to the starving. Fifty million died. This is the poster child for metrics.

The Tyranny of Metrics is an anecdote-filled examination of how industry, services and government have gone overboard with self-defeating metrics. Muller comes at it from a personal angle as well as a piece of research. He is a history professor, bogged down in the mindless world of forms and stats. It is crippling his own research and networking time, and does nothing to further his field, his career or his own satisfaction.

He says there are three components to the metrics fixation:
-replacing judgment with numerical values
-publicizing numbers to make institutions transparent and accountable
-managing people is to give them numerical targets and evaluations
Ironically, despite all this measurement, productivity is lagging. Make that because of.

This is the Soviet system, where the central authority set goals for factories, services and bureaucracies, and in which workers find ways to achieve those goals at the cost of quality, service, risk-taking and innovation. Examples are students who can pass tests but have little knowledge, subprime mortgages on false applications, police who hide crimes, and banks that open phony accounts for existing customers to meet quotas. Even the federal Government Accounting Office (GAO) got renamed Government Accountability Office, (which managed to prevent confusion by recycling its initials). There is no better way to see this in action than Ken Loach’s Palme D’Or-winning film I Daniel Blake, which shows the UK’s performance-obsessed unemployment office in action. The objective is to spend as little time as possible with customers, and refuse as many as possible for the slightest misstep.

Muller proves his points with perverse outcomes galore. Hospitals keep dying patients alive for 30 days because that’s the measure of treatment success. Surgeons decline to operate on iffy cases for fear of them dying too soon. Creative teachers quit and move to private schools where teaching to the test and abandoning real learning are not the main activities. Universities cripple research in favor of reporting on every aspect of education, right up to how much each graduate is earning ten years later. All their efforts are focused on moving up the rankings in the various, competing lists. The cheating by teachers, administrators and politicians on No Child Left Behind are legendary.

Muller ends with ten points to ponder when you find yourself in an evaluation situation. A simple rule of thumb, he says, is that if the object of your measurements can be by altered by the process of measurement, the results can be less than accurate.

A 2006 survey of HR managers found that metrics and rankings “resulted in lower productivity, inequity, skepticism, decreased employee engagement, reduced collaboration, damage to morale and mistrust in leadership.” So more and more firms do it every year.

David Wineberg
...more

flag 4 likes · Like · see review

May 26, 2018 Rhys rated it liked it

The Tyranny of Metrics is a clear expression of mainly the deleterious effects of using metrics in the workplace - where counting shouldn't be counted on.

"Dan Cable and Freek Vermeulen of the London Business School recall many of the problems we have explored: the depressive effect of performance pay on creativity; the propensity to cook the books; the inevitable imperfections of the measurement instruments; the difficulty of defining long-term performance; and the tendency for extrinsic ...more

flag 4 likes · Like · see review

May 18, 2018 Peter Geyer rated it it was amazing

Shelves: owned, organisations, psychology, society, culture, education, politics, economics

The title of this book suggests that it could be a polemic of sorts against measurement, more specifically "metrics" – something dismissed early in the piece by the author. Muller presents the idea, backed up by research and case studies, that the use of metrics to guide management of everything from management to education, the military to charities often achieves the opposite of its stated aim.

He begins with his own experience as a dean in a university faculty and the growing pressures to The title of this book suggests that it could be a polemic of sorts against measurement, more specifically "metrics" – something dismissed early in the piece by the author. Muller presents the idea, backed up by research and case studies, that the use of metrics to guide management of everything from management to education, the military to charities often achieves the opposite of its stated aim.

He begins with his own experience as a dean in a university faculty and the growing pressures to justify everything according to numbers and algorithms to such an extend that the actual purpose of the faculty – educating people – gets lost in an avalanche of form-filling and spreadsheets, results in the employment of more and more administrative staff to cater for the numerical information generated, assesses academic staff according to criteria that have little if anything to do with scholarship, sanctioning or firing those that don't meet these standards.

He notes that these processes also result in the system being gamed, so that employees meet the standards and so their jobs. Examples are given from several sectors: hospitals where surgeons to the easy cases, not the more difficult where someone might die or reclassifying outpatients and the like, as well as various strategies in the military and education as well as in business.

In the local news this morning, there's a piece about how employees of a bank which had a scheme to have primary school children put money in those accounts (their own or the bank's) which had no funds in them 30 days after being opened. They did this because otherwise they would not receive a particular reward related to the bank's incentive system. The bank found out about this and sanctioned no-one, nor did they notify the regulators. A currently-running Royal Commission into the finance industry has repeatedly exposed all sorts of gaming the syatem in this way.

A presumption of this approach is that people can only be extrinsically motivated i.e. by money and that intrinsic motivation, such as that you would hope to see in education, medicine, government and the military, as well as financial institutions. If you've been unemployed in Australia or the UK in particular, then this method has been applied to you, and those you have to consult, including outsourced "employment" entities, or the privatised education system where training is being offered.

Muller's book has a continuing theme of pay for performance, which some here want to apply to teachers as a motivator, and he demonstrates that this idea really achieves the opposite, from low-level jobs to CEOs and senior executives, because the process takes those engaged in it away from the aims of the organisation or their profession. I must admit that this doesn't surprise me at all,and is painfully obvious, from the time I encountered it in my workplace in the late 1980s (and no, it didn't apply to me: I was just asked to work longer hours without being paid).

As an historian, Muller provides background to this idea, which has come and gone over the years under the rubric of "scientific management" a downside of which included the decline of interest in the work put in front of you, because the individual, their knowledge, experience and expertise isn't valued as input or perhaps in principle.

Muller is an excellent writer, particularly in the first part, which he describes as more theoretical, the second part being given over to examples from various fields. It doesn't take long to read, and because it's not a polemic (he does provide some pages of remedial suggestions) the prose is also matter-of-fact and so largely unobtrusive. I'd never heard of him before, and I'm tempted to look for his earlier books, to see what I can find out from them and have a relaxing read as well.
...more

flag 3 likes · Like · see review

Apr 03, 2018 Matt Chester rated it really liked it

I really enjoyed Muller's 'The Tyranny of Metrics,' and felt it delivered exactly the type of read I thought it would. In a world that is becoming more digitized and people are pushing for increased transparency, Muller writes about instances where it appears we've become too reliant on blindly following the data behind metrics without understanding the harm that may come.

To be clear, Muller is not against metrics but he lays out very good arguments for why they may not always be appropriate, I really enjoyed Muller's 'The Tyranny of Metrics,' and felt it delivered exactly the type of read I thought it would. In a world that is becoming more digitized and people are pushing for increased transparency, Muller writes about instances where it appears we've become too reliant on blindly following the data behind metrics without understanding the harm that may come.

To be clear, Muller is not against metrics but he lays out very good arguments for why they may not always be appropriate, how they get gamed or manipulated, and why they are useless unless they are able to be judged subjectively by people with intimate knowledge of the field and where the shortcoming of those metrics might be. Stretching across areas like schools, police departments, militaries, governments, hospitals, charities and more, Muller gives great examples that are directly applicable and always thought provoking. If anything, I do wish there were a bit more meat on the bones of some of his examples with expanded discussion-- but perhaps there will be a follow up!

The only part that seemed a bit off was towards the end when he spent a short chapter equating modern society's growing need for transparency and metrics to the need for politicians, governments etc. to exercise more privacy and reduce transparency. The argument he was making was about how if every piece of email of a public official is made public, then they will be less likely to speak to their colleagues/opponents with candor and less able to reach the optimal compromises because they will have to worry about these negotiations being made public. While some of his arguments felt valid, it made me squirm in my seat a bit the more he pushed for the right of government to operate with reduced transparency.

All in all, though, I really enjoyed this book and would recommend it to anyone who lives in the world of data and metrics like I do. Muller provides a good mirror with which to look at how you treat metrics and consider whether you need to incorporate more of the 'art' into the 'science' of data and metrics. ...more

flag 3 likes · Like · see review

Feb 17, 2018 Matthew Trevithick rated it it was amazing

4 stars, but one more for novelty. At times conservative and at times liberal, a hard look at what the obsession with metrics is doing to modern life, across education, business, military, government, foreign aid, and more. The author makes a highly persuasive case using succinct arguments to deconstruct what the downsides are to the obsession with numbers. And it’s quite alarming.

flag 3 likes · Like · see review

Jun 14, 2019 Daniel rated it it was amazing

Oh, the metrics you will use! Mueller pointed out the problems of over-using metrics in society, especially when it is used for compensation purposes, and when assigned top-down without consultation with people who are actually involved in the work.

1. What is measurable may not be important, and what is important may not be measurable.
2. Metrics focus people’s attention to what is measured, to the neglect of everything that is not.
3. Metrics used for internal reporting is generally fine, but ...more

flag 2 likes · Like · see review

Oct 15, 2019 Lorenzo Barberis Canonico rated it it was amazing

I would consider myself fairly bullish on metrics, tracking and organization in general, so I took the time to listen to Muller’s counter-argument. Most of it boils down to Goodheart’s Law ("When a measure becomes a target, it ceases to be a good measure”). What really surprised me and I deeply appreciated was his final section arguing against transparency, which is very much a core value I strongly believe in. He makes a very persuasive case of the vital role privacy makes in our lives, as well ...more

flag 2 likes · Like · see review

Mar 29, 2018 Reid rated it liked it

Shelves: non-fiction

If your work is anything like mine, the bosses have become acolytes of metrics, the pseudo-science of numerically evaluating everything that everyone does as a way of tracking and improving service. On the surface, this makes a great deal of sense. How can you know that you have employed a mediocre teacher or an incompetent nurse without some data to demonstrate their ineffectiveness? How can you compare one hospital or college with another without knowing what their outcomes are? But, as anyone If your work is anything like mine, the bosses have become acolytes of metrics, the pseudo-science of numerically evaluating everything that everyone does as a way of tracking and improving service. On the surface, this makes a great deal of sense. How can you know that you have employed a mediocre teacher or an incompetent nurse without some data to demonstrate their ineffectiveness? How can you compare one hospital or college with another without knowing what their outcomes are? But, as anyone who has ever been subject to the blandishments of metric-driven management, things are not always as they seem.

Let me give you one excellent example. There are national statistics published yearly about hospitals. I work in a hospital with mortality and morbidity statistics that are appalling in comparison to most of the other hospitals in the area. We must be doing something wrong, right? Far from it. Because we are a Level One trauma center, we take on the most challenging of cases, the sickest of the sick and the most broken of the injured. More of them die; more of them are very ill. Now, the statisticians claim to take into account all of these factors with risk scores, yet we still do very poorly on these scales. A patient looking at those statistics without any background information will think that they must avoid our hospital like the plague, when it could well be that we may be just the place they need to be. (Fortunately, very few patients actually do this, but that's another story).

I am a middle manager and at every meeting a table of metrics is marched out that purports to measure just how well we are doing our jobs. Fortunately, I do not work in the kind of place that rewards good metrics with pay differentials or much other than praise, but even this publication of statistics (in the name of transparency) is misleading, demoralizing, and shaming. As the author of this fine book points out, those are some of the unintended consequences of a dependence on metrics.

The central thesis of this book is one with which I wholeheartedly agree: that which can be measured may not be the most important indicator of quality work. And (the corollary), those things which are most important may not be amenable to measurement. I can give you another excellent example from my own experience. Our clinic does very poorly when it comes to female patients who get their mammograms done. We encourage the providers to order them, even present them with report cards on how well they are doing (more metrics), yet the numbers rarely budge. As we dug a little deeper, it became evident that there were some serious systemic problems preventing some women from getting mammograms, particularly those at or around the poverty line. Our institution has no mammogram machine, and the most commonly recommended place to get this done does not accept the kind of insurance that most of these women have; transportation to a location that does accept their insurance is prohibitively expensive. And our clinic cares for more women who fit into this low income category than others. In other words, we can refer to our heart's content, but these mammograms are never going to get done until we pull down the institutional barriers to care. But none of this shows up in the metrics.

Now, I will admit (as does the author) that some professions respond well to metric measures of their work. In the financial sector, there really is no other valid way to evaluate someone's work other than the amount of money they bring in (though he also points out that this does not take into account the mentoring and teaching in which every true professional should engage). In professions where there is little in the way of personal motivation other than money, metrics may also be quite useful. But these professions are the exception rather than the rule.

One of the distinctions that Muller astutely points out is that between intrinsic and extrinsic motivation. In those jobs taken on solely for pay, extrinsic motivation (money, threat of demotion or termination) may be the sole reason a person comes to work at all. But for most of us, there is a great deal of intrinsic motivation; the feeling of being of use, pride in doing something well, appreciation of our company and its mission. For many professionals (professors and nurses come to mind), because there are many struggles when practicing those professions, there is almost always a great deal of intrinsic motivation (otherwise we would go do something else). The reason this distinction is so important is because the use of metrics can be so deeply demoralizing, reducing one's profession to a series of numbers while completely ignoring what is truly meaningful to the employee.

There are many other very important aspects of this craze for metrics in this book and I encourage you to give it a read if you are being judged on the basis of them. It does feel as if this is a once over lightly treatment of the topic; though he has many sections, several of them are only a few pages long and do not even begin to speak to all the implications of the use of metrics. Still, as an introduction to the hazards and pitfalls, as well as the legitimate uses of metrics, it is a fine read. ...more

flag 2 likes · Like · see review

Feb 05, 2018 Darren rated it it was amazing

Metrics is a word that can convey many feelings and emotions. It can be the means of beneficial analysis and quantification, as well as being something one is a slave to, recording unrealistic or unrequired data ‘just because’. For many, it is no longer just measuring the performance of something but the measurement of measuring itself that is in focus in the real world.

This book considers, therefore, the real tyranny of metrics and the obsession that many hold with this management elixir and ...more

flag 3 likes · Like · see review

Apr 14, 2018 Pete rated it really liked it

Shelves: nonfiction, economics

The Tyranny of Metrics (2018) by Jerry Z Muller is an interesting look at problems caused by misusing metrics. It's impressively short and Muller has read widely and pondered the problems caused by over relying on poor metrics.

Muller outlines why metrics have been used. It looks at increasing costs and people wanting, wisely, to improve productivity. Metrics were also seen as a way of resolving the principal / agent problem. They were also seen as a way of doing something objective to assess ...more

flag 2 likes · Like · see review

Aug 31, 2018 D rated it liked it

Very good introduction to some of the flaws of metrics. If you are already familiar with some of the literature - or are wary of metrics - this probably isn’t new information, but it is all put together and summarized very nicely. I wished he had engaged with the anthropology and STS work on this topic.

flag 2 likes · Like · see review

Sep 08, 2018 Lada rated it really liked it

"The point isn't to survey the field but to highlight a few case studies." Would have been cooler if metrics were brought down with really solid stats, but "judgement" wielded by the author was interesting too.

flag 2 likes · Like · see review

Jul 07, 2018 Hidde rated it really liked it

Muller explains the dangers of trusting metrics for anything, with examples from government, business and charities. It’s a good reminder of what could go wrong, and shows that judgment based on knowledge and experience trumps judgment based on metrics.

flag 2 likes · Like · see review

Sep 28, 2018 Kevin B. rated it really liked it

In the Tyranny of Metrics, Jerry Muller presents a clear case against the current, and growing, over-reliance on KPIs and other performance metrics for assessing people's and organizations' work. One of the more common admonitions in AI is to be careful what you ask for, or, as Russell & Norvig put it in their incredibly popular textbook on AI, "what you ask for is what you get." If your institutions set KPIs rewarding citations, for example, then you're likely to end up with illicit In the Tyranny of Metrics, Jerry Muller presents a clear case against the current, and growing, over-reliance on KPIs and other performance metrics for assessing people's and organizations' work. One of the more common admonitions in AI is to be careful what you ask for, or, as Russell & Norvig put it in their incredibly popular textbook on AI, "what you ask for is what you get." If your institutions set KPIs rewarding citations, for example, then you're likely to end up with illicit citation rings, with pals citing each other pointlessly -- pointless except for the KPI reward, of course. KPIs have a very strong tendency to replace the real objectives of an organization -- education, good governance, public health, common welfare -- with much lower quality ersatz objectives -- student popularity, volume of memoranda, quick hospital discharges, numbers of arrests for petty crimes.

Another common problem is using absolute metrics when only relativized metrics make sense. For example, insisting that education funding reward schools whose students perform better on standardized tests can make for a good sound-bite, but has always been understood to produce "teaching to the test" -- that is, a narrow educational focus that leads to ex-students poorly prepared to deal with a complex world with wide-ranging problems. But importantly it also leads to schools taking a low-risk approach to their education. Instead of seeking out and supporting students with disabilities or minority socioeconomic backgrounds, they will narrow their admissions to those already likely to perform well on standardized tests. That can really pay off for their budgets, but it's also letting society down. The major blame should fall on the politicians who force the performance metrics on the schools in the first place. Instead of an absolute test-performance standard, a relativized standard, comparing outcome performance to initial performance, would eliminate that particular distortion of educational goals. (While doing nothing about teaching to the test, of course, which requires some other response.)

Muller reviews many of the ways in which metrics can go wrong, and he specifically considers them in some of the more socially important domains in which they do go wrong: education (secondary and tertiary), policing, medicine, finance, the military, foreign aid. His book is an excellent starting point for thinking about the general subject of work performance measurement or its particular consideration in any of these domains.

I can't give Muller five stars here, however. One area in which he goes seriously astray is the issue of transparency in our institutions. Muller quite rightly points out that in many processes we require confidentiality and that the demand for transparency may well have gone too far. In diplomacy, privacy or secrecy can be essential to achieving a reasonable outcome. Diplomacy requires compromise, and compromise requires giving away something that you'd prefer not to. If the thing given way is made public too soon, then the diplomatic transaction may well implode before any compromise can be agreed. People have secrets for good reason. Similarly, the demand for honesty in politicians (generally an unsuccessful demand to be sure, but commonly loudly made in the media nonetheless) can turn into a fetish, where politicians who legitimately change their minds are excoriated for having no position or those who bend the truth to achieve a greater good are pummelled for dishonesty. The whole point of politics is for our politicians to achieve greater goods, not lesser goods, so these kinds of admonition by Muller are well taken.

Yet Muller goes far too far in this. Wikileaks' revelation of war crimes in Iraq was accompanied by the revelation of identities of intelligence agents around the world. Julian Assange didn't care about the safety of those agents or the future ability of, say, the CIA to recruit other agents. That's a failure of over-zealous transparency, for sure. But Muller gives no credit at all to the other side. For example, he fails to accept that the whistleblowing revelation of the war crimes itself was a good thing and should be legally protected. He fails to acknowledge any benefit from Freedom of Information laws. Worse still, he castigates Edward Snowden for revealing many of the secrets of the NSA. While the United States has the right to have a "no such agency", it doesn't have the right to have spying and anti-encryption programs of the depth and breadth of the NSA. What Snowden exposed was, and is, criminal activity endangering democracy (see, e.g., https://www.bbc.com/news/uk-45510662). I think it absurd that Muller can't put in even one word of support for such transparency. I wish for treatments of these subjects that show better balance and judgment than Muller's. ...more

flag 1 like · Like · see review

Mar 23, 2019 SocProf rated it liked it · review of another edition

Shelves: non-fiction, higher-ed

The topic is important. It is addressed rather well and covers pretty much what everyone working on those sectors has experienced. But this book could also have been written in the 1980s. There are a lot of books written in the same critical vein that take into account the rise of algorithms as part of the same processes (see Cathy O'Neill, obviously, but others as well).

But again, those of us who have experienced those developments will appreciate their treatment here.

flag 1 like · Like · see review

Nov 04, 2018 Jesse rated it liked it

The book has some interesting case studies on the misuse of metrics, but the entire argument starts to feel fairly belabored. At the end there's a chapter that's tacked on, meant to inform us on how we can use metrics in an effective way. It lacked depth and I was left wanting a bit more guidance (you'll want the guidance, after feeling that tracking any metric will lead to your or your business' untimely demise).

flag 1 like · Like · see review

Jan 30, 2019 Prabs rated it really liked it

Good Point of view, but the tone is very negative.
Conclusion is good enough to read, which also has adequate examples to understand the book.

flag 1 like · Like · see review

May 01, 2019 Alexandra rated it it was ok

found it difficult to focus on.

metrics can be gamed and once known and the basis for preferential treatment most certainly will be.

flag 1 like · Like · see review

Oct 11, 2018 Marc Gerstein rated it it was amazing · review of another edition

Shelves: non-fiction

An incredibly important book that does much to illuminate many of the major and often under-understood dysfunctions of modern life.

We live, today, in a world in which measurement, goals, data, tracking, etc. dominate more and more facets of life, from the traditional (counting profits and losses) to areas further and further afield than what one thought would be amenable to this sort of thing (doctor and patient, sports, etc.). The ideas were good, often noble. But not everything that sounds ...more

flag 1 like · Like · see review

Oct 10, 2018 Suddenven rated it liked it

This book discusses the pitfalls of metric fixation/obsession

flag 1 like · Like · see review

Oct 08, 2018 Erik Th rated it really liked it

Great read! Except the part on the explosion on education (and of BAs), which is a bit too American and might also miss that more educated people does not just mean decrease in salaries for qualifications, but actually strengthen a society.
Central point is how throughput qualities become output.

flag 1 like · Like · see review

Feb 15, 2018 Dave rated it really liked it

Shelves: biz

Once metrics drive promotions, bonuses, and firings, they will be gamed. Just because something can be measured doesn't mean that should be. For profit organization have one goal - profit, non-profits including government agencies have many, they shouldn't be run like a business. This is Muller's argument in a nutshell of a nutshell.

He gives his argument in the first chapter and then expands and supports through out the rest of the books.

It's been my experience that once a metric become a ...more

flag 1 like · Like · see review

Apr 22, 2018 Ed Barks rated it really liked it

Numbers have come to dominate business thinking. Everything needs to be measured. This wrong-headed thinking blinds executives from trusting their own expertise and experience and leads to poor decision making.

Muller sets forth an excellent argument regarding the unwise use of relying exclusively on metrics, noting:

• (W)hat can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than

...more

flag 1 like · Like · see review

Nov 25, 2018 Lee rated it really liked it

A great little book that serves as a counterpoint to the social obsession with metrics. Muller does not call for discarding all metrics, he just suggests that we need to think more seriously about how to use metrics. His advice is to use metrics to help bulster the judgement of experts rather than to replace them.

This book had a few flaws. He suggests that we, as a society, are too credulous in believing people with a professional interest in maintaining the metritocracy when they say, "the ...more

flag 1 like · Like · see review

Jun 03, 2018 Sarah Glen rated it it was ok

The important point of this book is simple, and yet frequently elusive in real life: "Measurement demands judgment."

While I enjoyed some of the case studies, especially those about the founding of accountability metrics in education, Muller could've used a stronger edit. He waxes profusely, frequently restating the same point across 4-5 sentences. My advice: Read the Quartz write-up rather than the whole book.

A few other quotes I enjoyed:

"There are things that can be measured. There are things ...more

flag 1 like · Like · see review