The Efficiency Paradox: What Big Data Can't Do

by Edward Tenner

A "skillful and lucid" ( The Wall Street Journal ) way of thinking about efficiency, challenging our obsession with it—and offering a new understanding of how to benefit from the powerful potential of serendipity.

Algorithms, multitasking, the sharing economy, life our culture can't get enough of efficiency. One of the great promises of the Internet and big data revolutions is the idea that we can improve the processes and routines of our work and personal lives to get more done in less time than we ever have before. There is no doubt that we're performing at higher levels and moving at unprecedented speed, but what if we're headed in the wrong direction?

Melding the long-term history of technology with the latest headlines and findings of computer science and social science, The Efficiency Paradox questions our ingrained assumptions about efficiency, persuasively showing how relying on the algorithms of digital platforms can in fact lead to wasted efforts, missed opportunities, and, above all, an inability to break out of established patterns. Edward Tenner reveals what we and our institutions, when equipped with an astute combination of artificial intelligence and trained intuition, can learn from the random and unexpected.

Genres: Nonfiction, Business, Technology, Science, Artificial Intelligence, Philosophy, Economics

320 pages, Paperback

First published January 1, 2018

About the author

Edward Tenner is the author of Our Own Devices and Why Things Bite Back, former college teacher and executive editor in book publishing, now an independent writer and speaker on technology and society and contributor to major newspapers, magazines, and web sites.

Reviews

[Mehrsa · Rating 3 out of 5]

Why did I even finish this book? I have no idea. A compulsion? I mean, I get the main point, which is that some things were better the old fashioned way (taking notes, reading, doctors, driving, etc), but there's just no real data or new information or even a new idea. All of this and more has been remarked upon by others in a variety of different ways.

[Crystal · Rating 4 out of 5]

Non-Fiction about different industries and how technology has changed them. Is it all for the better? 4 stars--It's good and I would recommend it, but it's not for non-fiction phobes because it's a little like a series of mildly interesting lectures.

Tenner goes through the factory, academia, cartography, media and medicine to explain a brief history of the industries and how they have been transformed by technology over the years. There were some fun terms that fellow nerds will appreciate:
desirable difficulty When something is too easy and requires little to no effort to process then we don't learn anything. If notetaking or navigating requires the individual to process the information required for the task then the individual learns something. If you just type out a lecture verbatim on a laptop or follow Siri on a trip then you don't really learn the skills you need to understand things.
productive boredom Having idle time to tinker or contemplate or even discuss things with other humans can lead to some great things. As we attempt to fill every second of our day with personalized entertainment the serendipitous conversations and revelations are fewer.
waylosing Ever been on a road trip and just gone down the wrong road to find something worthwhile and you're SO GLAD you made that wrong turn? Not that GPS and map programs eliminate this entirely, but going directly from point A to point B without any off-the-cuff detours means no more surprises. Those surprises are fun sometimes!

This is not meant to be a very strong argument against technology...it's more of a call to understand the context and intention of technology that we use. It's great if we are being more efficient but ask, "Is this really more efficient?" "Is there anything that I'm losing by doing it this way?" "Should efficiency be my goal in this case?" Definitely not for Luddites, but more for people who can say they're better at researching things today because they had to learn to use a library reference system in school.

There are lots of references for further reading on any topic he discusses. I really enjoyed the brief histories of the various industries. Sometimes a 'tech' book can seem out of date the moment it's published, but this seemed relevant to me.

"The best font is the one that is unnoticed."
"A book is not just an object, but a terrain."
"Professionalism and populism rarely coexist."

[Eustacia Tan · Rating 3 out of 5]

I knew I had to read this book once I saw it because the title is too good to resist. Plus everyone is generally so positive about big data I was quite interested to see some criticism of it.

First, a definition of efficiency. The author defines it as “producing goods, providing services or information, or processing transactions with a minimum of waste”. The book basically goes through the history of the idea of efficiency and then goes on to discuss specific examples: the internet and democratisation of information, teaching, GPS, and medicine.

The way I see it, the entire argument can be summed into: automating things can be inefficient because innovation requires serendipity (which algorithms cannot provide). In other words, innovation requires inefficiency, i.e. ‘wasted’ or failed ideas.

To be honest, I’m not entirely convinced by some of the arguments. I can see how innovation may require ‘waste’, but for specific examples like GPS, it may be that we don’t have enough information to for the algorithms to be useful. Same for teaching - the bigger the database, the easier it is for lessons to be customised for the student.

And for some things, like medicine, I feel that the argument took the wrong direction: the author argues that automated note-taking doesn’t make things more efficient as doctors still spend a lot of time on the computer, but the author doesn’t discuss how the shared information may make the process more efficient if the patient moves between departments. So I felt this was more of a misdirected argument (and not that relevant to the central premise).

Another example would be the argument about ‘waylosing’ (finding something unexpected when you get lost). I agree that it does have benefits, but I don’t think that ‘the inefficient wanderer, on the other hand, will be using his or her time more efficiently by discovering what is less documented, or even undocumented.’ That really depends on your aim in travelling - to learn something new or to get from point A to point B.

Overall, I thought that the book was interesting, but it didn’t really fulfil the ‘promise’ of telling me the limitations of big data. Instead, the arguments in the book talk more about how we’re misusing the platform innovations, which is interesting but not quite the point. Perhaps these are the ‘limitations’ that the subtitle was talking about, but there really should be a clearer line between the limitations and the purpose of each technology.

This review was first posted at Inside the mind of a Bibliophile

[Nick · Rating 1 out of 5]

By the time I finally reached chapter two I closed the book and gave up on it. It is a book without merit that I cannot recommend.

As someone who works in tech, I am continually led towards ideas of techno-optimism and solutionism and hyper-productivity and I don't want to be led down a path erroneously, so I wanted to read this book to get a differing perspective and see if I can learn something.

Unfortunately, you will not get anything of the sort from this book.

The first chapter is the author complaining about financialization, but then uses that to argue that we're not innovating anymore which is... deeply wrong? COVID vaccines, CRISPR, solar prices, and many other examples of modern innovation come to mind immediately.

The second chapter is the author calling us stupid and inefficient because sometimes we need to go to the second page of Google results, which is insulting at best and fundamentally flawed at worst. All through this book, the author seems to have a fetishization for efficiency while lambasting our attempts at dealing with it. Google's attempts at efficiency are bad, because efficiency is bad, but their attempts are bad because they're not efficient enough?

That's a summary of the first two chapters, and I'm fairly deep into the book. The author seems to write a lot of prose unnecessarily (inefficiently) diluting their point with a lot of flowery language. I just can't see myself getting anything else from this book, so I've stopped reading it at that point.

Perhaps there's a glimmer of irony here, that I stopped reading a book against efficiency because it's not efficient enough in making its points. And yet the author seems to be mistaken in the entire enterprise of efficiency and the benefit of material gains. We moved from family farms to industrial ones, providing us with an abundance of produce.

Why am I supposed to grind through hundreds of pages of prose for a nugget of wisdom? Isn't that nugget the point? Maybe something the length of a tweet isn't enough, but if your goal is to convince me of your philosophy or impart some wisdom, it should be done in a way that vibes with the reader.

[Evan Crane · Rating 2 out of 5]

Unfocused

[Adelyne · Rating 5 out of 5]

This was a book I enjoyed: one that I pre-ordered a long time ago and read almost as soon as it was released. Tenner presents various examples of how overuse of big data can harm, instead of promote, efficiency and makes a strong case for properly distinguishing between superficial and true efficiency. While this varies on a case-by-case or industry-by-industry basis, he argues that many traditional metrics of efficiency may in fact promote the converse. A good read to start off the month with, I read this in a slightly over-fragmented way for proper understanding and this is really one which is best read in several big chunks (at least a chapter at a time). Considering a re-read, maybe after I’ve had some time to think things over, also I'm reading another book (Martin Ford's Rise of the Robots) which makes the converse argument that can be depressing at times so might need to revisit this one afterwards. 4.5 stars, 5 on GoodReads.

[Azure · Rating 2 out of 5]

Didn't really learn anything new here, coming from an analytical background adjusting to healthcare. Very meh overall.

[Chris Esposo · Rating 1 out of 5]

The disappointing under-developed counterpoint to the promise of the IT/Web 2.0 revolution, and to the information explosion in general. Though all the points mentioned, more or less variants of the law of unexpected consequences, there's not much holding the book together.

Each chapter felt like a separate submitted chapter on various domains impacted by the information and web 2.0 revolution, from newspapers, education, to various automatons like those applied to drive. But nothing is much said besides, people were promised A, but B1, B2,..., Bn occurred, which maybe "approximate" A in some sense, some subsequence of which was undesirable. A well-read reader would have been exposed to 90%+ of these use-cases also, making this book blase.

Also, it's not really about big data, one would have expected a detractor of "big data" to make an argument on say how big data does not yield causal models by default, and so the way the analysis is leveraged is erroneous. Or maybe make an argument that "big data" does nothing more but add power to something that could be discerned with small data. I've heard something like those two statements made in other books. This book doesn't ever go remotely deep into the mechanism.

Thus, making this book a discursive collection of domains of impact, with no unified theme or insight into what may threat these use cases together. Save your money and read some back issues of the Atlantic or Pacific Standard instead. Not recommended, except for individuals who are very very fresh to this topic.

[Ronald J. · Rating 2 out of 5]

“Efficiency is mostly good, until it isn’t. Even an excess of water can be lethal.” I agree with this, and indeed, have written about the difference between efficiency and effectiveness in my work and books. He defines efficiency as “producing goods, providing services or information, or processing transactions with a minimum of waste.” Fine. I say it’s a ratio of outputs divided by inputs. Either way, it’s a formula. But not every that matters can be measured. As Napa’s winemakers like to say, “It’s easier to count the bottles than to describe the wine.” Plus, there’s no such thing as generic efficiency. It depends on our objectives and what we are willing to pay, which is why we splurge on the Golden Gate Bridge and the Sydney Opera House—not very efficient, but highly effective for the human soul. The quest for constant efficiency does attempt to make the world more predictable, but innovation is the antithesis of efficiency. Efficiency’s contribution to economic growth is dwarfed by innovation and dynamism that is unleashed by free markets.

But this book isn’t what I thought it would be, which makes any review I write seem terribly biased. There were interesting things, but it just didn’t do much for me.

[Eric · Rating 2 out of 5]

An interesting 300 page problem description

This book contains a lot of well researched salient examples, but never really builds to anything other than its initial observation - that out blind pursuit of efficiency in a localized context is paradoxically inefficient. It was an interesting read, but very dense, overly verbose, and shallow in recommending a path forward for the presented problem.

[Santosh Shevade · Rating 2 out of 5]

A mix of pop science, history of various domains and peppered with anecdotal thoughts on efficiency and tech..not much about big data, not much about a consistent thought process of what's efficiency...overall a disappointment

[Ross · Rating 1 out of 5]

I got through 10% of this book without learning a single thing about an efficiency paradox, and simply had to give up.

[Spencer Fuentes · Rating 3 out of 5]

I had conflicted feelings reading The Efficiency Paradox. I am not typically a nonfiction reader, however the topic seemed interesting enough. However, when I began to read this book, I found that the topics and viewpoints in it were -- quite frankly -- boring. The main focus of the book works in the idea that mass efficiency causes it's own issues and shortcomings, but fails to recognize the massive positives of these innovations. Furthermore, the argument isn't made concisely, rather every single point being drawn out needlessly without effectively addressing the topic.
When discussing different viewpoints, or addressing claims which go against the author's argument, they were mostly just shrugged off or disregarded without much detailed reasoning as to why. Other times, however, some arguments would be stated as if they were facts despite them being straight- up wrong, or misjudged to fit the narrative and the author's arguments.
This leads me into the biggest issue I found about the book. It seems to lack any sort of goal chapter to chapter. While the overarching argument is very clear, it could have been argued without the pages upon pages of useless filler and pointless complication. I seems to attempt to be more sophisticated than is needs to be, with paragraphs upon paragraphs filled with information that isn't relevant to any sort of argument made within the chapter.
My favorite quote within this novel is. "Efficiency is mostly good, until is isn't. Even an excess of water can be lethal.", as it, in a way, represents my experience reading the book. The author's argument is well supported, however, there is a point in which there is an unnecessary amount of information that diminishes the effectiveness of what they are trying to say. The needless filler is what made reading this book rather unenjoyable.
The author begins with a preface defining what he believes to be the characteristics of efficiency and what makes it "evil". He continues with a history of efficiency in factories, discussing the hyperefficiency of modern factories when compared to the founding principles of efficiency. From there, he discusses the information explosion, and how trying to measure the science of efficiency lead to the targeted online culture of major companies. Afterwards, he discusses what he believes to be the failures of learning and how efficiency has caused a decrease of ease for learners. He continues with a discussion of efficiency in travel, discussing the use of geographic data and it's hinderance to efficiency. Finally, he discusses what he calls the managed body - the effects of efficiency on human behaviour. His conclusion to the novel is that the only way to truly be efficient is to find a balance of efficiency and inefficiency.
This is a theme that I do not, generally, disagree with. However, it is the ways that the author chose to support this theme that I find to be full of holes and fallacies. It is a given that pure robotic efficiency would be ineffective in any real scenario, however the given ideas that the author discusses to support this are misconstrued at best and not true at worst.

[Mindo'ermatter · Rating 4 out of 5]

When Everything's Important, Nothing Is!

This serious book begins an essential discussion. It is not a fun read but a critically insightful one from a big-picture perspective, but with excruciating detail. Enlightening but not entertaining, written as my father would describe as a book for a straight-back chair.

The author advocates that increased efficiency usually creates unexpected inefficiencies and often destructive errors. Good ideas usually include many bad or unanticipated outcomes, with overplanned standardization and consistency preventing innovation and creativity. Tenner challenges technological and efficiency extremes that ignore human involvement and supervision, but recognizes the benefits of a balanced hybrid approach.

The book is also an excellent reference source for multitudes of real-world examples that support his arguments. This is a serious read that should generate broader discussions and more critical observations of current and widely accepted practices. It is worthwhile for both academics and organizational leaders, while also serving well as a wake-up call to everyone else. Just because something is possible does not mean it is wise.

Tenner's well-documented presentation and honest analysis are both eye opening and thought provoking. The preface alone is worthy of close attention by raising several critical issues, questions, and counterintuitive conflicts.

The author's preface and conclusions surround five themed chapters that explore the myths and realities of technology applications and their unintentional inefficiencies. Two chapters focused on education (learning) and healthcare, where the best intentions for quality and quantity improvements through big data and technology proved problematic by creating more problems and reduced effectiveness.

The well-researched supporting references include many direct links to original sources (active hyperlinks in the Kindle version). These citations occasionally include the author's unique observations or explanations, but typically include multiple or gunny sack sources, making them less efficient for the interested researcher to identify direct attribution, something the author likely intended to encourage broader exploration and analysis of differing but related perspectives, to encourage discovery and serendipity learning through "planned" inefficiencies.

I found the Audible narration extremely valuable to leverage my reading experience.

There's much in the book to unpack and consider, so I plan to further review its many arguments and wise observations.

[Ajay · Rating 1 out of 5]

Shallow, Inefficient, and Disappointing - The Efficiency Paradox attempts to traverse an incredible range of topics ranging from the industrial revolution, teaching in the digital age, and questions of what is progress; however each chapter seems to have been written in isolation, with little tying the book together into a coherent work. The end result is superficial at best - I would not recommend.

The title and description are also misleading - this book has very little to do with technology, computing, and big data and far more to do with the author's conservative world view (his snide remarks about "the Left" can be found throughout the book with little relevance).

This is made even more disappointing because the central premise - that the pursuit of optimal efficiency paradoxically makes us less efficient in the long run - is quite interesting. However, there is not coherent argument here - only fluid and vague definitions and no real data or new information to support or even flesh out the premise. Outside of this book, I've encountered valid arguments for the limitations of efficiency as the central metric and the drawbacks of optimizing for it to the exclusion of all else - but this book fails to discuss it's own essence.

[Kes · Rating 3 out of 5]

The logistic snafus brought on by Covid is probably a good way to describe this book: from a company's perspective, it's good to be efficient (keeping low stock, and minimising stock kept in storage); but when something happens (Covid / logistical issues), having some inefficiency / redundancy (having stock allowing you to maintain sales/production) is probably for the better.

The author deals with this efficiency paradox in several different areas. Chapter 1 deals with technology (factory work to platform technologies in electronic network); Chapter 2 expands on Chapter 1 (but focusing on analog media / technologies / search abilities and allowing for serendipity); Chapter 3 on education; Chapter 4 on geography (basically: being able to navigate without GPS); Chapter 5 on medicine (such as AI in diagnosing people and fitness trackers). The conclusion is that we should still have some inefficiency in process as it allows for things like creative waste (and a few other things).

It's an interesting insight into why being inefficient is helpful. 3.5/5 stars

[David · Rating 3 out of 5]

Quite interesting, but the conclusions are too rooted in the current state of technology. For instance, self driving cars are rejected as a viable alternative to human drivers because of a deficiency in a particular implementation that led to a fatal crash, even though the problem was almost immediately corrected and it turned out that user error was the real cause. There is no possibility in the author’s mind that the systems will continue to improve over the coming years and decades and ultimately do a much better job of driving than callable humans, because somehow machines are just not as good. And even if they were, the thought of giving up the control is anathema, he frets that in a self-driving car he won’t be able to get lost and meet people by stopping to ask for directions, or decide on a whim to head down a side road, or make an unscheduled stop at a roadside attraction. The loss of these “inefficiencies” is not worth the saving 30,000+ lives lost every year in car accidents caused by human error.

[Drtaxsacto · Rating 4 out of 5]

Tenner is adept at explaining what big data (the accumulation of information in new ways) and does a pretty good job of comparing this revolution to the revolution of the 19th Century in industrial production. But he is also not a Pollyanna - he has a good balance and communicates it well.

It is a superb book to understand what the capabilities of big data can be and what it cannot offer us.

He pulls in a little Kanneman (everything does not need to be efficient in the classic definition) and poses some questions to better understand how the evolution of analytics is affecting and could affect us in the future.

It gave me a lot to think about on this area.

[Nick · Rating 4 out of 5]

We live in an age that glorifies efficiency—but at what cost? Tracking efficiency back to its roots in 19th-century work mill "efficiency" studies (which made people operate more like machines), Tenner shines a light on the uncritical "digital transformation" of everything. Critical domains of human experience are resistant to digital efficiency: education, health, safety systems. Silicon Valley regularly sells us on a vision of "friction-free" everything, but "inefficient" strategies of creative waste, desirable difficulty, and serendipity lead to real, lasting, impactful outcomes. Recommended, although at times a bit of a slog.

[Elizabeth · Rating 1 out of 5]

It's rare when I stop reading a book. That being said, Tenner's book has an amazing topic, but it's poor work. The book basically reads like a big, giant literature review. Also, it reads similar to essays that try to back a lot of information in a few short pages. It would be better if Tenner just focused on a few examples throughout the book. It seems as if though he was trying to show how much he did his research.

Shame. I was really looking forward to reading this book. I work with data myself, and I am very interested in the efficiency paradox.

[Venkatesh-Prasad · Rating 4 out of 5]

Interesting but tedious read

The book tackles a very interesting topic and provides lot of food for thought. The primary thesis of the book is efficiency may not be ideal in all situations and the act of achieving efficiency may be inefficient. The author supports this PoV with examples and evidence from studies on most occasions. So, lots of good stuff.

However, the writing style makes it a tedious read. I think the book could have been quite a bit shorter, a faster read, and just as effective.

[Josh U · Rating 3 out of 5]

Tenner knows what he is talking about, and his points are easy to comprehend.. maybe too easy.

He lists 5 main reasons why big data and complete automation can never fully replace human labor on the last page, and frankly, you could probably look at that and get the gist of the book.

That said, the introduction was quite well done, but get a bit deeper into the book and it tends to repeat itself and gets a bit lost in its examples.

[Jean · Rating 4 out of 5]

A good book that explains the difference with examples between efficacy and efficiency.
It also explains how efficiency can trouble creativity and how in the context of efficiency, a groundbreaking will be first inefficient needing many false starts to find more productive concepts.
Finally, it talks about how tacit knowledge, physical presence, creative waste, analog serendipity, desirable difficulties and cognitive bootstrapping are necessary strategies to achieve inspired "inefficiency".

[Lee Barry · Rating 4 out of 5]

I liked this book very much, particularly the “Moving Targets” chapter. In a culture preoccupied with automation and convenience, the side-effect of unintended inefficiency is not insignificant. I would agree with the one-star review, citing the excessive citations, but since they support the premise, they make the book more interesting and useful.

[Claire Binkley · Rating 5 out of 5]

If I am going into the study of this kind of thing, then I needed particularly to pay attention to all that Mr. Tennet had to say about it! With this thought, I delved into this book. What he presents is that his research flouts common assumptions about efficiency. This is what I wanted to see. So I am pleased with it. It is one of the books I picked for the 2019 Read Harder Challenge, and I am glad to have found it.

I particularly loved it for all the links to what else I could read in the field. And that I had already read Freakonomics: A Rogue Economist Explores the Hidden Side of Everything a long time ago, but not a whole lot of the other things.

Hmm, seeing that the other reviews say this reads like a lit review. Yes? Is this bad? As I see it, an efficient way to share information is to relay what books are good in the field. If you are looking for something more like Gone with the Wind or Charlotte's Web they should be shelved appropriately far in different parts of the library.

For what it is, I am pleased with what I found and what I hope to discover.

[Scott Wozniak · Rating 3 out of 5]

Instead of being a deep book on efficiency, it ended up being a survey of a few areas (information tech, navigation, etc.) and how they have unintended consequences but aren't all bad. So, I got some interesting history on how the tech was built and some cool stories on how it has been used well--and misused. But the analysis was very thin and insights banal.

[Zachary Ellis · Rating 3 out of 5]

Great introduction, great conclusion. The rest can get tedious. It can be summed up as technology should never make decisions for you but you should use it to assist you. And don’t get overly dependent on it. If you want examples to back up the claims give it a read. If you’re intrigued but not that much, check out the authors blog at EdwardTenner.com

[A S Jayanth Varma · Rating 5 out of 5]

Skim read the book. Got to know the author gave clarity about the prompt engineering of digital search engines. Rather than choosing between the options, the technology should give the correct path for the user per the question asked and so does the user should ask the right question to improve all their decisions by saving time to choose between options on the internet

[Groucho42 · Rating 2 out of 5]

Subtitled "What Big Data Can't Do" just to help in search engines. Not at all about it. Ok storytelling that doesn't strongly address efficiency other than in the most general sense. Defintely not one to add to my shelves.

[Walter · Rating 3 out of 5]

This book was kind of all over the place and didn't really say much. I don't know if I agree with the theme either, and wasn't particularly compelled by the arguments. It felt like it pieced together a lot of stuff that has been touted in other books, but didn't do so in a very captivating way.