Calling Bullshit: The Art of Skepticism in a Data-Driven World

by Carl T. Bergstrom

It violates what we term the principle of proportional ink: When a shaded region is used to represent a numerical value, the size (i.e., area) of that shaded region should be directly proportional to the corresponding value.

Indeed, machine learning and artificial intelligence live and die by the data they employ. With good data you can engineer remarkably effective algorithms for translating one language into another, for example. But there’s no magical algorithm that can spin flax into gold. You can’t compensate for bad data. If someone tells you otherwise, they are bullshitting.

Because the data are central to these systems, one rarely needs professional training in computer science to spot unconvincing claims or problematic applications. Most of the time, we don’t need to understand the learning algorithm in detail. Nor do we need to understand the workings of the program that the learning algorithm generates. (In so-called deep learning models, no one—including the creators of the algorithm—really understands the workings of the program that algorithm generates.) All you have to do to spot problems is to think about the training data and the labels that are fed into the algorithm. Begin with bad data and labels, and you’ll get a bad program that makes bad predictions in return. This happens so often that there is an acronym for it in computer science: GIGO—garbage in, garbage out. We’d like to deflate the hype behind AI, with a focus on the data rather than the details of the algorithm inside the black box.

As Zachary Lipton, an AI researcher at Carnegie Mellon University, explains, “Policy makers [are] earnestly having meetings to discuss the rights of robots when they should be talking about discrimination in algorithmic decision making.” Delving into the details of algorithmic auditing may be dull compared to drafting a Bill of Rights for robots, or devising ways to protect humanity against Terminator-like superintelligent machines. But to address the problems that AI is creating now, we need to understand the data and algorithms we are already using for more mundane purposes.

We want to provide an antidote to this hype, but first, let’s look at an instance in which machine learning delivers. It involves a boring, repetitive, everyday task that has received too little limelight rather than too much.

This is just a toy example, but the same issues arise in most machine learning applications. Complicated models do a great job of fitting the training data, but simpler models often perform better on the test data than more complicated models. The trick is figuring out just how simple of a model to use. If the model you pick is too simple, you end up leaving useful information on the table.

Why is it so important to understand the role of training data in machine learning? Because it is here where the most catastrophic mistakes are made—and it is here where an educated eye can call bullshit on machine learning applications. In chapter 3, we told a story about a machine learning algorithm that was supposed to detect who was a criminal, but instead learned to detect who was smiling. The problem was with the training data. The criminal faces used to train the algorithm were seldom smiling, whereas the noncriminal faces were usually smiling. In the real world, smiling is not a good indicator of whether someone is a criminal or not, but the machine didn’t know it was supposed to be finding signs of criminality. It was just trying to discriminate between two different sets of faces in the training set. The presence or absence of a smile turned out to be a useful signal, because of the way the training images were chosen. No algorithm, no matter how logically sound, can overcome flawed training data. Early computing pioneer Charles Babbage commented on this in the nineteenth century: “On two occasions I have been asked, ‘Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’…I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.”

To call bullshit on the newest AI startup, it is often sufficient to ask for details about the training data. Where did it come from? Who labeled it? How representative is the data? Remember our black box diagram: If the data going into the black box pass your initial interrogation, skip the algorithm and focus instead on the other end of the chain: what comes out of the black box and how it’s interpreted.

It is not difficult to question the training data, as we did for the criminal faces paper in chapter 3, or the interpretation of the results, as we did for the gaydar paper in this chapter. And you can do both without opening the black box; in neither case did we have to discuss how a neural network operates.

Guestrin and his co-authors found that the algorithm was not paying much attention to snouts, eyes, fur, or any of the morphological features a person would use to distinguish a husky from a wolf. Instead, it was picking up on something external, a correlate of being a wolf that was present in the images. The machine learned that the wolf images but not the husky images tended to be shot in the snow, and exploited this difference in making its decisions.

When machine learning algorithms key in on auxiliary features of this sort, they may do well at analyzing images exactly like the ones they were trained on, but they will not be able to generalize effectively to other contexts. John Zech and colleagues at California Pacific Medical Center wanted to investigate how well neural networks could detect pathologies such as pneumonia and cardiomegaly—enlarged heart—using X-ray images. The team found that their algorithms performed relatively well in hospitals where they were trained, but poorly elsewhere. It turned out that the machine was cueing on parts of the images that had nothing to do with the heart or lungs. For example, X-ray images produced by a portable imaging device had the word PORTABLE printed in the upper right corner—and the algorithm learned that this is a good indicator that a patient has pneumonia. Why? Because portable X-ray machines are used for the most severely ill patients, who cannot easily go to the radiology department of the hospital. Using this cue improved prediction in the original hospital. But it was of little practical value. It had little to do with identifying pneumonia, didn’t cue in on anything doctors didn’t already know, and wouldn’t work in a different hospital that used a different type of portable X-ray machine. Machines are not free of human biases; they perpetuate them, depending on the data they’re fed. In the case of criminal sentencing, ProPublica and others have shown that algorit...

Algorithmic biases can be particularly difficult to eradicate. Policy makers may require rules that forbid decisions based on race or gender, but it is often not sufficient to simply omit that information in the data provided to an algorithm. The problem is that other pieces of information may be correlated with race or gender, particularly when considered in concert. For example, if you build a machine to predict where the next domestic violence event will occur, the machine may choose an apartment over detached housing since those who share a wall are more likely to report the event. This kind of prediction then resurfaces racial elements that you tried to remove in the first place. If you remove names from résumés as a way of eliminating gender discrimination, you may be disappointed, as Amazon was, when the machine continues to preferentially choose men over women. Why? Amazon trained the algorithm on its existing résumés, and there are features on a résumé besides a name that can reveal one’s gender—such as a degree from a women’s college, membership in a women’s professional organization, or a hobby with skewed gender representation.

In 2014, TED Conferences and the XPrize Foundation announced an award for “the first artificial intelligence to come to this stage and give a TED Talk compelling enough to win a standing ovation from the audience.” People worry that AI has surpassed humans, but we doubt AI will claim this award anytime soon. One might think that the TED brand of bullshit is just a cocktail of sound-bite science, management-speak, and techno-optimism. But it’s not so easy. You have to stir these elements together just right, and you have to sound like you believe them. For the foreseeable future, computers won’t be able to make the grade. Humans are far better bullshitters.

SCIENCE IS HUMANITY’S GREATEST INVENTION. Our species has evolved to act within a narrow band of time scales, ranging from a few milliseconds to a few decades. We have evolved to operate within a similarly narrow band of spatial scales, ranging from micrometers to kilometers. Yet temporal and spatial scales of the universe run from unimaginably larger to incomprehensibly smaller. Science allows us to transcend these limitations. It gives us the tools to understand what happened in the first picoseconds following the Big Bang, and how the universe has evolved over the 13.7 billion years since. Science allows us to model the internal geometry of a single atom and to grapple with cosmological distances so large that we record them in light-years. For all that it does, however, it would be a mistake to conclude that science, as practiced today, provides an unerring conduit to the heart of ultimate reality. Rather, science is a haphazard collection of institutions, norms, customs, and traditions that have developed by trial and error over the past several centuries. Science is culturally contingent on its emergence from European natural philosophy as practiced a few centuries ago, and evolutionarily contingent on the cognitive capacities of the species that practices it. Science is jury-rigged atop the evolved psychology of one particular species of ape, designed to provide members of this species with adequate incentives to work in concert toward a better understanding of the ...

Sir Francis Bacon, often credited as an early architect of the modern scientific method, exhorts his followers to epistemic purity in the preface of his Instauratio Magna: Lastly, I would address one general admonition to all; that they consider what are the true ends of knowledge, and that they seek it not either for pleasure of the mind, or for contention, or for superiority to others, or for profit, or fame, or power, or any of these inferior things; but for the benefit and use of life; and that they perfect and govern it in charity. The rest of us—including every living scientist we know—are epistemically sullied. We act from the same human motivations as everyone else. This doesn’t mean that scientists are irresponsible, untrustworthy, or unethical; it just means that they have other interests beyond the pure quest for understanding. To understand how science works and where things can go wrong, we need to understand what these interests are. So before addressing bullshit in science, let’s look at what motivates its stewards. While scientists tend to be intensely curious people who love solving puzzles, they are in most respects just like everyone else, striving to earn money and win status among their peers. We scientists want to understand how the world works, but we also want to impress our friends and colleagues, win the next promotion, and—if things go really, really well—make the big time with a guest appearance on the Daily Show or Last Week Tonight. Scientists...

Even science’s deepest foundations can be questioned, and at times replaced, when they prove incompatible with current discoveries. Geneticists and evolutionary biologists had long assumed that genes were the sole molecular vehicles of inheritance. Offspring resemble their parents because they share the same DNA sequences in their genomes. But when genetic sequencing became inexpensive and new molecular biology techniques gave us ways to measure how genes were being activated, strong evidence began accumulating that this was not the full picture. In addition to passing their genes to their offspring, parents sometimes pass a second layer of nongenetic information about what genes to activate, when. This became known as epigenetics. Because our scientific understanding of the world can change in light of new evidence, science has proven resilient to the occasional wrong turn and even to deliberate misdirection in the form of scientific fraud.

But look at it from the researcher’s perspective. Imagine that you have just spent months collecting a massive data set. You test your main hypothesis and end up with results that are promising—but nonsignificant. You know you won’t be able to publish your work in a good journal this way, and you may not be able to publish at all. But surely the hypothesis is true, you think—maybe you just don’t have quite enough data. So you keep collecting data until your p-value drops below 0.05, and then stop right away lest it drift back above that threshold. Or maybe you try a few other statistical tests. Since the data are close to significant, the right choice of measures and of tests might allow you to clear the critical p = 0.05 bar. Sure enough, with a bit of tinkering, you find an approach that gives you a significant result. Or maybe your hypothesis appears to be correct only for men, and the meaningful pattern is getting swamped out by including women in your sample. You look—and lo and behold, once you look at men only, you’ve got a statistically significant result. What are you going to do? Scrap the whole project, abandon thousands of dollars invested, ask your graduate student to delay her graduation by another six months…or just write up the result for men and submit to a top journal? Under those circumstances it may feel easy to rationalize the latter approach. “I’m sure the trend is really there,” you might tell yourself. “I was thinking about omitting women from the s...

The problem arises because news sources often fail to clearly indicate the preliminary nature of the findings that they report, and worse yet, they seldom report when the studies covered previously fail to pan out.

Popular science writing often allows, if not actively encourages, a fundamental misunderstanding about what the results of a single study mean for science. In the news media and even in textbooks, scientific activity is often portrayed as a collecting process, and a scientific paper as the report of what has been collected. By this view, scientists search out the facts that nature has hidden away; each uncovered fact is published in a scientific paper like a stamp laid in a collector’s album; textbooks are essentially collections of such facts. But science doesn’t work like this. The results of an experiment are not a definitive fact about nature. The results of experiments involve chance, as well as a web of assumptions about how such results are properly assessed. The interpretation of experimental results involves a yet more complex network of models and assumptions about how the world works. Rather than representing a definitive fact about nature, each experiment or collection of observations merely represents an argument in favor of some particular hypothesis.

Worse still, there is a powerful selection bias in what scientific studies we end up hearing about in the popular press and on social media. The research studies that are reported in the popular press are not a random sample of those conducted, or even of those published. The most surprising studies are the ones that make the most exciting articles. If we fail to take this into account, and ignore all of the less surprising findings from related studies, we can end up with an unrealistic picture of how scientific knowledge is developing.

In a typical economic market for consumer goods, money flows in one direction; goods flow in the other. A steel company provides raw materials to an auto manufacturer, which assembles cars and sells them to consumers. The consumers pay the manufacturer, which in turn pays the steel company. The market for scholarly journals is different. A scholar pours her heart and soul into writing a scholarly paper. She provides her work to an academic publisher, which collects a number of such articles and bundles them together as a journal issue. The publisher charges libraries to subscribe, but doesn’t pass any of the revenue on to the authors.

Some journals even require that the author pay to have her work published. Why would anyone pay? Recall what motivates academic scholars. As we discussed earlier in the chapter, scholars are rewarded for the reputations they accrue, and publishing is how reputations are made.

Here is Goodhart’s law’s again: When a measure becomes a target, it ceases to be a good measure. This has happened to a substantive swath of the scientific literature. When scientists started judging one another by the number of papers published, a market sprung up for journals willing to publish low-quality work. At the bottom of that particular barrel are journals produced by so-called predatory publishers. These parasites on the scientific publishing system provide little to nothing by way of rigorous peer review. Today they are sucking tens of millions of dollars from the academic system and polluting the literature with millions of unreliable articles.

As a result, there is no surefire way for you, as the reader, to know, beyond a shadow of a doubt, that any particular scientific paper is correct. Usually the best you can hope to do is to determine that a paper is legitimate. By legitimate, we mean that a paper is (1) written in good faith, (2) carried out using appropriate methodologies, and (3) taken seriously by the relevant scientific community.

Another issue to be on the lookout for is whether the claims in a paper are commensurate with the venue in which it is published. As we mentioned, journals occupy different positions in a hierarchy of prestige. All else equal, papers published in top journals will represent the largest advances and have the highest credibility. Less interesting or less credible findings will be relegated to less prestigious outlets. Be wary of extraordinary claims appearing in lower-tier venues. You can think of this as the scientist’s version of “if you’re so smart, why aren’t you rich?” Thus if a paper called “Some Weights of Riparian Frogs” lists the weights of some frogs in the little-known Tasmanian Journal of Australian Herpetology, there is relatively little cause for concern. A table of frogs’ weights, while possibly useful to some specialists in the area, is far from a scientific breakthrough and is well suited for the journal in question. If, however, a paper called “Evidence That Neanderthals Went Extinct during the Hundred Years’ War” appears in the equally low-profile Journal of Westphalian Historical Geography, this is serious cause for concern. That finding, if true, would revolutionize our understanding of hominin history, and shake our notion of what it means to be human. Such a finding, if true, would appear in a high-profile journal.

Finally, irrespective of the problems we have discussed in this chapter, science just plain works. As we stated at the start of this chapter, science allows us to understand the nature of the physical world at scales far beyond what our senses evolved to detect and our minds evolved to comprehend. Equipped with this understanding, we have been able to create technologies that would seem magical to those only a few generations prior. Empirically, science is successful. Individual papers may be wrong and individual studies misreported in the popular press, but the institution as a whole is strong. We should keep this in perspective when we compare science to much of the other human knowledge—and human bullshit—that is out there.

If bullshit is everywhere, how can we avoid being taken in? We think it is crucial to cultivate appropriate habits of mind. After all, our habits of mind keep us safe on a daily basis.

1. QUESTION THE SOURCE OF INFORMATION Journalists are trained to ask the following simple questions about any piece of information they encounter: Who is telling me this? How does he or she know it? What is this person trying to sell me?

In short, people may be trying to sell you used cars or life insurance or beauty treatments—or they be trying to sell you ideas, viewpoints, and perspectives. Some sales jobs get you to part with your hard-earned money. Other sales jobs convince you to believe something that you didn’t believe before, or to do something you wouldn’t have done otherwise. Everyone is trying to sell you something; it is just a matter of figuring out what.

BEWARE OF UNFAIR COMPARISONS “Airport Security Trays Carry More Germs Than Toilets!” Media outlets around the world ran some version of this headline after a research study was published in September 2018, confirming the fears of every germophobe who has ever suffered through the airport security screening process. But the claim is somewhat disingenuous. The scientists who did this study looked only at respiratory viruses, the kind transmitted through the air or through droplets on people’s hands when they cough or sneeze. It isn’t surprising that security trays have more respiratory viruses than toilet seats. People don’t usually cough or sneeze onto toilet seats, nor do they tend to touch them extensively with their hands. Toilet seats have plenty of germs, just not of the kinds that the researchers were tallying.

Let’s look at another example. People have always enjoyed ranked lists. In a clickstream economy, where advertising revenues depend on page views, they’re gold. A single top-ten list can generate ten page views per reader by putting each list item on a separate page. Farewell Casey Kasem, hello “12 Reasons Why Sam, the Cat with Eyebrows, Should Be Your New Favorite Cat.”

This example of violent crime rates serves to illustrate a more general principle: Ranked lists are meaningful only if the entities being compared are directly comparable.

IF IT SEEMS TOO GOOD OR TOO BAD TO BE TRUE…

The new story was then presumably misinterpreted by whoever runs the social media feed for NBC, and a decline at 40 percent of schools was transposed into a decline by 40 percent. This improbably large effect is where the “if it sounds too good or bad to be true…” rule of thumb comes into play. We find that this third rule is particularly good for spotting bullshit that spreads across social media. In a social media environment, the posts that are spread most widely are often those that shock, spark a sense of wonder, or inflame feelings of righteous indignation: namely, those that make the most extreme claims. And the most extreme claims are often too good or too bad to be true.

THINK IN ORDERS OF MAGNITUDE Think back to philosopher Harry Frankfurt’s distinction between bullshit and lies. Lies are designed to lead away from the truth; bullshit is produced with a gross indifference to the truth. This definition gives us a considerable advantage when trying to spot bullshit. Well-crafted lies will be plausible, whereas a lot of bullshit will be ridiculous even on the surface. When people use bullshit numbers to support their arguments, they are often so far off that we can spot the bullshit by intuition and refute it without much research.

This process of making back-of-the-envelope approximations is known as Fermi estimation, after the physicist Enrico Fermi, who was famous for estimating the size of an atomic blast using these simple methods.*5

At a May 2018 hearing of the US House Committee on Science, Space and Technology, Representative Mo Brooks (R-Ala.) speculated that perhaps rising sea levels could be attributed to rocks falling into the ocean. For example, he asked his constituents to consider the continuously eroding White Cliffs of Dover. These have to be filling up the ocean over time, and all the water that they displace must be going somewhere. It is comforting that like Aesop’s crow,*6 Representative Brooks understands the consequences of putting rocks in water. But this is an entirely inadequate explanation that belies Representative Brooks’s ability to grasp how vast the oceans are.

If you are going to make up a number out of whole cloth, be sure to make up one that actually supports your argument.

AVOID CONFIRMATION BIAS Extreme claims do well on social media; so do posts that reaffirm things about the world that we already believe to be true. This brings us to our next rule of thumb for spotting bullshit: Avoid confirmation bias.

CONSIDER MULTIPLE HYPOTHESES In this chapter we have mainly looked how you can spot bullshit in the form of incorrect facts. But bullshit also arises in the form of incorrect explanations for true statements. The key thing to realize is that just because someone has an explanation for some phenomenon doesn’t mean that it is the explanation for that phenomenon.

Corroborate and triangulate. If you come across a surprising claim or dramatic news report from an unknown source, use a search engine to see if you can find the same claims from other sources. If not, be very suspicious. Even when one news outlet has a big scoop, other papers quickly report on the fact that the first outlet broke the story. Be sure that those reporting on the story include reliable sources. Disinformation campaigns may plant multiple versions of the same false tale in unreliable outlets. Pay attention to where information comes from. If you find a piece of candy lying in the street, you are not going to eat it or share it with your friends. Unsourced information is the same. All too often, someone we don’t know posts a factoid or statistic or data graphic on social media without listing its source—and we share it anyway. Dig back to the origin of the story. This takes time and effort, but if you want to avoid spreading misinformation, it is effort well spent. Don’t simply read a headline or tweet; read the full news story. If the news story is from an outlet that tends to sensationalize, don’t stop there. Dig back to the primary article or report that the story is talking about. Or dig really deep and take a look at the data yourself. Use reverse image lookup. Several search engines provide a reverse image lookup service in which you upload a picture or a few frames from a video, and the search engine tells you where on the Web that picture or video can b...

Most important: When you are using social media, remember the mantra “think more, share less.” The volume of information on social media, and the speed at which it allows us to interact, can be addictive. But as responsible citizens, we need to keep our information environments as clean as possible. Over the past half century people have learned not to litter the sides of the interstates. We need to do the same on the information superhighway. Online, we need to stop throwing our garbage out the car window and driving away into the anonymous night.

Calling bullshit is a performative utterance in which one repudiates something objectionable. The scope of targets is broader than bullshit alone. You can call bullshit on bullshit, but you can also call bullshit on lies, treachery, trickery, or injustice.

Calling bullshit is itself a performative utterance—and this observation is important for understanding what it means to call bullshit upon some claim. When I call bullshit, I am not merely reporting that I am skeptical of something you said. Rather, I am explicitly and often publicly pronouncing my disbelief. Why does this matter? Performative utterances are not idle talk. They are powerful acts, to be used with prudence. Calling bullshit is the same. Don’t call bullshit carelessly—but if you can, call bullshit when necessary.

Any group adopts wrong ideas at times, and these ideas require forceful public repudiation. But if you want to call bullshit, it is important to do so responsibly, appropriately, and respectfully. This is not an oxymoron. We the authors do this for each other on a daily basis. We understand that the proper target of calling bullshit is an idea, not a person. We realize that we will sometimes be on the producing end of bullshit. We’ve learned to accept and acknowledge our mistakes with a modicum of grace when bullshit is called on us. Spotting bullshit is a private activity. Calling bullshit is a public one. If you can spot bullshit, you can keep yourself safe from its effects. If you can call bullshit, you can protect your entire community. Of course not everyone will feel comfortable calling bullshit—and that’s fine.

Carelessly calling bullshit is a quick way to make enemies of strangers and strangers of friends. Because calling bullshit is a performative utterance, it is particularly important to be correct when you do so. People despise hypocrites, and being full of shit when you call bullshit is about as hypocritical as one can get. It’s worse if you’ve called it aggressively and judgmentally. There’s a thin line between being a tough-minded skeptic and a domineering jerk. We want to make sure you don’t end up on the wrong side of the line.

Sir—A. J. Tatem and colleagues calculate that women may outsprint men by the middle of the twenty-second century (Nature 431, 525; 200410.1038/​431525a). They omit to mention, however, that (according to their analysis) a far more interesting race should occur in about 2636, when times of less than zero seconds will be recorded. In the intervening 600 years, the authors may wish to address the obvious challenges raised for both time-keeping and the teaching of basic statistics. This response is both humorous and highly effective. In it, Rice employed one of our favorite refutation strategies: reductio ad absurdum.

Either we have stumbled onto a rather amazing discovery in terms of post-mortem ichthyological cognition, or there is something a bit off with regard to our uncorrected statistical approach.

The salmon study was followed by a number of more technical and less entertaining research articles that further dissected the problem, estimated its magnitude, and proposed solutions. These projects were critical for the field to advance, but none were as effective as the original salmon study in drawing attention to the basic problem. Humor is not a requirement for reductio ad absurdum, but when integrated well it can be highly effective. It is memorable and spreads ideas quickly through informal conversation.

Reductio ad absurdum can be fun and effective but nothing is as devastating to specious arguments as a simple counterexample. If someone claims that A implies B, find a case in which A is true but B is not. A in this case is a large, long-lived multicellular organism; B is having an adaptive immune system. Trees can be classified as A, but do not have B; therefore A does not imply B.*3 It takes practice to find good counterexamples, and few will be as effective and crushing as the tree example. If and when you do find one, please be kind. If the claims were made in earnest with no ill intent, help your target recover. We all make mistakes. Hopefully your counterexample will be instructive and will lead to better analyses with stronger claims in the future.