Everybody Lies

by Seth Stephens-Davidowitz

Even the most sophisticated neuroimaging methodologies can tell us how a thought is splayed out in 3-D space, but not what the thought consists of.

As if the tradeoff between tractability and richness weren’t bad enough, scientists of human nature are vexed by the Law of Small Numbers—Amos Tversky and Daniel Kahneman’s name for the fallacy of thinking that the traits of a population will be reflected in any sample, no matter how small.

Google was invented so that people could learn about the world, not so researchers could learn about people. But it turns out the trails we leave as we seek knowledge on the internet are tremendously revealing.

But the power of Google searches is not that they can tell us that God is popular down South, the Knicks are popular in New York City, or that I’m not popular anywhere. Any survey could tell you that. The power in Google data is that people tell the giant search engine things they might not tell anyone else.

And Google searches presented a picture of America that was strikingly different from that post-racial utopia sketched out by the surveys. I remember when I first typed “nigger” into Google Trends. Call me naïve. But given how toxic the word is, I fully expected this to be a low-volume search. Boy, was I wrong.

On Obama’s first election night, when most of the commentary focused on praise of Obama and acknowledgment of the historic nature of his election, roughly one in every hundred Google searches that included the word “Obama” also included “kkk” or “nigger(s).”

Google searches, in other words, helped draw a new map of racism in the United States—and this map looked very different from what you may have guessed. Republicans in the South may be more likely to admit to racism. But plenty of Democrats in the North have similar attitudes. Four years later, this map would prove quite significant in explaining the political success of Trump.

12 percent of search queries with “Trump” also included the word “Clinton.” More than one-quarter of search queries with “Clinton” also included the word “Trump.” We have found that these seemingly neutral searches may actually give us some clues to which candidate a person supports. How? The order in which the candidates appear.

Black Americans told polls they would turn out in large numbers to oppose Trump. But Google searches for information on voting in heavily black areas were way down. On election day, Clinton would be hurt by low black turnout.

Silver found that the single factor that best correlated with Donald Trump’s support in the Republican primaries was that measure I had discovered four years earlier. Areas that supported Trump in the largest numbers were those that made the most Google searches for “nigger.”

it is not necessary to have some connection between mistakes and the forbidden, some theory of the mind where people reveal their secret desires via their errors. These slips of the fingers can be explained entirely by the typical frequency of typos.

The porn data and the Google search data are not just new; they are honest. In the pre-digital age, people hid their embarrassing thoughts from other people. In the digital age, they still hide them from other people, but not from the internet

First, and perhaps most important, if you are going to try to use new data to revolutionize a field, it is best to go into a field where old methods are lousy.

What Gentzkow and Shapiro found interesting, then, about the gay marriage story was not that news organizations differed in their coverage; it was how the newspapers’ coverage differed—it came down to a distinct shift in word choice.

The scholars wondered whether language might be the key to understanding bias. Did liberals and conservatives consistently use different phrases? Could the words that newspapers use in stories be turned into data?

So what can we learn from the frequency with which words or phrases appear in books in different years? For one thing, we learn about the slow growth in popularity of sausage and the relatively recent and rapid growth in popularity of pizza.

If my book were written in 1800, I would have said, “The United States are divided.” This little usage difference has long been a fascination for historians, since it suggests there was a point when America stopped thinking of itself as a collection of states and started thinking of itself as one nation.

So what did the linguistic data tell us? First, how a man or woman conveys that he or she is interested. One of the ways a man signals that he is attracted is obvious: he laughs at a woman’s jokes. Another is less obvious: when speaking, he limits the range of his pitch. There is research that suggests a monotone voice is often seen by women as masculine, which implies that men, perhaps subconsciously, exaggerate their masculinity when they like a woman. The scientists found that a woman signals her interest by varying her pitch, speaking more softly, and taking shorter turns talking. There are also major clues about a woman’s interest based on the particular words she uses. A woman is unlikely to be interested when she uses hedge words and phrases such as “probably” or “I guess.”

For women, there is some bad news here, as the data seems to confirm a distasteful truth about men. Conversation plays only a small role in how they respond to women. Physical appearance trumps all else in predicting whether a man reports a connection. That said, there is one word that a woman can use to at least slightly improve the odds a man likes her and it’s one we’ve already discussed: “I.” Men are more likely to report clicking with a woman who talks about herself. And as previously noted, a woman is also more likely to report a connection after a date where she talks about herself.

the words most disproportionately used by people of different ages on Facebook. I call this graphic “Drink. Work. Pray.” In people’s teens, they’re drinking. In their twenties, they are working. In their thirties and onward, they are praying.

A team of scientists, led by Andy Reagan, now at the University of California at Berkeley School of Information, downloaded the text of thousands of books and movie scripts. They could then code how happy or sad each point of the story was.

As the authors conclude, “Content is more likely to become viral the more positive it is.” Note this would seem to contrast with the conventional journalistic wisdom that people are attracted to violent and catastrophic stories.

Sometimes Democrats and Republicans use different phrasing to describe the same concept. In 2005, Republicans tried to cut the federal inheritance tax. They tended to describe it as a “death tax” (which sounds like an imposition upon the newly deceased). Democrats described it as an “estate tax” (which sounds like a tax on the wealthy). Similarly, Republicans tried to move Social Security into individual retirement accounts. To Republicans, this was a “reform.” To Democrats, this was a more dangerous-sounding “privatization.”

Many people, particularly Marxists, have viewed American journalism as controlled by rich people or corporations with the goal of influencing the masses, perhaps to push people toward their political views. Gentzkow and Shapiro’s paper suggests, however, that this is not the predominant motivation of owners. The owners of the American press, instead, are primarily giving the masses what they want so that the owners can become even richer.

In fact, the liberal bias is well calibrated to what newspaper readers want. Newspaper readership, on average, tilts a bit left. (They have data on that.) And newspapers, on average, tilt a bit left to give their readers the viewpoints they demand. There is no grand conspiracy. There is just capitalism.

The authors’ rather unconventional idea? They could help measure GDP based on how much light there is in these countries at night. They got that information from photographs taken by a U.S. Air Force satellite that circles the earth fourteen times per day.

These days, photographs of supermarket lines are valuable data. The fullness of supermarket bins is data. The ripeness of apples is data. Photos from outer space are data. The curvature of lips is data. Everything is data! And with all this new data, we can finally see through people’s lies.

Many people underreport embarrassing behaviors and thoughts on surveys. They want to look good, even though most surveys are anonymous. This is called social desirability bias.

Lying to oneself may explain why so many people say they are above average. How big is this problem? More than 40 percent of one company’s engineers said they are in the top 5 percent. More than 90 percent of college professors say they do above-average work. One-quarter of high school seniors think they are in the top 1 percent in their ability to get along with other people. If you are deluding yourself, you can’t be honest in a survey.

And there’s another huge advantage that Google searches have in getting people to tell the truth: incentives. If you enjoy racist jokes, you have zero incentive to share that un-PC fact with a survey. You do, however, have an incentive to search for the best new racist jokes online. If you think you may be suffering from depression, you don’t have an incentive to admit this to a survey. You do have an incentive to ask Google for symptoms and potential treatments.

One of the more common questions for Google regarding men’s genitalia is “How big is my penis?” That men turn to Google, rather than a ruler, with this question is, in my opinion, a quintessential expression of our digital era.

interest in beauty and fitness is 42 percent male, weight loss is 33 percent male, and cosmetic surgery is 39 percent male. Among all searches with “how to” related to breasts, about 20 percent ask how to get rid of man breasts.

In 2004, in some parts of the United States, the most common search regarding changing one’s butt was how to make it smaller. The desire to make one’s bottom bigger was overwhelmingly concentrated in areas with large black populations. Beginning in 2010, however, the desire for bigger butts grew in the rest of the United States. This interest, if not the posterior distribution itself, has tripled in four years.

Google searches about one’s wife and breast implants are evenly split between asking how to persuade her to get implants and perplexity as to why she wants them. Or consider the most common search about a girlfriend’s breasts: “I love my girlfriend’s boobs.” It is not clear what men are hoping to find from Google when making this search.

Many of our deepest fears about how our sexual partners perceive us are unjustified. Alone, at their computers, with no incentive to lie, partners reveal themselves to be fairly nonsuperficial and forgiving. In fact, we are all so busy judging our own bodies that there is little energy left over to judge other people’s.

Many people are, for good reason, inclined to keep their prejudices to themselves. I suppose you could call it progress that many people today feel they will be judged if they admit they judge other people based on their ethnicity, sexual orientation, or religion.

Obama seemed to say all the right things. All the traditional media congratulated Obama on his healing words. But new data from the internet, offering digital truth serum, suggested that the speech actually backfired in its main goal. Instead of calming the angry mob, as everybody thought he was doing, the internet data tells us that Obama actually inflamed it. Things that we think are working can have the exact opposite effect from the one we expect.

There is, though, an alternative explanation for the discrimination that African-Americans feel and whites deny: hidden explicit racism.

Parents are two and a half times more likely to ask “Is my son gifted?” than “Is my daughter gifted?” Parents show a similar bias when using other phrases related to intelligence that they may shy away from saying aloud, like, “Is my son a genius?”

What then are parents’ overriding concerns regarding their daughters? Primarily, anything related to appearance. Consider questions about a child’s weight. Parents Google “Is my daughter overweight?” roughly twice as frequently as they Google “Is my son overweight?”

Members of a hate site perusing the oh-so-liberal nytimes.com? How could this possibly be? If a substantial number of Stormfront members get their news from nytimes.com, it means our conventional wisdom about white nationalists is wrong. It also means our conventional wisdom about how the internet works is wrong.

So what does the data tell us? In the United States, according to Gentzkow and Shapiro, the chances that two people visiting the same news site have different political views is about 45 percent. In other words, the internet is far closer to perfect desegregation than perfect segregation. Liberals and conservatives are “meeting” each other on the web all the time.

In other words, Facebook exposes us to weak social connections—the high school acquaintance, the crazy third cousin, the friend of the friend of the friend you sort of, kind of, maybe know. These are people you might never go bowling with or to a barbecue with. You might not invite them over to a dinner party. But you do Facebook friend them. And you do see their links to articles with views you might have never otherwise considered.

So when the Supreme Court was recently looking into the effects of laws making it more difficult to get an abortion, I turned to the query data. I suspected women affected by this legislation might look into off-the-books ways to terminate a pregnancy. They did. And these searches were highest in states that had passed laws restricting abortions.

Women living in states with the fewest abortion clinics had 54 percent fewer legal abortions—a difference of eleven abortions for every thousand women between the ages of fifteen and forty-four. Women living in states with the fewest abortion clinics also had more live births. However, the difference was not enough to make up for the lower number of abortions. There were six more live births for every thousand women of childbearing age.

We can’t blindly trust government data. The government may tell us that child abuse or abortion has fallen and politicians may celebrate this achievement. But the results we think we’re seeing may be an artifact of flaws in the methods of data collection. The truth may be different—and, sometimes, far darker.

In other words, while people were joining in a big public uproar over how unhappy they were about seeing all the details of their friends’ lives on Facebook, they were coming back to Facebook to see all the details of their friends’ lives. News Feed stayed. Facebook now has more than one billion daily active users.

In fact, I think Big Data can give a twenty-first-century update to a famous self-help quote: “Never compare your insides to everyone else’s outsides.” A Big Data update may be: “Never compare your Google searches to everyone else’s social media posts.”

The final—and, I think, most powerful—value in this digital truth serum is indeed its ability to lead us from problems to solutions. With more understanding, we might find ways to reduce the world’s supply of nasty attitudes.

When we lecture angry people, the search data implies that their fury can grow. But subtly provoking people’s curiosity, giving new information, and offering new images of the group that is stoking their rage may turn their thoughts in different, more positive directions.