The Skeptics' Guide to the Universe: How to Know What's Really Real in a World Increasingly Full of Fake

by Steven Novella

Interpret Disagreements About Details As If They Call the Deeper Consensus into Question

Deny and Distort the Consensus

Appeal to Conspiracy, Question the Motives of Scientists

Appeal to Academic/Intellectual Freedom

Argument from Consequences

This feature also often provides a clue as to the true motivation of the denial. The science is secondary: It’s the moral hazard they’re truly concerned about.

It is clear from history that science is the most powerful tool we have for understanding the world and improving our position in it. But science requires courage—the courage to face reality and accept its findings, even if they upset us or are disruptive to our comfortable ideology.

24.

Skeptics’ Guide Entry: P-Hacking and Other Research Foibles

Section: Science and Pse...

See also: Pseud...

The nature of science is not that of a steady, linear progression toward the Truth, but rather a tortuous road, often characterized by dead ends and U-turns, and yet ultimately inching toward a better, if tentative, understanding of the natural world. —Massimo Pigliucci

Even respected scientists may succumb to the temptation to tweak the rules after the outcomes are known—they just do it in a much more subtle and complex way. They may not even be aware they are doing it.

How do we tell the difference between reliable, solid, scientific research and low-quality research?

The Problem with p-Values

The term derives from the statistical calculation known as the p-value. The p-value is just one way to look at scientific data. It first assumes a specific null hypothesis (such as: There is no correlation between these two variables) and then asks, What is the probability that the data would be at least as extreme as they are if the null hypothesis were true? A p-value of 0.05 (a typical threshold for being considered “significant”) indicates a 5 percent probability that the data are due to chance rather than a real effect. Except—that’s not actually true. That is how most people interpret the p-value, but that’s not what it actually says. P-values don’t consider many other important variables, like prior probability, effect size, confidence intervals, and alternative hypotheses. For example, if we ask, “What is the probability of a new set of data replicating the results of a study with a p-value of 0.05?” we get a very different answer.

A study with a p-value of 0.01 may have only a 50 percent chance in an exact replication of producing another p-value of 0.01 (not the 99 percent chance that most people would assume).

It’s only a smell test to see if the data are at all interesting or just random noise.

Just as with breast cancer, you have to know what the base rate is. We also call this the prior probability. With a scientific hypothesis this often means the scientific plausibility. The lower the plausibility (just like the lower the rate of cancer in the population), the smaller the percentage of positive or statistically significant studies will be true positives. This means that we can’t know the probability that a hypothesis is true just from the p-value of one study. We need to know the plausibility of the hypothesis, and we need to know what all the other relevant studies showed. We call this a Bayesian approach—you take the new information, you add it to the prior existing information, and you come up with a new probability that the idea is true. While it’s possible to debate about how plausible a hypothesis is, one thing is clear: p-values (statistical significance) don’t mean nearly as much as most people think they do. Even a significant study doesn’t alter the prior probability by that much. It takes a convergence of multiple studies, and multiple independent lines of evidence, before we can be confident that a hypothesis is likely to be true.

Eric Loken and Andrew Gelman point out that measurement error also greatly exacerbates the problem. This is why we consider the signal-to-noise ratio in whatever is being measured in a scientific study. In a noisy environment, measurement errors are magnified, and the predictive value of a p-value plummets.

P-Hacking

A 2009 PLOS One review found that 33 percent of surveyed researchers admitted to committing one or more dubious practices. What dubious practices? Those that amount to p-hacking. It’s likely that most p-hacking is innocent, meaning that the researchers don’t realize they’re essentially cheating. For example, if you survey the data as you collect it, you might decide that once you cross over the p = 0.05 threshold you can stop collecting data and publish.

P-hacking is essentially mining the data or taking multiple throws of the dice and only counting the results you like.

Independent replications really are the ultimate arbiters of what is real in science. Any one study can be a fluke or the product of biased research. Only phenomena that are real, however, should show up in the data no matter who is doing the research.

The Fix

One important fix, as discussed above, is valuing exact replications more. Value is easy to determine in science—what gets published, what gets funded, and what gets you academically promoted.

Statistician Andrew Gelman of Columbia University suggests that researchers should do research in several steps. First, collect preliminary data and, if they look promising, design a replication where all the decisions about data collection are predetermined. Then register the study methods before collecting any data. Finally, collect a fresh set of data according to the published methods. At least then we’ll have honest p-values and eliminate p-hacking.

Researchers should not rely only on p-values. They should also report effect sizes and confidence intervals, which are more thorough ways of looking at the data. Tiny effect sizes (a one-week cold was reduced on average by one hour—wow!), no matter how significant, are always dubious because subt...

Simonsohn advocates researchers disclosing everything they do—all decisions about data collection and analysis. This way at least they cannot hide their p-hacking, an...

Nuzzo and others recommend more use of Bayesian analysis, which we discussed earlier—asking, What is the overall prob...

Where Do We...

This is what it takes for research to be convincing: 1. Rigorous studies that appear to minimize the effect of bias or unrelated variables 2. Results that are not only statistically significant but are significant in effect size as well (reasonable signal-to-noise ratio) 3. A pattern of independent replication consistent with a real phenomenon 4. Evidence proportional to the plausibility of the claim

Homeopathy, acupuncture, and ESP research are all plagued by these deficiencies. They haven’t produced research results anywhere near the threshold of acceptance. Their studies reek of p-hacking, they generally have tiny effect sizes, and there is no consistent pattern of replication, just chasing different quirky results.

All the public hears is, “Scientists report a significant result,” but most of what gets relayed as excited scientific breakthroughs is simply crap cluttering up the scientific journals.

25.

Skeptics’ Guide Entry: Conspiracy Theories

Section: Science and Pse...

See also: Grand Con...

A conspiracy theory, or more precisely a grand conspiracy, is a belief system that involves at its core the claim that a vastly powerful group is carrying out a deception a...

The government kinda sucks at keeping sec...

The Grand Conspiracy

No one doubts the existence of actual conspiracies. Whenever two people get together and agree to work together to perpetrate a crime, that’s a conspiracy.

A grand conspiracy is much bigger. This is a conspiracy that by its very nature must span many people and organizations, perhaps even multiple nations and generations.

The grand conspiracy forms a triangle of sorts. First there are the conspirators. This is typically a large, powerful, and shadowy organization with vast resources and control. They need to be powerful in order to fake moon landings, poison the public through jet exhaust, or frame terrorists for 9/11. Then there are the conspiracy theorists, an “Army of Light” that is able to see through the conspiracy (because they are just so clever). Finally, there is everyone else, the dupes or “sheeple” who believe the standard explanation of history and current events.

Conspiracy T...

Conspiracy theories are often elaborate exercises in special pleading.

The cover-up is where grand conspiracies tend to get into a death spiral of special pleading.

Individual pharmaceutical companies compete against each other, and new start-ups form all the time. What’s stopping a new company from marketing an alleged cure? What about other countries?

Who actually controls research? Most basic cancer research is government funded and carried out by researchers at university labs. How would Big Pharma control this research?

Pharmaceutical companies get involved only in the last step, taking a possible new drug target and developing a workable pharmaceutical.