The Skeptics' Guide to the Universe: How to Know What's Really Real in a World Increasingly Full of Fake

by Steven Novella

Using a broad definition and looking at a large data set, it’s almost guaranteed that one will find apparent anomalies.

The assumption that anomalies must be significant rather than random is an error in the understanding of statistics, a form of innumeracy.

Often the conclusion that something is an anomaly derives from a lack of familiarity or expertise. We may, for example, be unfamiliar with conditions in exotic environments, and something may seem like an anomaly simply because you lack the specialized knowledge (scientific, technical, historical) to know the true explanation.

Once an apparent anomaly (broadly defined) is found in the data, Jonny Pseudoscientist will then tend to commit a pair of logical fallacies. First, he confuses unexplained with unexplainable. This leads him to prematurely declare something a true anomaly without first exhaustively trying to explain it with conventional means. Second, he uses the argument from ignorance, saying that because the anomaly cannot be explained, his specific pet theory must be true.

Examples of Anomaly Hunting

Ghost Anomalies

The JFK Assassination and the Umbrella Man

The man (Louie Steven Witt) was asked to come forward and explain his actions before Congress, and he testified that the umbrella was a protest of Joseph Kennedy’s appeasement policies when he was ambassador to the Court of St. James’s in 1938–39. The umbrella was a reference to the umbrella often carried by British prime minister Neville Chamberlain. This is not as random as it may seem. An open umbrella was a common protest of appeasement policies.

Moon Landing Hoax

Flat-Earthers

17.

Skeptics’ Guide Entry:

Data ...

Data mining is the process of sifting through large sets of data looking for any possible correlation, many of which will occur by chance. While this is a legitimate method for generating hypotheses, such data are not confirmatory and the method is easily abused.

Torture the data, and it will confess to anything. —Ronald Coase

The problem of data mining is both common and often subtle and missed. It’s rooted in both the nature of human brain function and a common logical fallacy. The former is that of pattern recognition and the latter the fallacy of confusing correlation with causation.

As Carl Sagan eloquently pointed out, randomness is clumpy. We tend to notice clumps. They stick out as potential patterns.

Our brains evolved to capture everything and then weed out the fake patterns, but we are better at step one than step two.

Science is partly the task of separating those patterns that are real from those that are accidents of random clumpiness. Science is formalized reality testing.

Mining large sets of data dramatically increases the probability of seeing apparent patterns.

Therefore, any pattern or correlation that is the result of searching (again, whether actively or passively) a large data set for any potential correlation should be viewed as only a possible correlation requiring further validation. Such correlations can be used as the beginning of meaningful research (not the conclusion). Legitimate scientific procedure is to then ask the question, Is this correlation real?

To confirm a correlation, you must ask ahead of time, What is the probability of this specific correlation occurring? and then test a new or fresh set of data to see if the correlation holds up. Since you’re looking for a specific predetermined correlation, now you can apply statistics and ask what is the probability of that correlation occurring by chance.

When looking at the new data set, you can’t include the original data that made you notice the correlation in the first place. The new data have to be entirely independent.

prominent in epidemiological studies, which are basically studies that sift through large data sets to look for correlations. To be fair, most of the time they’re presented properly as preliminary data that needs verification. Such correlations are then run through the mill of science and either hold up or they don’t.

But this process may take years. Meanwhile, the media often reports such preliminary correlations as if they were conclusions, without putting them into proper scientific context. Scientists and the institutions that support them are also often to blame, for example sending out press releases to announce an interesting new health correlation before it has been verified.

We are compelled by the patterns we see. They speak to us. Our “common sense” often fails to properly guide us, as we tend to err hugely on the side of accepting whatever patterns we see.

The only way to navigate through the sea of patterns is with the systematic methods of logic and testing that collectively are known as science.

18.

Skeptics’ Guide Entry: Coincidence

The term “coincidence” refers to the chance alignment of two variables or events that seem to be independent, especially if it seems as if the occurrence defies the odds.

Dramatic experiences leave a much more indelible mark on our minds than any other. Therefore it’s only natural to remember unusual experiences and forget routine ones.

Superstitions

Belief in superstitions is further driven by both a need for control and feeling a lack of control. Research shows that the more people feel at the mercy of forces beyond their control, the more they reach for superstitious beliefs to provide the illusion of control.

The Monty Hall Problem

SCIENCE AND PSEUDOSCIENCE

Bem took standard psychological experiments and simply reversed their order. For example, he tested subjects’ ability to recall words and then let them practice with a set of words. He claims that words were easier to recall if they were studied afterward—so the effect of studying would have had to travel back in time to influence performance. ESP proponents were thrilled. Bem’s studies followed rigorous scientific protocols, were published in a peer-reviewed journal, and showed impressive statistical significance. Skeptics were not impressed—reversing the arrow of time is as close to impossible as you get in science, and at the very least it is an extraordinary claim. It’s much more likely that Bem simply made an error than that he overturned a fundamental scientific principle. The psychological and wider scientific communities, however, were worried. Bem followed all the rules, and yet his results could not be true. That implies there might be something wrong with the rules—with science itself. Everyone hoped that the conflict would be resolved with replications, which are the ultimate test of a scientific finding. If the effect is real, then anyone can find it. Most attempts at replicating Bem’s studies (which, to his credit, he made easy by providing documentation for all of his methods) were negative. However, a few ESP proponents were able to replicate his results. In the end, nothing was resolved—each side had its replications to point to: ESP proponents argued that...

19.

Skeptics’ Guide Entry: Methodological Naturalism and Its Critics

Section: Science and Pse...

See also: Materialism, Po...

Methodological naturalism is the philosophical basis for scientific methodology that proceeds as if the universe follows natural laws in ...

It’s an important concept to understand for several reasons. First, it is vital to be aware of implicit philosophical positions that you take: If you advocate for science, you need to understand the philosophical underpinnings of science. Second, there are those (as we’ll see below) who oppose or deny science by attacking those philosophical underpinnings. You can’t recognize this strategy or defend against it if you don’t know the philosophy.

Materialism

Put simply, it is the philosophical position that all physical effects have physical causes.

Materialism stands specifically in opposition to dualism and other philosophies that posit a spiritual or nonmaterial aspect of existence.

Any viable modern definition of materialism must include energy, forces, space-time, dark matter, and possibly dark energy—and anything else discovered by science to exist in nature.

In this way materialism is really just a manifestation of naturalism, the philosophy that says that nature (in all of its aspects) is all there is—there is nothing supernatural.

Another way to look at it is this: If scientists discover that something previously believed to be supernatural exists, then it will become natural, as it will have been demonstrated to be part of nature.

Some argue that materialism is therefore a useless tautology (a definition that references itself), but this misses the point. The definition of materialism is really more about method; it’s about testable causes that we can investigate scientifically. “Supernatural,” therefore, is untestable magic.

Methodological Naturalism