Statistics Done Wrong: The Woefully Complete Guide

by Alex Reinhart

The first principle is that you must not fool yourself, and you are the easiest person to fool. —Richard P. Feynman

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of. —R. A. Fisher

“torture the data until it confesses.”

In science, it is important to limit two kinds of errors: false positives, where you conclude there is an effect when there isn’t, and false negatives, where you fail to notice a real effect. In some sense, false positives and false negatives are flip sides of the same coin. If we’re too ready to jump to conclusions about effects, we’re prone to get false positives; if we’re too conservative, we’ll err on the side of false negatives.

Papers usually explain the statistical analysis performed but don’t always explain why researchers chose one method over another or what the results would have been had they chosen a different method. Researchers are free to choose whatever methods they feel appropriate—and though they may make good choices, what happens if they analyze the data differently?

Unlike in basketball, there is no academic credit for assists; if you won’t get coauthor credit, why bother sharing the data with anyone?