Psychology: Everything Matters (more or less, or hardly at all)

Some psychologists find ways to find significance in lots and lots of places, says this study:

“False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Joseph P. Simmons, Leif D. Nelson and Uri Simonsohn, Psychological Science, November 2011, vol. 22, no. 11, pp. 1359-1366. (Thanks to investigators Don Shearman and Neil Martin for bringing this to our attention.) the authors, at the University of Pennsylvania and the University of California, Berkeley, explain:

“we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis.”

Here’s detail from the study, showing one trick (which some people in the trade may not even realize is a trick) of the trade: