The Art of Statistics: Learning from Data

by David Spiegelhalter

Activities that are intended to create statistically significant results have come to be known as ‘P-hacking’, and although the most obvious technique is to carry out multiple tests and report the most significant, there are many more subtle ways in which researchers can exercise their degrees of freedom. Does listening to the Beatles’ song ‘When I’m Sixty-Four’ make you younger? You might feel fairly confident about the correct answer to this question. Which makes it all the more impressive that Simonsohn and colleagues managed, admittedly by some fairly devious means, to get a significant positive result.9 University of Pennsylvania undergraduates were randomized to listen to either ‘When I’m Sixty-Four’ by the Beatles, or ‘Kalimba’, or ‘Hot Potato’ by the Wiggles. Then the students were asked when they were born, how old they felt, and a series of delightfully irrelevant questions.fn3 Simonsohn and his colleagues repeatedly analysed the data in every way they could think of, and kept enrolling participants until they found some kind of significant association. This happened after 34 subjects, and although there was no significant relationship between the age of the participants and the record they listened to, by only comparing ‘When I’m Sixty-Four’ and ‘Kalimba’, they managed to get P < 0.05 in a regression that adjusted for father’s age. Naturally, they only reported the significant analysis without at first mentioning the vast number of tweaks, fiddles and selective ...

HARking