The Art of Statistics: Learning from Data

by David Spiegelhalter

To summarize, I have described the following steps: Set up a question in terms of a null hypothesis that we want to check. This is generally given the notation H0. Choose a test statistic that estimates something that, if it turned out to be extreme enough, would lead us to doubt the null hypothesis (often larger values of the statistic indicate incompatibility with the null hypothesis). Generate the sampling distribution of this test statistic, were the null hypothesis true. Check whether our observed statistic lies in the tails of this distribution and summarize this by the P-value: the probability, were the null hypothesis true, of observing such an extreme statistic. The P-value is therefore a particular tail-area. ‘Extreme’ has to be defined carefully – if say both large positive and large negative values of the test statistic would have been considered incompatible with the null hypothesis, then the P-value has to take this into account. Declare the result statistically significant if the P-value is below some critical threshold. Ronald Fisher used P < 0.05 and P < 0.01 as convenient critical thresholds for indicating significance, and produced tables of the critical values of test statistics needed to achieve these levels of significance. The popularity of these tables led to 0.05 and 0.01 becoming established conventions, although it is now recommended that exact P-values should be reported. And it is important to emphasize that the exact P-value is conditional not...

Null hypothessis significance test