The Art of Statistics: Learning from Data

by David Spiegelhalter

This claim, partly based on the ‘Bayesian’ reasoning outlined in the next chapter, has led a prominent group of statisticians to argue that the standard threshold for a ‘discovery’ of a new effect should be changed to P < 0.005.11 What effect might this have? Changing the criterion for ‘significance’ from 0.05 (1 in 20) to 0.005 (1 in 200) in Figure 10.5 would mean that instead of having 45 false-positive ‘discoveries’, we would have only 4.5. This would reduce the total number of discoveries to 84.5, and of these only 4.5 (5%) would be false discoveries. Which would be a considerable improvement from 36%. Fisher’s original idea for testing hypotheses has been of great benefit to the practice of statistics and the prevention of unjustified scientific claims. But statisticians have frequently complained about the willingness of some researchers to lurch casually from P-values taken from poorly designed studies to confident generalizable inferences: a kind of alchemy for turning uncertainty into certainty, mechanically applying statistical tests to split all results into ‘significant’ and ‘non-significant’. We shall see some of the poor consequences of this behaviour in Chapter 12, but first turn to an alternative approach to statistical inference that entirely rejects the whole idea of null-hypothesis significance testing. So, as another mind-stretching requirement of statistical science, it would help if you could (temporarily) forget everything you may have learned from t...

P shud be 1 in 200 vs 1 in 20