The Art of Statistics: Learning from Data

by David Spiegelhalter

In Chapter 6 we saw that an algorithm might win a prediction competition by a very small margin. When predicting the survival of the Titanic test set, for example, the simple classification tree achieved the best Brier score (average mean squared prediction error) of 0.139, only slightly lower than the score of 0.142 from the averaged neural network (see Table 6.4). It is reasonable to ask whether this small winning margin of −0.003 is statistically significant, in the sense of whether or not it could be explained by chance variation. This is straightforward to check, and the t-statistic turns out to be −0.54, with a two-sided P-value of 0.59.fn4 So there is no good evidence that the classification tree is truly the best algorithm! This type of analysis is not routine in Kaggle-type competitions, but it seems important to know that the winning status depends on the chance selection of cases in the test set. Researchers spend their lives scrutinizing the type of computer output shown in Table 10.5, hoping to see the twinkling stars indicating a significant result which they can then feature in their next scientific paper. But, as we now see, this sort of obsessive searching for statistical significance can easily lead to delusions of discovery.

Neural network efficiency may be stastically insignificant