The Art of Statistics: Learning from Data

by David Spiegelhalter

But there is a long history of claimed margins of error from such experiments later being found to be hopelessly inadequate: in the first part of the twentieth century, the uncertainty intervals around the estimates of the speed of light did not include the current accepted value. This has led organizations that work on metrology, the science of measurement, to specify that margins of error should always be based on two components: Type A: the standard statistical measures discussed in this chapter, which would be expected to reduce with more observations. Type B: systematic errors that would not be expected to reduce with more observations, and have to be handled using non-statistical means such as expert judgement or external evidence. These insights should encourage us to have humility about the statistical methods we can bring to a single data source. If there are fundamental problems with the way the data have been collected then no amount of clever methods can eliminate these biases, and we have to use our background knowledge and experience to temper our conclusions.

Type a and type b od standard errors