The Art of Statistics: Learning from Data

by David Spiegelhalter

So let’s put all this together in Bayes’ theorem, which simply says that the initial odds for a hypothesis × the likelihood ratio = the final odds for the hypothesis For the doping example, the initial odds for the hypothesis ‘the athlete is doping’ is 1/49, and the likelihood ratio is 19, so Bayes’ theorem says the final odds are given by 1/49 × 19 = 19/49 These odds of 19/49 can be transformed to a probability of 19/(19+49) = 19/68 = 28%. So this probability, which was obtained from the expected frequency tree in a rather simple way, can also be derived from the general equation for Bayes’ theorem. In more technical language, the initial odds are known as the ‘prior’ odds, and the final odds are the ‘posterior’ odds. This formula can be repeatedly applied, with the posterior odds becoming the prior odds when introducing new, independent items of evidence. When combining all the evidence, this process is equivalent to multiplying the independent likelihood ratios together to form a composite likelihood ratio. Bayes’ theorem looks deceptively basic, but turns out to encapsulate an immensely powerful way of learning from data.

Bayes odds

This value of 3/7 may seem odd, as the intuitive estimate might be 2/5 – the proportion of red balls landing to the left of the line. Instead Bayes showed that in these circumstances we should estimate the position as This means, for example, that before any red balls are thrown at all, we can estimate the position to be (0 + 1)/(0 + 2) = ½, whereas the intuitive approach might suggest that we could not give any answer since there is not yet any data. Essentially Bayes is making use of the information about how the position of the line has been initially decided, since we know it is picked at random by throwing the white ball. This initial information takes the same role as the prevalence used in breast screening or dope-testing – it is known as prior information and it influences our final conclusions. In fact, since Bayes’ formula adds one to the number of red balls to the left of the line and two to the total number of red balls, we might think of it as being equivalent to having already thrown two ‘imaginary’ red balls, and one having landed each side of the dashed line. Note that if none of the five balls had landed to the left of the dashed line, we would not have estimated its position as 0/5, but instead as 1/7, which seems much more sensible. The Bayes’ estimate can never be 0 or 1, and is always nearer to ½ than the simple proportion: this is known as shrinkage, in that estimates are always pulled in or shrunk, towards the centre of the initial distribution, in t...

Bayesian application

The Bayesian response to this problem is known as multi-level regression and post-stratification (MRP). The basic idea is to break down all possible voters into small ‘cells’, each comprising a highly homogeneous group of people – say living in the same area, with the same age, gender, past voting behaviour, and other measurable characteristics. We can use background demographic data to estimate the number of people in each of these cells, and these are all assumed to have the same probability of voting for a certain party. The problem is working out what this probability is, when our non-random data may mean that we only have a few people, or maybe none at all, in a particular cell. The first step is to build a regression model for the probability of voting a particular way given the characteristics of the cell, so our problem is reduced to estimating the coefficients of the regression equation. But there are still too many coefficients to estimate reliably using the standard methods, and this is where Bayesian ideas come in. Coefficients corresponding to different areas are assumed to be similar, a sort of intermediate point between assuming they are precisely the same, and assuming they are utterly unrelated. Mathematically, this assumption can be shown to be equivalent to assuming all these unknown quantities have been drawn from the same prior distribution, and this enables us to move many individual, rather imprecise, estimates towards each other, which results in sm...

MRP FOR POLL research usi ng baysian approach

Robert Kass and Adrian Raftery are two renowned Bayesian statisticians who proposed a widely used scale for Bayes factors, shown in Table 11.3. Note the contrast to the scale in Table 11.2 for verbally interpreting likelihood ratios for legal cases, where a likelihood ratio of 10,000 was required to declare the evidence as ‘very strong’, in contrast to scientific hypotheses only needing a Bayes factor of greater than 150. This perhaps reflects the need to establish criminal guilt ‘beyond reasonable doubt’, whereas scientific claims are made on weaker evidence, with many being overturned on further research. Our chapter on hypothesis testing contained a claim that a P-value of 0.05 was only equivalent to ‘weak evidence’. The reasoning for this is partly based on Bayes factors: P = 0.05 can be shown to correspond, under some reasonable priors under the alternative hypothesis, to Bayes factors between 2.4 and 3.4, which Table 11.3 suggests is weak evidence. As we saw in Chapter 10, this led to a proposed reduction in the necessary P-value for claiming a ‘discovery’ to 0.005. Unlike null-hypothesis significance testing, Bayes factors treat the two hypotheses symmetrically, and so can actively support a null hypothesis. And if we are willing to put prior probabilities on hypotheses, we might even calculate posterior probabilities of alternative theories for how the world works. Suppose, based on theoretical grounds alone, we judged it 50:50 whether the Higgs boson existed, corr...

Bayes factor vs P value

Activities that are intended to create statistically significant results have come to be known as ‘P-hacking’, and although the most obvious technique is to carry out multiple tests and report the most significant, there are many more subtle ways in which researchers can exercise their degrees of freedom. Does listening to the Beatles’ song ‘When I’m Sixty-Four’ make you younger? You might feel fairly confident about the correct answer to this question. Which makes it all the more impressive that Simonsohn and colleagues managed, admittedly by some fairly devious means, to get a significant positive result.9 University of Pennsylvania undergraduates were randomized to listen to either ‘When I’m Sixty-Four’ by the Beatles, or ‘Kalimba’, or ‘Hot Potato’ by the Wiggles. Then the students were asked when they were born, how old they felt, and a series of delightfully irrelevant questions.fn3 Simonsohn and his colleagues repeatedly analysed the data in every way they could think of, and kept enrolling participants until they found some kind of significant association. This happened after 34 subjects, and although there was no significant relationship between the age of the participants and the record they listened to, by only comparing ‘When I’m Sixty-Four’ and ‘Kalimba’, they managed to get P < 0.05 in a regression that adjusted for father’s age. Naturally, they only reported the significant analysis without at first mentioning the vast number of tweaks, fiddles and selective ...

HARking

Ten Questions to Ask When Confronted by a Claim Based on Statistical Evidence HOW TRUSTWORTHY ARE THE NUMBERS? How rigorously has the study been done? For example, check for ‘internal validity’, appropriate design and wording of questions, pre-registration of the protocol, taking a representative sample, using randomization, and making a fair comparison with a control group. What is the statistical uncertainty / confidence in the findings? Check margins of error, confidence intervals, statistical significance, sample size, multiple comparisons, systematic bias. Is the summary appropriate? Check appropriate use of averages, variability, relative and absolute risks. HOW TRUSTWORTHY IS THE SOURCE? How reliable is the source of the story? Consider the possibility of a biased source with conflicts of interest, and check publication is independently peer-reviewed. Ask yourself, ‘Why does this source want me to hear this story?’ Is the story being spun? Be aware of the use of framing, emotional appeal through quoting anecdotes about extreme cases, misleading graphs, exaggerated headlines, big-sounding numbers. What am I not being told? This is perhaps the most important question of all. Think about cherry-picked results, missing information that would conflict with the story, and lack of independent comment. HOW TRUSTWORTHY IS THE INTERPRETATION? How does the claim fit with what else is known? Consider the context, appropriate comparators, including historical data, and what othe...

10 questions to asmk when assessing statistkcs based claims

These ‘rules’ should be fairly self-evident, and rather neatly summarize the issues tackled in this book. Statistical methods should enable data to answer scientific questions: Ask ‘why am I doing this?’, rather than focusing on which particular technique to use. Signals always come with noise: It is trying to separate out the two that makes the subject interesting. Variability is inevitable, and probability models are useful as an abstraction. Plan ahead, really ahead: This includes the idea of pre-specification in confirmatory experiments – avoiding researcher degrees of freedom Worry about data quality: Everything rests on the data. Statistical analysis is more than a set of computations: Do not just plug into formulae or run procedures in software, without knowing why you are doing so. Keep it simple: The main communication should be as basic as possible – do not show off skills in complex modelling unless they are really necessary. Provide assessments of variability: With the warning that margins of error are generally bigger than claimed. Check your assumptions: And make clear when this has not been possible. When possible, replicate!: Or encourage others to do so. Make your analysis reproducible: Others should be able to access your data and code.

Good sttistics based science rules