The Art of Statistics: Learning from Data

by David Spiegelhalter

Essentially Bayes is making use of the information about how the position of the line has been initially decided, since we know it is picked at random by throwing the white ball. This initial information takes the same role as the prevalence used in breast screening or dope-testing – it is known as prior information and it influences our final conclusions.

The Bayes’ estimate can never be 0 or 1, and is always nearer to ½ than the simple proportion: this is known as shrinkage, in that estimates are always pulled in or shrunk, towards the centre

Bayesian analysis uses knowledge about how the position of the dashed line was decided to establish a prior distribution for its position, combines it with evidence from the data known as the likelihood, to give a final conclusion known as the posterior distribution, which expresses all we currently believe about the unknown quantity.

The main controversy about Bayesian analysis is the source of the prior distribution. In Bayes’ billiard table, the white ball was thrown at random on to the table and so everyone would agree that the prior distribution is uniformly spread over the whole line between 0 and 1.

When this kind of physical knowledge is unavailable, suggestions for obtaining prior distributions include using subjective judgement, learning from historical data, and specifying objective priors that try to let the data speak for themselves without introducing subjective judgement.

Perhaps the most important insight is that there is no ‘true’ prior distribution, and any analysis should include a sensitivity analysis to a number of alternative choices, enco...

The Bayesian response to this problem is known as multi-level regression and post-stratification (MRP). The basic idea is to break down all possible voters into small ‘cells’, each comprising a highly homogeneous group of people – say living in the same area, with the same age, gender, past voting behaviour, and other measurable characteristics. We can use background demographic data to estimate the number of people in each of these cells, and these are all assumed to have the same probability of voting for a certain party. The problem is working out what this probability is, when our non-random data may mean that we only have a few people, or maybe none at all, in a particular cell.

Bayesian learning is also now seen as a fundamental process of human awareness of the environment, in that we have prior expectations about what we will see in any context, and then only need to take notice of unexpected features in our vision which are then used to update our current perceptions. This is the idea behind the so-called Bayesian Brain.6 The same learning procedures have been implemented in self-driving cars, which have a probabilistic ‘mental map’ of their surroundings that is constantly being updated by recognition of traffic lights, people, other cars, and so on: ‘In essence, a robot car “thinks” of itself as a blob of probability, traveling down a Bayesian road.

But the argument about the ‘correct’ way to do statistical inference is even more complex than a simple dispute between frequentists and Bayesians. Just like political movements, each school splits into multiple factions who have often been in conflict with each other.

In contrast, the Neyman–Pearson approach, which as we have seen was known as ‘inductive behaviour’, was very much focused on decision-making: if you decide the true answer is in a 95% confidence interval, then you will be right 95% of the time, and you should control Type I and Type II errors when hypothesis testing.

The problems that arise in statistical science do not generally come from the philosophy underlying the precise methods that are used. Instead, they are more likely to be due to inadequate design, biased data, inappropriate assumptions and, perhaps most important, poor scientific practice. And in the next chapter we shall take a look at this dark side of statistics.

When the prior distribution comes from some physical sampling process, Bayesian methods are uncontroversial. However generally a degree of judgement is necessary.

This points to an important bias in the scientific literature: a study which has found something ‘big’, at least some of which is likely to have been luck, is likely to lead to a prominent publication. In an analogy to regression to the mean, this might be termed ‘regression to the null’, where early exaggerated estimates of effects later decrease in magnitude towards the null hypothesis.

Then the Planning can go wrong, for example by Choosing a sample that is convenient and inexpensive rather than representative, for example telephone polls before elections.

Asking leading questions or using misleading wording in surveys, such as ‘How much do you think you can save by buying online?

Failing to make a fair comparison, such as assessing homeopathy by only observing vo...

Designing a study that is too small and so has low power, which means that fewer true alternat...

Failing to collect data on potential confounders, lack of blinding in random...

Activities that are intended to create statistically significant results have come to be known as ‘P-hacking’, and although the most obvious technique is to carry out multiple tests and report the most significant, there are many more subtle ways in which researchers can exercise their degrees of freedom.

pharmaceutical companies, in particular, have been accused in the past of hiding studies whose outcomes did not suit them. This leaves valuable data sitting in the ‘file drawer’, and creates a positive bias to what appears in the literature. We do not know what we are not being told.

A fine example of imaginative storytelling happened when a worthy but rather dull study found that 10% of people carried a gene which protected them against high blood pressure. The communications team reframed this as ‘nine in ten people carry a gene which increases the risk of high blood pressure’: this negatively-framed message duly received international press coverage.13

few newspaper readers realize that the person who wrote the article generally has minimal control over the headline, and headlines are of course there to attract readers.

The gripping headline ‘Why Binge Watching Your TV Box-Sets Could Kill You’ arose from an epidemiological study that estimated an adjusted relative risk of 2.5 for a fatal pulmonary embolism associated with watching more than 5 hours TV a night compared with less than 2.5 hours. But careful scrutiny of the absolute rate in the high-risk group (13 in 158,000 person-years) could be translated as meaning you can expect to watch more than 5 hours TV a night for 12,000 years before experiencing the event. This somewhat lessens the impact.14

Given the examples in the last chapter, it should come as no surprise that many of the suggestions in the manifesto concern statistical practice, and in particular the appeal to pre-register studies is intended to guard against the kind of behaviour that was so vividly illustrated in the last chapter, where the design, hypotheses and analysis of a study were adapted to the data as it arrived.

But she also points out that evidence of trustworthiness is required, and this means being transparent – not by just dumping masses of data on audiences, but being ‘intelligently transparent’.9 This means that claims based on data need to be: Accessible: audiences should be able to get at the information. Intelligible: audiences should be able to understand the information. Assessable: if they wish to, audiences should be able to check the reliability of the claims. Useable: audiences should be able to exploit the information for their needs.

Ten Questions to Ask When Confronted by a Claim Based on Statistical Evidence