The Art of Statistics: Learning from Data

by David Spiegelhalter

Theory shows that the normal distribution can be expected to occur for phenomena that are driven by large numbers of small influences, for example a complex physical trait that is not influenced by just a few genes.

We see that the measures used to summarize data sets in Chapter 2 can be applied as descriptions of a population too – the difference is that terms such as mean and standard deviation are known as statistics when describing a set of data, and parameters when describing a population.

A great advantage of assuming a normal form for a distribution is that many important quantities can be simply obtained from tables or software.

Figure 3.2(d) shows that we would expect 1.7% of babies in this group to be low birth weight – in fact the actual number was 14,170 (1.3%), in close agreement with the prediction from the normal curve.

So a population can be thought of as a physical group of individuals, but also as providing the probability distribution for a random observation. This dual interpretation will be fundamental when we come to more formal statistical inference

But what is the study population? We have data on all the children and all the hospitals, and so there is no larger group from which they have been sampled. Although the idea of a population is usually introduced rather casually into statistics courses, this example shows it is a tricky and sophisticated idea that is worth exploring in some detail, as a lot of important ideas build on this concept.

There are three types of populations from which a sample might be drawn, whether the data come from people, transactions, trees, or anything else.

literal population. This is an identifiable group, such as when we pick a person...

A virtual population. We frequently take measurements using a device, such as taking someone’s blood pressure or measuring air pollution. We know we could always take more measurements and get a slightly different answer, as you will know if you have ever taken repeat blood pressure measurements. The closeness of the multiple readings depends on the precision of the device and the stability of the circumstances – we might think of this as drawing observations from a virtual population of all the measurements that could be taken if we had enough time.

A metaphorical population, when there is no larger population at all.

Think of the number of murders that occur each year, the examination results for a particular class, or data on all the countries of the world – none of these can be considered as a sample from an actual population.

Nevertheless it is extremely valuable to keep hold of the idea of an imaginary population from which our ‘sample’ is drawn, as then we can use all the mathematical techniques that have been developed for sampling from real populations.

And to give them credit, the authors of the paper doubted it too, adding, ‘Completeness of cancer registration and detection bias are potential explanations for the findings.’ In other words, wealthy people with higher education are more likely to be diagnosed and get their tumour registered, an example of what is known as ascertainment bias in epidemiology.

There is even a word for the tendency to construct reasons for a connection between what are actually unrelated events – apophenia – with the most extreme case being when simple misfortune or bad luck is blamed on others’ ill-will or even witchcraft.

Causation is a deeply contested subject, which is perhaps surprising as it seems rather simple in real life: we do something, and that leads to something else. I jammed my thumb in the car door, and now it hurts.

For example, the medical community now agrees that smoking cigarettes causes lung cancer, but it took decades for doctors to come to this conclusion. Why did it take so long? Because most people who smoke do not get lung cancer. And some people who do not smoke do get lung cancer. All we can say is that you are more likely to get lung cancer if you smoke than if you do not smoke, which is one reason why it took so long for laws to be enacted to restrict smoking.

So our ‘statistical’ idea of causation is not strictly deterministic. When we say that X causes Y, we do not mean that every time X occurs, then Y will too. Or that Y will only occur if X occurs. We simply mean that if we intervene and force X to occur, then Y tends to happen more often. So we can never say that X caused Y in a specific case, only that X increases the proportion of times that Y happens.

A proper medical trial should ideally obey the following principles:

We need an intervention group, who will be given statins, and a control group who will be given sugar pills or placebos

Allocation of treatment: It is important to compare like with like, so the treatment and comparison groups have to be as similar as possible.

People should be counted in the groups to which they were allocated: The people allocated to the ‘statin’ group in the Heart Protection Study (HPS) were included in the final analysis even if they did not take their statins. This is known as the ‘intention to treat’ principle, and can seem rather odd. It means that the final estimate of the effect of statins really measures the effect of being prescribed statins rather than actually taking them.

If possible, people should not even know which group they are in:

Groups should be treated equally: If the group allocated to statins were invited back for more frequent hospital appointments, or examined more carefully, it would be impossible to separate the benefits of the drug from the benefits of increased general care.

If possible, those assessing the final outcomes should not know which group the subjects are in:

Measure everyone: Every effort must be made to follow everyone up, as people who drop out of the study might, for example, have done so because of the drug’s side effects.

Don’t rely on a single study

Review the evidence systematically: When looking at multiple trials, make sure to include every study that has been done, and so create what is known as a systematic review. The results may then be formally combined in a meta-analysis.

The only apparent effect was a small increase in complications in the group that knew they were being prayed for: one of the researchers commented, ‘It may have made them uncertain, wondering, “Am I so sick they had to call in their prayer team?”.’

The challenge is then to try to explain this association. Do ears carry on growing with age? Or did people who are old now always have bigger ears, and something has happened over the last decades to make more recent generations have smaller ears? Or is it that men with smaller ears simply die earlier for some reason; there is a traditional Chinese belief that big ears predict a longer life.

Similarly any correlation between ice-cream sales and drownings is due to both being influenced by the weather. When an apparent association between two outcomes might be explained by some observed common factor that influences both, this common cause is known as a confounder: both the year and weather are potential confounders since they can be recorded and considered in an analysis.

Take a careful look at Table 4.2. Although overall the acceptance rate was higher for men, the acceptance rate in each subject individually was higher for women. How can this apparent paradox occur? The explanation is that the women were more likely to apply for the more popular and therefore more competitive subjects with the lowest acceptance rate, such as medicine and veterinary medicine, and tended not to apply to engineering, which has a higher acceptance rate. In this case, therefore, we might conclude that there is no evidence of discrimination.

This is known as Simpson’s paradox, which occurs when the apparent direction of an association is reversed by adjusting for a confounding factor, requiring a complete change in the apparent lesson from the data.

The claim that a nearby Waitrose ‘adds £36,000 to house price’ was credulously reported by the British media in 2017.

The correlation almost certainly reflects Waitrose’s policy of opening stores in wealthier locations, and is therefore a fine example of the actual chain of causation being the precise opposite of what has been claimed. This is known, unsurprisingly, as reverse causation

Another amusing exercise is to try to invent a narrative of reverse causation for any statistical claim based on correlation alone.

Potential common causes that we do not measure are known as lurking factors, since they remain in the background, are not included in any adjustment, and are waiting to trip up naïve conclusions from observational data.

The myriad ways we can be caught out might encourage the idea that we can never conclude causation from anything other than a randomized experiment. But, perhaps ironically, this view was counteracted by the man responsible for the first modern randomized clinical trial.

These criteria have been subsequently much debated, and the version shown below was developed by Jeremy Howick and colleagues, separated into what they call direct, mechanistic and parallel evidence.12

Direct evidence: The size of the effect is so large that it cannot be explained by plausible confounding.

There is appropriate temporal and/or spatial proximity, in that cause precedes effect and effect occurs after a plausible interval, and/or caus...

Dose responsiveness and reversibility: the effect increases as the exposure increases, and the evidence is even stronger if the effec...

There is a plausible mechanism of action, which could be biological, chemical, or mechanical, with external evidence for a ‘causal chain’.

Parallel evidence: The effect fits with what is known already. The effect is found when the study is replicated. The effect is found in similar, but not identical, studies.

But some courts have accepted that, on the ‘balance of probabilities’, a direct causal link has been established if the relative risk associated with the exposure is greater than two. But why two? Presumably the reasoning behind this conclusion is as follows:

This kind of argument has led to a new area of study known as forensic epidemiology, which tries to use evidence derived from populations to draw conclusions about what might have caused individual events to occur. In effect this discipline has been forced into existence by people seeking compensation, but this is a very challenging area for statistical reasoning about causation.

Other advanced statistical methods have been developed to try to adjust for potential confounders and so to get closer to an estimate of the actual effect of the exposure, and these are largely based on the important idea of regression analysis. And for this we must acknowledge, yet again, the fertile imagination of Francis Galton.

Causation, in the statistical sense, means that when we intervene, the chances of different outcomes are systematically changed.

Observational data may have background factors influencing the apparent observed relationships between an exposure and an outcome, which may be either observed confounders or lurking factors.

this chapter we meet the important idea of a statistical model, which is a formal representation of the relationships between variables, which we can use for the desired explanation or prediction.

For any straight line we choose, each data-point will give rise to a residual (the vertical dashed lines on the plot), which is the size of the error were we to use the line to predict a son’s height from his father’s. We want a line that makes these residuals small, and the standard technique is to choose a least-squares fitted line, for which the sum of the squares of the residuals is smallest.