The Art of Statistics: Learning from Data

by David Spiegelhalter

Ten Questions to Ask When Confronted by a Claim Based on Statistical Evidence HOW TRUSTWORTHY ARE THE NUMBERS? How rigorously has the study been done? For example, check for ‘internal validity’, appropriate design and wording of questions, pre-registration of the protocol, taking a representative sample, using randomization, and making a fair comparison with a control group. What is the statistical uncertainty / confidence in the findings? Check margins of error, confidence intervals, statistical significance, sample size, multiple comparisons, systematic bias. Is the summary appropriate? Check appropriate use of averages, variability, relative and absolute risks. HOW TRUSTWORTHY IS THE SOURCE? How reliable is the source of the story? Consider the possibility of a biased source with conflicts of interest, and check publication is independently peer-reviewed. Ask yourself, ‘Why does this source want me to hear this story?’ Is the story being spun? Be aware of the use of framing, emotional appeal through quoting anecdotes about extreme cases, misleading graphs, exaggerated headlines, big-sounding numbers. What am I not being told? This is perhaps the most important question of all. Think about cherry-picked results, missing information that would conflict with the story, and lack of independent comment. HOW TRUSTWORTHY IS THE INTERPRETATION? How does the claim fit with what else is known? Consider the context, appropriate comparators, including historical data, and what othe...

10 questions to asmk when assessing statistkcs based claims