How to Make the World Add Up : Ten Rules for Thinking Differently About Numbers

by Tim Harford

a statistical metric may be a pretty decent proxy for something that really matters, but it is almost always a proxy rather than the real thing. Once you start using that proxy as a target to be improved, or a metric to control others at a distance, it will be distorted, faked or undermined. The value of the measure will evaporate.

GDP: used as a proxy for welfare. But could using other stats in a Prosperity index then create unintended statistical pressures?

For instance, when testing a drug, your statistical analysis begins with the assumption that the drug does not work; when you observe that lots of the patients taking the drug are doing much better than the patients who are taking a placebo, you revise that assumption. In general, if the chances of randomly observing data at least as extreme as you collect are less than 5 per cent,

the results are ‘significant’ enough to overturn the assumption: we can conclude with a sufficient degree of confidence that the drug works, large displays of jam discourage people from buying jam, and that precognition exists.

The problems are obvious. 5 per cent is an arbitrary cut-off point – why not 6 per cent, or 4 per cent? – and it encourages us to think in black-and-white, pass-or-fail terms, instead of embracing degrees of uncertainty. And if you found the previous paragraph confusing, I don’t blame you. Conceptually, statistical significance is baffling, almost backwards: it tells us the chance of observing the data given a particular theory, the theory that there is no effect. Really, we’d like to know the opposite, the probability of a part...

The Cochrane Library, by contrast, aims to provide an accessible summary of everything we know about yoga and incontinence – if anything. It’s also on the first page of Google search results. Cochrane is not a secret.

Alchemy is not the same as gathering big datasets and developing pattern-recognising algorithms. For one thing, alchemy is impossible, and deriving insights from big data is not. Yet the parallels should also be obvious. The likes of Google and Target are no more keen to share their datasets and algorithms than Newton was to share his alchemical experiments. Sometimes there are legal or ethical reasons – if you’re trying to keep your pregnancy a secret, you don’t want Target publicly disclosing your folic acid purchases – but most obviously the reasons are commercial. There’s gold in the data that Amazon, Apple, Facebook, Google and Microsoft have about us. And that gold will be worth a lot less to them if the knowledge that produces it is shared with everyone.

What makes a superforecaster? Not subject-matter expertise: professors were no better than well-informed amateurs. Nor was it a matter of intelligence, otherwise Irving Fisher would have been just fine. But there were a few common traits among the better forecasters.

First, encouragingly for us nerds, it did help to have some training – of a particular kind. Just an hour of training in basic statistics improved the performance of forecasters by helping them turn their expertise about the world into a sensible probabilistic forecast, such as ‘the chance that a woman will be elected President of the US within the next ten years is 25 per cent’. The tip that seemed to help most was to encourage them to focus on something called ‘base rates’.13

The importance of the base rate was made famous by the psychologist Daniel Kahneman, who coined the phrase ‘the outside view and the inside view’. The inside view means looking at the specific case in front of you: this couple. The outside view requires you to look at a more general ‘comparison class’ of cases – here, the comparison class is all married couples. (The outside view needn’t be statistical, but it often will be.) Ideally, a decision-maker or a forecaster will combine the outside view and the inside view – or, similarly, statistics plus personal experience. But it’s much better to start with the statistical view, the outside view, and then modify it in the light of personal experience than it is to go the other way around.

Second, keeping score was important. As Tetlock’s intellectual predecessors Fischhoff and Beyth had demonstrated, we find it challenging to do something as simple as remembering whether our earlier forecasts were right or wrong. Third, superforecasters tended to update their forecasts frequently as new information emerged, which suggests that a receptiveness to new evidence was important. This willingness to adjust predictions is correlated with making better predictions in the first place:

Which points to the fourth and perhaps most crucial element: superforecasting is a matter of having an open-minded personality. The superforecasters are what psychologists call ‘actively open-minded thinkers’ – people who don’t cling too tightly to a single approach,

‘As time goes on, I get more and more convinced that the right method in investment is to put fairly large sums into enterprises which one thinks one knows something about and in the management of which one thoroughly believes.’

Forget what the economy is doing; just find well-managed companies, buy some shares, and don’t try to be too clever. And if that approach sounds familiar, it’s most famously associated with Warren Buffett, the world’s richest investor – and a man who loves to quote John Maynard Keynes.