Tim Harford's Blog, page 120

How to give feedback

I recently spoke at Wired 2012 and I felt it went well (video to follow, when they put it up).

Afterwards, people came up, shook my hand, patted me on the back and told me I did a great job. That felt nice, but it won’t help me to do a better job next time.

Elsewhere in the building, other people gathered in corners and grumbled about all the things I did or got wrong. (I don’t know if this happened. I assume it did. You can’t please everyone.) That didn’t help me to do a better job next time, either.

But someone did something helpful. Bruno Giussani of TED, seeing someone praise me for speaking without slides,  immediately got to the point. “You talked about the Spitfire,” he said, “But this is an international audience. Many people won’t know what you’re talking about. You should have shown just one slide: a photograph of a Spitfire. Then everyone would have understood.”

Next time I give a similar speech, I’ll be showing one slide: a photograph of a Spitfire.

It isn’t easy to get straight to the point and offer a single, focused suggestion for improvement. And the truth is, we rarely seek that kind of feedback. When we ask “what do you think?”, we’re usually looking for those confidence-boosting pats on the back. But giving such feedback – and seeking it out – is hugely important.

(On which topic, Peter Sims has an excellent book, “Little Bets“, which among many excellent topics discusses this kind of focused feedback at Pixar. Buy it for Christmas and enjoy.)

Tim Harford at the Sydney Opera House: “Make more Misstakes”

“In a complex and fast-moving world, if we want to move ahead in leaps and bounds, rather than in small steps, we need to rethink the conditions for making progress in science, business and society in a fundamental way. We need to realise there is no ‘right way’, lose our fear of failure, embrace opportunity and take risks.

We need to stop looking for leaders who can provide us with all the answers, and encourage the search for many solutions – for that is where we will find the ‘one in a hundred’ that delivers real results. We need to understand that to adapt to the challenges of the future, we must make mistakes, lots of them.”

“” is here – one hour. Enjoy!

Unloading the dice on research

A compulsory register of trials could give a more accurate view of studies and test results

I just rolled a six-sided die a few times. I rolled: six, five, five, six, five. My question is: do you think the die is biased? One way to think about that question is to ask how likely I would be to roll only fives and sixes completely by chance. Not that likely: the odds are 242 to 1 against.

Before you conclude that I have crooked dice, let me mention something that had slipped my mind. As well as those fives and sixes, I also rolled four, three, three, three, two, two and one. But those results just didn’t seem that interesting, so I didn’t tell you about them. If that omission seems relevant, you’re beginning to appreciate the importance of the nerdy-sounding “trial register”.

Every day, across the world, researchers are conducting randomised controlled trials (RCTs). Some are rigorous, painstaking quests for truth, while others may cut a few corners in the search for a career-defining publication, or a licence for a new drug compound. But even if each individual trial was unimpeachable, the results would mean very little if there was a systematic bias in favour of a particular kind of result.

I reported my die-rolling with a bias towards high numbers. You might uncharitably suspect that an industry-sponsored drug trial would be more likely to see the light of day if the results show that the drug works, and you would be correct (a state of affairs ably summarised by Ben Goldacre in his new book Bad Pharma). Trials can also go missing because they end in chaos or disaster: such stories may not be worth an academic paper but they must be recorded. And trials go missing because the results are so boring that the researchers cannot bring themselves to write them up properly for publication.

As my die-rolling shows, unless we see every trial that was begun, we have a distorted picture of what is happening. There is probably only one way to achieve that goal: a compulsory register of trials. Researchers who conduct trials and abandon them or don’t publish, need to become pariahs. Systematic reviews of a particular field will be able to consult the registers and track down any unpublished trials.

A few years ago, the International Committee of Medical Journal Editors announced that the prestigious journals under its control would no longer publish research based on clinical trials unless those trials had been formally registered before they began. This had the very welcome effect of increasing, dramatically, the number of registered trials and the rate at which new trials were registered. Unfortunately, as Sylvain Mathieu and others explained in the Journal of the American Medical Association in 2009, more than half the research they examined flouted that rule and was published anyway. The threat not to publish seems to have been empty.

And what of economics, which has in recent years discovered the joys of randomised trials? There is good news: the American Economic Association is creating a trial register for trials in economics; it is due to be up and running next year. The register will be voluntary for now, but two leading practitioners, Esther Duflo of MIT and Dean Karlan of Yale, both told me they are hopeful that a strong social norm will form in favour of registering trials. We shall see.

Trial registries are a particular challenge in the social sciences. While an RCT in medicine is designed to test whether a specific treatment does or does not work, in the social sciences they are more likely to be used to search for interesting hypotheses. Social science RCTs are often partnerships between academics and practical organisations, and the trial may evolve over time in a way that a clinical trial would not.

All this complicates the business of registering the trial and then updating the entry as things change. It makes a trial registry harder to maintain. But it also makes the registry even more essential.

Also published at ft.com.

A battle for our green and pleasant land

‘There has been no change in the government’s policy on renewable energy, the prime minister has said.’ – BBC News

That’s news?



That is news because it follows hard on the heels of a declaration by John Hayes, the new energy minister, that “enough is enough” as far as onshore wind farms are concerned.

That sounds like a change in the government’s energy policy.



It does indeed. The number of wind farms has been growing steadily, and RenewableUK, the industry body, recently announced that approvals for new onshore wind farms were up for the first time in five years, and sharply so. Mr Hayes, in contrast, was quoted as saying: “I can’t single-handedly build a new Jerusalem but I can protect our green and pleasant land.”

I’m guessing he’s a Tory rather than a Liberal Democrat, somehow.



Given the new Jerusalem reference, I am starting to suspect that Mr Hayes is not a Conservative but instead a comic actor taking the role of a Tory MP and playing it for laughs. Officially Mr Hayes is a Conservative but his boss, Ed Davey, is a Lib Dem and Mr Davey was quick to overrule Mr Hayes.

But what was Mr Hayes planning to do, then, before he was overruled?



He was planning to commission new evidence on which to base his future “enough is enough” policy, suggesting that the existing policy wasn’t based on evidence but on a “bourgeois left article of faith based on some academic perspective”.

I’m confused. How does he know what the new evidence will say before he’s seen it?



It is odd. Mr Hayes is not the only politician to rely on clairvoyant evidence, as the science writer Mark Henderson has pointed out. Politicians have a habit of announcing what the research will discover before the research has been carried out. Scientists, rather quaintly, undertake research because they don’t know what the results will be.

Oh, those pesky scientists. Don’t worry about them. They have an academic perspective, I think you will find.



Apparently so.

So what is the situation with wind power, then?



For a windy country, the UK’s installed capacity is middling at best. In Europe, its capacity is just behind that of Italy and France but dwarfed by Spain and Germany. But Mr Hayes may have reason to be more worried about recent trends: in 2011, the UK installed more capacity than any European country other than Germany.

What about offshore wind? Surely that is no threat to the new Jerusalem.



According to RenewableUK, the UK has more offshore wind capacity installed than the rest of the world put together.

Splendid news!



I wouldn’t be so sure about that. Renewable energy requires substantial subsidy. Why do we subsidise it? Chiefly because of climate change and perhaps because of the fear that fossil fuels will eventually run out. But these issues are both global and long-term, which means that what really matters is to develop technologies that have a chance of being adopted across the world.

And we’re a global leader in offshore wind farms.



I fear that we may remain a global leader in offshore wind farms because so few other countries are interested in the technology, which is an expensive way to pander to the tastes of small, rich coastal areas. I wonder if wind farms optimised for the plains of Mexico or the steppes of Mongolia might be a better bet than trying to figure out how to plant windmills in the Irish Sea and keep them from corroding.

You seem rather cynical about the whole thing.



I am. Climate change is no joke and dealing with it is going to require some big technological leaps. We need a substantial and credible carbon price and a far more radical scheme of funding low-carbon innovation. Instead we have an ugly game of political football. The spat between Mr Hayes and Mr Davey will make renewable energy providers nervous and make it even more expensive for the taxpayer to persuade them to make risky investments.

Do you want these investments or don’t you?



I suspect that the future of low-carbon technology is going to involve a lot more nuclear and geothermal energy and a lot less wind. But I am probably wrong; nobody knows. I am sure of this: it’s expensive enough to build wind farms already without stirring up uncertainty and then having to pay danger money just to persuade investors to brave our shambolic energy policy.

Also published at ft.com.

Why aren’t we doing the maths?

The practical implications of misplaced confidence when dealing with statistical evidence are obvious and worrying

A little two-part test for you. Imagine you’re a doctor, considering whether to recommend a particular kind of cancer screening, “A”. You discover that this form of screening improves five-year survival rates from 68 per cent to 99 per cent. (The five-year survival rate is the proportion of patients alive five years after the cancer was discovered.) The question is: does the screening test “A” save lives?

Part two: now you consider an alternative screening test, “B”. You discover that test “B” reduces cancer deaths from two per 1,000 people to 1.6 per 1,000 people. So: does screening test “B” save lives?

The second question is easier. Screening test “B” unambiguously saves lives: to be precise it saves 0.4 lives per 1,000 people. That might not seem a lot – and if the test is expensive or has unpleasant side-effects it might not be worth it – but that is the nature of cancer screening. Most people don’t have the cancer in question so most people cannot be helped by the test.

What about screening test “A”? This question is harder. The numbers look impressive, but survival rates are a treacherous way to evaluate a screening programme. Imagine a group of 60-year-olds who all develop an incurable cancer that will kill them at 70. They have no symptoms until age 67, so the five-year survival rate when they are diagnosed at 67 is, I’m afraid, zero. Introduce a screening programme and you can discover the cancer much earlier, at age 62. The five-year survival rate is now 100 per cent. But the screening hasn’t saved any lives: it’s merely given early warning of a disease that cannot be treated.

In general, screening programmes look impressive when evaluated by survival rates, because the purpose of screening is to detect the cancer earlier. Whether any lives are saved or not is a different issue entirely.

I’ll admit, this is a tricky pair of questions. You’d have to be a doctor, rigorously trained in how to handle the evidence base for medical treatments, to get this sort of thing right. But here’s the bad news: doctors do not get this sort of thing right.

An article published in the Annals of Internal Medicine in March put these questions to a panel of more than 400 doctors with relevant clinical experience. Eighty-two per cent thought they’d been shown evidence that test “A” saved lives – they hadn’t – and of those, 83 per cent thought the benefit was large or very large. Only 60 per cent thought that test “B” saved lives, and fewer than one-third thought the benefit was large or very large – which is intriguing, because of the few people on course to die from cancer, the test saves 20 per cent of them. In short, the doctors simply did not understand the statistics on cancer screening.

The practical implications of this are obvious and worrying. It seems that doctors may need a good deal of help interpreting the evidence they are likely to be exposed to on clinical effectiveness, while epidemiologists and statisticians need to think hard about how they present their discoveries.

The situation could be worse. A recent survey by the Royal Statistical Society’s “getstats” campaign asked MPs to give the probability of getting two heads when tossing a coin twice. More than half failed to get the answer correct – including a humiliating three-quarters of Labour MPs.

The answer, of course, is 25 per cent, and is appallingly basic stuff. If I try to translate from numeracy to literacy, I’d say that the doctors’ failure was the equivalent of being unable to write a decent essay about “The Waste Land”, while the MPs’ failure was more like the inability to read a newspaper.

The Royal Statistical Society reported that about three-quarters of MPs said they felt confident when dealing with numbers. This confidence is misplaced.

Also published at ft.com.

The benefits of being in this together

‘George Osborne was warned by Tory MPs on Tuesday to expect a voter backlash and “widespread non-compliance” as the chancellor prepares to roll out his plan to remove child benefit from better-off taxpayers.’

Financial Times, October 24



The revolt of the yummy mummy set!

And the relevant fathers, husbands and boyfriends, yes – although such people are rarely described as “yummy”.

Why the fuss?

Where to begin? The idea is that high-income households will no longer receive child benefit, hitherto a universal right. Either the mother can rescind her claim to child benefit, or a high earner in the household can declare child benefit was received and pay an offsetting tax charge. Some worry about the principle of universal benefits being eroded; some worry it is an odd way to raise more tax; and some worry that administratively, the thing is a regurgitated dog’s breakfast.

How can a policy offend people in so many ways?

George Osborne is a man of unusual talents. I speculate that he wrote down a few useful rules of thumb for what a sensible tax system should look like, and then tried to figure out how to break every one of them.

What rules of thumb do you have in mind?

First, keep marginal tax rates simple and as low as possible to avoid discouraging work. Marginal tax rates were proliferating under the chancellor’s predecessors and now we have a new suite of high and fecundity-dependent marginal tax rates for those earning between £50,000 and £60,000 a year.

But isn’t it reasonable that high earners are denied child benefit?

Well, that ignores the second rule of thumb, which is that we should evaluate the tax system as a whole, rather than individual chunks of it. Universal child benefit is already progressive, but if we think the rich should pay more, that’s easy: raise income taxes. Mr Osborne is doing the opposite. Perhaps this is because by targeting this measure on households with children, he shields the over-55s from the worst of it.

Why would he want to do that?

That’s been Conservative party policy for ever. I’ve stopped asking why. Rule number three is the Colbert rule, named after a French finance minister: pluck the goose, but minimise the hissing. Instead, in a display of forehead-smacking incompetence, Mr Osborne plans to write to his core constituents specifically informing them that he is about to take money away they regard as a fundamental right. For an encore he will make an extra half a million people complete a self-assessment tax return, which is no trip to Disneyland. It’s hard to see how the opposition could draw as much attention to this tax increase as Mr Osborne has.



But at least it’s all going to be administratively convenient?

Very droll. The fourth principle is to make up your mind what you’re taxing: households or individuals. Our benefits system applies to households but our tax system has always applied to individuals. The way this child-benefit withdrawal is going to be introduced mixes up the two systems.

Is that a problem?

The obvious objection is that a couple earning £60,000 and £20,000 respectively will lose child benefit, while a couple earning £40,000 each will lose nothing. It also means husbands and wives have to tell each other financial details that for hundreds of years have been regarded as private. More deliciously, HM Revenue & Customs has to take a view on what constitutes “living together as husband and wife (or civil partners)”. This should be fun.

Why so?

If a mother is living with her high-earning husband, no child benefit. If she is a single parent living with her high-earning mother, child benefit is due. If she lives with a high-earning friend, she gets child benefit. If she starts sleeping with the friend, she does not get child benefit. I’ve asked HMRC how they plan to find out who’s having sex with whom.

What did they say?

They said they’d get back to me. I also failed to establish what happens if a mother has two overlapping romantic partners. Nor do we yet know whether tax returns will contain a box to allow people to specify how many weeks they’ve been sexually involved with someone claiming child benefit. It will certainly spice up the old tax return, eh?

Why is the government doing this?

Simple. Mr Osborne thinks that it will be a vote-winner. That appears to be all that matters.

Also published at ft.com.

A short interview at Wired 2012

Click here to view the embedded video.

Believe the hype in hyperinflation

Extremely high price increases are not produced by central bankers but are the result of a total failure of the political system

Despite fringe commentators shrieking about imminent hyperinflation, it has failed to appear. The Fed’s announcement last month of further quantitative easing can be interpreted as an attempt to promise inflation tomorrow in the hope of getting people to spend money today. Rather than creating far too much inflation, the Federal Reserve is struggling to create much inflation at all.

Unexpected inflation moves money from to debtors to from creditors. It creates a variety of minor costs associated with redrafting contracts and rewriting menus and price lists; but it creates some benefits, too, by eroding stubbornly high wages and promoting job creation.

Hyperinflation, though, is a different beast from ordinary inflation. Hyperinflation shreds every monetary contract, makes much of modern economic life impossible, and all but guarantees that a totally different form of money, anything from foreign dollars to cigarettes, will be pressed into service. Yet the historical record is in some ways encouraging.

A new Cato Institute working paper tries to document every hyperinflationary episode in history. Getting the numbers isn’t easy, as the authors, Steve Hanke and Nicholas Krus of Johns Hopkins University, are at pains to point out. Hyperinflation is a time of civic dysfunction. By the time the price level is doubling every couple of months, or days, people have often stopped collecting credible statistics.

Hanke and Krus report 56 episodes of hyperinflation. A 57th, in North Korea in the past few years, is excluded for lack of sound data. Given that the authors were able to scrape together price indices for the Free City of Danzig in 1923 and for the Japanese-occupied Philippines in 1944, the failure to establish facts about present-day North Korea tells you something about Pyongyang’s attitude to statistical outreach. Hanke says that since the paper was released, the hyperinflation club has a new member: Iran.

A few facts leap out. First, hyperinflation is a phenomenon of the modern era: with a single exception, every hyperinflation has occurred since the end of the first world war. The outlier is revolutionary France, where monthly inflation passed 300 per cent in the summer of 1796.

Second, three-quarters of these hyperinflations – 43 out of 56 – occurred in one of three clear historical clusters. The first cluster is central European states after the first world war. It provides the most famous hyperinflation in history: Weimar Germany. The second cluster is during or immediately after the second world war, and it includes history’s worst: Hungary in 1946. Those inflation rates defy comprehension – 41,900,000,000,000,000 per cent a month, compounded, is (I believe) an annual inflation rate with 178 digits. It makes more sense as 207 per cent a day.

The third cluster is that of Eastern bloc countries as the Soviet Union disintegrated, and it comprises over half of all the 20th century’s hyperinflations. These are all examples of hyperinflation going hand in hand with an extremely stressed political and social system. Most of the remaining examples, from Zimbabwe to late revolutionary France, exhibit that same stress.

There’s a lesson here: regardless of the fears of some US Republicans and German hard-money fans, hyperinflation is not produced by central bankers. It is the result of a total failure of the political system.

There are a few people who are simultaneously buying gold in expectation of hyperinflation in western economies and stockpiling bullets in anticipation of a calamity for western civilisation. I will give the survivalists this much credit: the scenarios are consistent. But calamity arrives first, and hyperinflation follows.

Also published at ft.com.

How to find a perfect match for a Nobel

‘The Nobel Prize for economics has been awarded to Americans Alvin Roth and Lloyd Shapley for their independent work into how best to bring different parties together for mutual benefit.’

Financial Times, October 16

It’s not really a Nobel Prize, is it?

I’m glad you’ve mentioned that because it’s an important point that is rarely made – except, of course, any time anybody says or types the words “Nobel Prize for economics”, at which point a spontaneous chorus of “there is no Nobel Prize for economics” will break out. No, it is more recent than the original Nobel Prizes, and has a lengthy name. It undoubtedly does not deserve the status of, for instance, the Nobel Prize for literature (never awarded to Leo Tolstoy or James Joyce) or the Nobel Prize for peace (awarded to Henry Kissinger, the EU and most controversially to the Cabbage Patch Kids).

You seem touchy.

Perhaps a bit, yes.

Apparently this week it was awarded to Alvin Roth and some other guy.

I feel a little sorry for Lloyd Shapley, and not just because he should have been awarded the prize back in 1994. He has been somewhat eclipsed by Prof Roth, and this does happen. In 2005, the prize was awarded to Thomas Schelling, a brilliant writer who analysed everything from quitting smoking to winning the cold war, and Robert Aumann, who did something so abstruse and mathematical that nobody knew what it was. Not surprisingly everybody talked about Prof Schelling. It’s similar here: Prof Roth seems an intensely practical economist, charging around writing papers about dwarf-tossing and setting up life-saving kidney transplant operations, while all Prof Shapley produced was some mouldy old maths.

I thought he ran a dating agency, or something?

With the late David Gale, he designed a matching algorithm and used marriage as an example.

You economists are such romantics. You’re making me blush.

The algorithm was not actually designed to arrange marriages. It was to analyse ways to match things up. Imagine a world where husbands and wives were pair up by some automatic, algorithmic process.

They aren’t.

No, but schools and schoolchildren are, for example, as are teaching hospitals and medical students. Prof Shapley and Gale discovered a simple rule that would allow these matches to be made. After the algorithm had done its work, no two people would have an incentive to peel off and do some kind of side deal together, which is important if the algorithm is supposed to function in a free society.

How does the algorithm work?

One simple variant is for each of the men to propose marriage to one of the women. The women must reject all but one, and hold on to the best one temporarily. Each rejected man gets to make a second offer, and again the women get to hold on to the offer they had before, or alternatively reject it and hold a different offer. The point at which no man is rejected is the point at which you can close the market and tie the knot. And the allocation will be stable.

It seems rather traditional. Can’t the women propose instead?

Sure. If they do, it turns out that you will get a different but still stable set of matches, and this set will be one that the women prefer. So this was the theory, and Prof Roth developed the theory and put it into practice.

And it was Prof Roth who set up the dating agency?

He patched up the way in which medical students were matched with their first hospital jobs, which is more complex because some students are couples and want to apply for pairs of jobs in the same city. When Prof Roth looked at the problem, in the mid-1990s, the matching procedure being used was unstable. Offers – or ultimatums – were handed out many months in advance because people preferred their side deals to the centralised system. This is a classic symptom of unstable matches. He fixed it.

And got a Nobel Prize for that?

Yes, along with designing algorithms to match students with schools and kidney donors with recipients. And he deserves it. Prof Roth does three things that are far too rare in economics. First, he really engages with the anthropological reality of how individual markets work, rather than immediately leaping to an abstract model. Second, he recognises ethical or cultural constraints – we don’t like the idea of buying and selling kidneys, for instance. Too many economists have simply argued that the constraints make no sense, shrugged and walked away. Prof Roth accepted them as a fact of life and worked around them.

He’s almost human.

Indeed. And an important point is that many of the markets Prof Roth has designed do not and cannot work perfectly. A lot of theoretical economics is concerned with proving that perfection is possible, or impossible. Prof Roth is interested in answering the pragmatic question “is this good enough?”.

Also published at ft.com.

Lloyd Shapley and Alvin Roth win the Nobel memorial prize in economics

I took a particular interest in the FT leader on the subject:

The award of the Nobel memorial prize in economics to Lloyd Shapley and Alvin Roth is overdue: Mr Shapley should have shared the 1994 prize with John Nash and others, while Mr Roth has been a leading contender in recent years. The choice is a particularly good one because economists have acquired some bad habits, and Mr Roth’s example may serve to break them.

Mr Shapley is one of the key figures in co-operative game theory, which for decades looked both abstract and pointless, a poor relation to regular game theory. However, co-operative game theory is finally coming into its own in a world where computerised auctions are used to award assets or contracts in clusters. Mr Shapley has added a new page to the thick catalogue of useless ideas that turned out to be useful after all.

In contrast, the practical application of Mr Roth’s ideas has never been in doubt. Building on work by Mr Shapley and David Gale, Mr Roth designs algorithms for matching things. Mr Gale and Mr Shapley considered a whimsical problem: how to design a centralised system for allocating husbands and wives in a way that there is no possible male-female combination with a mutual desire to elope.

Mr Roth has developed this theory, but he has also put it into practice, allocating medical students to teaching hospitals and children to schools. Along the way he has resolved difficulties such as the fact that some pairs of medical students are married couples and want to live in the same city.

Most famously of all, he is part of a team that has designed kidney exchanges. If a person with kidney failure has a willing donor who is not a biological match, Mr Roth’s kidney exchange finds another pair in a similar situation – or however many pairs are necessary to find everyone a compatible organ.

Mr Roth’s work is clever and useful but he is an example to his fellow economists in other ways. While others argued that it should be legal to buy and sell organs, Mr Roth tried to understand why we find such transactions repugnant, and designed a practical alternative. He has also advocated an engineering approach to economics: rather than simply proving that something can or cannot be done, his ideas on matching ask fuzzier questions such as “will this work in most cases?” or “is this as good as we can get?”

Above all, Mr Roth has understood that if you test theory in a real-world environment, the theory will improve. So may the world itself.