Tim Harford's Blog, page 35

Hard truths about the gender pay gap

We all know that there’s a gap between the money men earn and the money women earn, but I hadn’t grasped quite how enormous that gap still is. Even in egalitarian Finland, men earn a third more than women. In the UK and the US, men earn two-thirds more. In the Netherlands and Austria, men earn twice as much. In Japan, men earn four times as much as women.

These staggering numbers (from a 2017 study by the economists Henrik Kleven and Camille Landais) refer not to pay for doing the same job, but to the total gap in labour market earnings between men and women. That gap exists not just because women are, on average, paid less than men, but also because women do much more household work that is not paid at all.

While it is natural to think of such huge disparities as a problem of fairness, it is also useful to think of them as a problem of efficiency. If in Japan men as a whole earn four times more than women, what does that suggest about the waste of female talent? (For that matter, how many Japanese men would be brilliant stay-at-home dads, if only they had the chance?) Japan is an outlier among rich countries, but the same pattern can be seen across the developed world.

Last week, the London School of Economics launched the Hub for Equal Representation in the Economy, whose aim is to find “effective ways to improve representation of women and minorities at work”. One of its first research studies, still in progress, aims to quantify the problem of untapped talent by looking at pre‑pandemic data from a multinational company, covering about 100,000 employees across 100 countries. The researchers, economists Nava Ashraf, Oriana Bandiera, Virginia Minni and Victor Quintas-Martinez, can compare the pay earned by equally experienced men and women.

As one might fear, there’s a gap: men are paid more than women, on average. What is counterintuitive about the data is that in some countries, the pay gap inverts: women are paid more than men. And those countries aren’t the ones we might expect. They are places such as Pakistan, where few women work in the paid labour force at all.

What explains this? Simply, that in a country where the barriers to paid work for women are high, the few women who do have jobs at multinational companies are outstandingly good. These high‑flyers are promoted and paid more than the average man. This suggests that there are women outside the workforce who, if they did have paid jobs, would be well above average even if they weren’t superstars. If the barriers to their workforce participation could be lowered, they would raise the productivity of the companies that employed them.

“If you equalised the barrier,” says Professor Bandiera, “some men would move out of the labour force, some women would move in, and productivity would increase by 32 per cent.”

That productivity gain — almost a third — is the average across all the countries studied; in the places with particularly unequal labour force participation, it’s much higher. It’s quite a lot of money to be leaving on the table.

“Some people are good at some things and some at others,” says Bandiera. “But there’s a big mismatch between those aptitudes and how we actually assign roles. Gender is just the most obvious example of that mismatch.”

With both economics and fairness pointing in the direction of greater equality, it is no surprise that we have seen some progress over the decades. That progress, however, has been slow.

Bandiera is hopeful that the pandemic, with its shake-up of the way we all work, might accelerate things. But while we all love an optimist, recent research from the Institute for Fiscal Studies suggests a grimmer conclusion. When British schools were closed in the spring of 2020, with many people trying to do office work from home, families shuffled the chores in order to cope with the constant presence of children.

Alison Andrew and her IFS colleagues found that in households where the father earned more than the mother, men did much less housework and less childcare than women, while doing more paid work and vastly more uninterrupted paid work. That is what economic logic might suggest.

But even in households where the mother earned more than the father, women did more childcare, more household work and less uninterrupted paid work than men. Economic incentives matter, but our gendered expectations of who takes care of housework have their own perverse force.

There has been progress over time, notably in the educational attainment of women, who in the UK are now more likely than men to be university graduates. But this is a campaign to be fought on many fronts. The inequality between men and women is showing no signs of disappearing of its own accord.

Written for and first published in the Financial Times on 21 January 2022.

The paperback of The Data Detective was published on 1 February in the US and Canada. Title elsewhere: How To Make The World Add Up.

I’ve set up a storefront on Bookshop in the United States and the United Kingdom. Links to Bookshop and Amazon may generate referral fees.

How Music Works, by David Byrne

Having somehow let Talking Heads pass me by as a teenager, I’ve been discovering the joys of David Byrne recently. The Strong Songs podcast had a terrific analysis of the Stop Making Sense concert movie, which is astonishingly good on every level. The BBC’s Soul Music also covered Once In A Lifetime a year ago.

What better time to pull David Byrne’s How Music Works off my bookshelf, where it’s been lurking unread for nearly a decade? I didn’t regret the decision. I expected to like the book; in fact I loved it.

How Music Works is a wide-ranging look at every aspect of music from culture to collaboration to finances. What makes the book so compelling is a delightful mixture of three things. First, Byrne has had a long, varied musical career and there’s lots of biographical detail here. Second, Byrne is widely read and very interested in ideas. Third, he writes in a straightforward, unpretentious way. These three things make the book a winner.

To give a sense of the range, consider three chapters. One, “My Life In Performance”, skips through Byrne’s teenage experiments playing folk music at the local coffee house, through the early days of Talking Heads, the making of Stop Making Sense, up to his 2008 tour. Along the way he reflects on his influences (Peking Opera!), the finances of touring, his self-diagnosed “very mild (I think) form of Asperger’s syndrome”, and even the choice of clothes. It’s fast-moving, full of telling detail, mistakes made and lessons learned.

A second, “Technology Shapes Music: Digital” explores how digital tools have not just changed the business of music but the music itself. Byrne reflects on how the studio itself becomes a compositional tool, and on how software pushes musicians towards some decisions and away from others.

A third, “Business and Finances”, explains the business of music, the different business models for a record release, and compares and contrasts (with pie charts!) two moderately successful Byrne albums, one released through a record company, and one self-published. Lots of detail, lots of ideas, no filler.

And there’s much more – the importance of space, thoughts on how a vibrant music “scene” emerges, reflections on whether recorded music is the death of amateur performances… Recommended!

The paperback of The Data Detective came out on 1 February in the US and Canada. Title elsewhere: How To Make The World Add Up.

I’ve set up a storefront on Bookshop in the United States and the United Kingdom. Links to Bookshop and Amazon may generate referral fees.

Known unknowns, an impossibility theorem, and the power of regret

I’ve been enjoying The Innovation Delusion by Lee Vinsel and Andrew Russell. Particularly interesting when it gets to the joys and challenges of maintenance.

My colleague Gavin Jackon has recently published Money in One Lesson in the UK (US publication in April) – a really fun, vivid book with lots of lovely nuggets of information.

Wikenigma is a wiki full of known unknowns…

The Power of Regret by @DanielPink is an excellent read. Dan points out that the romantic Piafian ideal of living life without regrets is… well, kind of absurd. Regret can be useless crying over spilt milk, but it is also often a process of reflecting on our mistakes. We learn to do better in future, and we learn also about what really matters to us.

Pink has conducted a chunky survey of people’s regrets, and he groups them into four basic groups: we regret messing around and squandering opportunities ; we regret being timid; we regret our moral lapses; and we regret failures to connect with others.

As you might expect from @DanielPink it’s a breeze to read, rigorous without being overly chewy, and offers plenty to think about and to kick off good conversations.

If you want to see what happens when I try to explain Ken Arrow’s impossibility theorem, here’s your chance, courtesy @WhyInteresting.

And here’s a one-hour NPR interview with me about how to uncover truth in a world “infested” with data.

The paperback of The Data Detective was published on 1 February in the US and Canada. Title elsewhere: How To Make The World Add Up.

I’ve set up a storefront on Bookshop in the United States and the United Kingdom. Links to Bookshop and Amazon may generate referral fees.

Is inflation worse for the poorest households? Incredibly, we’ve decided it’s fine not to know the answer

What colour is a rainbow? On average, white. And what is the current level of UK inflation? On average, 5.4 per cent. Both answers are true. Both are missing something important.

The consumer price index, or CPI, aims to measure the average price paid by UK consumers. But when the latest CPI numbers came out, the food writer and poverty campaigner Jack Monroe took to Twitter with some examples from her local supermarket: 500g of cheap pasta up from 29p to 70p year on year; 1kg rice up from 45p to £2; baked beans up from 22p to 32p. As the examples piled up, Monroe concluded that the inflation rate faced by poor households was much higher than the CPI’s 5.4 per cent.

“The system by which we measure the impact of inflation is fundamentally flawed,” she added — in a thread that has since been read by millions of people. I agree, for different reasons. Monroe is worried that prices are soaring for the poorest households. I’m worried that our current inflation-measuring process simply can’t tell us if she is right or wrong.

Instead, we have to rely on clues. One clue comes from looking at a reweighted CPI. The FT’s Chris Giles did this. He pointed out that while energy prices were rising fast, so were the prices of treats such as restaurant meals. Reweight the CPI to reflect the spending of a poorer household (more food and heating) or a richer household (more eating out and flights) and the inflation rate is much the same for everyone.

Another clue comes from industry watchers. Retail analyst Steve Dresser notes that Jack Monroe’s data came from Asda, but “Asda have been cutting value tier products for a while . . . it’s not indicative of the wider market.” 

A third clue comes from a large survey of consumer spending recently published by academic economists in the US. They conclude that the inflation spike in late 2020 was higher for “low income, low education and Black households”. However, the difference is half a percentage point or less.

But we shouldn’t have to rely on clues. Reweighting the CPI cannot show us what is happening to the super-cheap food products that are so important to someone shopping on a tight budget. Indeed, the head of inflation at the Office for National Statistics, Mike Hardie, tells me that the ONS data gatherers target the most popular products, based on shelf space. Are these products the cheapest staples or not? “We haven’t done the analysis to know,” acknowledges Hardie.

I have some sympathy. It makes sense to calculate inflation by looking at the same goods, month after month. But a savvy budget shopper might first head to the “reduced to clear” section. Then they might pick up some cheap carbohydrates. Rice one week, spaghetti the next — whatever was on special offer. What is the inflation rate for such a shopper? Even in theory, the answer is elusive.

Nevertheless, the answer matters. It may not matter to the Bank of England as it tries to ensure price stability across the entire economy, but it should matter to the government as it sets the level of benefits and the state pension. And it certainly matters to a desperate parent at the checkout, trying to figure out if they can afford a jar of peanut butter or whether the toast will be dry this week.

Jack Monroe — who tells me that her “key interests are maths and cheap pasta” — is now pulling together a voluntary effort to produce a price index of cheap food staples and other basics such as menstrual products, toothpaste and shampoo. But that cannot compete with the resources that the Office for National Statistics devotes to compiling the CPI, collecting 6,000 price quotations a day across the country. The ONS has just announced plans to beef up this effort with scanner data from supermarkets.

Is this statistical firepower well aimed? Yes, if the aim is to measure inflation across the economy. No, if we want to understand what is happening to the poorest households. It is both expensive and invaluable to gather really good data. But it is often shocking to realise that there are certain questions that have slipped between the cracks.

In 2007, we were stunned to realise how much we didn’t know about risk in the banking sector. In 2019, Caroline Criado Perez’s Invisible Women made a compelling case that data relevant to women were often neglected. In 2020, the ONS was quick to set up an infection survey representative of the population, but such a survey could have been established years before. Now Jack Monroe has pointed to another gap.

Nerds and bean-counters everywhere — my kind of people — need to try to fill such gaps in our knowledge rather than stumble into them. There are white rainbows everywhere, and it is not always easy to pick out the colours. But we must try.

Written for and first published in the Financial Times on 4 February 2022.

The paperback of The Data Detective was published on 1 February in the US and Canada. Title elsewhere: How To Make The World Add Up.

I’ve set up a storefront on Bookshop in the United States and the United Kingdom. Links to Bookshop and Amazon may generate referral fees.

How to change the world with data visualisation

“She may look at it because it has pictures.” That was Florence Nightingale’s withering comment, as she sent a report about public health reform to Queen Victoria.
Nightingale was not much impressed with the Queen (“The least self-reliant person I have ever known”). But she clearly understood her audience: if you want to get a message across, paint a picture.
Nightingale’s message could hardly have been more important. After famously serving in an Istanbul hospital during the Crimean war, she returned from the place she called “the kingdom of hell” with a reforming mission.
The hospital had indeed been hellish. Men would arrive, bleeding from abdominal wounds, their bodies crawling with vermin; and leave, stitched up in their own blankets to be carried to a mass grave.
In January 1855 alone, the British army in Crimea lost one man in ten to the ravages of diseases such as dysentery and cholera. Nightingale was attempting – at first without success – to prevent a humanitarian catastrophe, as infectious disease tore the British Army to shreds.
But Nightingale found that the death toll in the hospitals was dramatically reduced after improvements to hygiene – whitewashing walls, for example, and pulling a dead horse out of the water supply.
She believed that similar efforts at public hygiene could dramatically improve public health back in Britain. What had worked in Istanbul could work elsewhere. “Nature is the same everywhere, and never permits her laws to be disregarded with impunity,” she noted.
The insight was to drive her campaigning back in the UK. She was one of the only figures to emerge from the disaster of the Crimean War with reputation intact. But despite her celebrity, she had an uphill struggle to convince the medical establishment.
With germ theory in its infancy, Nightingale’s ideas were viewed as radical and, by many doctors, implausible. The chief medical officer, John Simon, opined in 1858 that as a cause of premature death, contagious diseases were “practically speaking, unavoidable”.
Nightingale was not only a nurse and a national icon, but a statistician, the first female fellow of the Royal Statistical Society. She had used her understanding of data to track the link between sanitary improvements and the falling death rate in the Istanbul hospitals.
To turn that understanding into action required statistical persuasion. With geek allies such as William Farr and John Sutherland, Nightingale began to campaign for better public health measures. But the crucial weapon in that campaign was Nightingale’s data visualisation – most famously her “rose diagram”.

The most famous of Florence Nightingale’s “rose diagrams”, “Diagram of the causes of mortality in the army of the east”

This diagram, easily dismissed as mere decoration, was to change the world. As a piece of statistical rhetoric it is breathtaking, telling a compelling story of disaster before the sanitary improvements, and redemption afterwards. Those two pale circles delivered a powerful two-part payload; John Simon and his allies felt the force of both barrels.
But as striking as the diagram itself was Nightingale’s insight into the importance of data visualisation, at a time when British statisticians would invariably rely on tables of data.
On Christmas Day 1857, she sketched out a plan to use data visualization for social change. She declared her plan to have her diagrams glazed, framed, and hung on the wall at the Army Medical Board, Horse Guards, and War Department. “This is what they do not know and did ought to.”
And she planned to distribute her diagrams to exactly the right people.
“None but scientific men even look into the appendices of a Report, and this is for the vulgar public . . . Now, who is the vulgar public who is to have it? … The queen … Prince Albert . . . all the crown heads in Europe, through the ambassadors or ministers of each … all the commanding officers in the army … all the regimental surgeons and medical officers . . . the chief sanitarians in both houses [of Parliament] … all the newspapers, reviews and magazines.”
John Simon and his allies were helpless before this onslaught. Nightingale and her allies – and, particularly, her graphical rhetoric – proved irresistible. Public health practice evolved, new sanitary laws were passed, and John Simon quietly revised his views about the inevitability of death from contagion.
Every modern data visualisation expert has an opinion on Nightingale’s graphs. Some find them breathtaking, others confusing or even misleading. But to my mind, there is something shockingly, brilliantly modern about the battle she decided to fight, and the way she used persuasive data visualisations as her weapon.
More than ever, we need to understand how data visualisation works. We need to understand it as consumers – who may be enlightened or bamboozled, depending on our own “graphicacy” and the choices made by the producers of graphs.
And we need to understand data visualisation as producers. Graphs and diagrams are powerful tools, supercharged by the ready availability of data and versatile software. But like any tool, they can be used skilfully or clumsily. They may be used to build something wonderful, or repurposed as weapons.
For an insight into the power and the sorrows of data visualisation, we could do worse than think of Florence Nightingale’s campaign to change the world with a souped-up pie chart.

To find out more about Florence Nightingale, data visualisation and the uses and misuses of data, please note that the paperback of The Data Detective was published on 1 February in the US and Canada. Title elsewhere: How To Make The World Add Up.

If you would like to hear Helena Bonham Carter in the role of Florence Nightingale (she’s incredible) then check out this episode of Cautionary Tales.

Assorted news and links

The Prime Minister made much the same false statement to parliament on 24 Nov, 5 Jan, 12 Jan and 2 Feb. @FullFact have repeatedly requested a correction and the Office for Statistics Regulation have written to his office to ask him to stop.

The claim is important in its own right (it’s that there hundreds of thousands more people in employment now than before the pandemic; in fact there are hundreds of thousands less) but the principle is important too. You can be thrown out of the House of Commons for accusing someone of lying – but not, it seems, for repeatedly making untrue statements?

(More on this story in this week’s @BBCMoreorLess featuring @PuzzlesTheWill.)

More cheerful news is that I have discovered a legendary Washington Post typo. Instead of reporting that President Woodrow Wilson was seen “entertaining” his fiance Edith Galt, the paper wrote that the President was seen “entering” Miss Galt. Oops.

Interesting-looking books are arriving on my pile.

John List @Econ_4_Everyone offers “The Voltage Effect” – about the importance and difficulty of scaling ideas. Slightly unfocused introduction (List is too keen to offer personal anecdotes) but I’m expecting the meat in the intellectual sandwich to be good.

Dan Pink @DanielPink is back with “The Power of Regret“; self-recommending, I look forward to reading it. I doubt Pink has ever written a dull book.

I recently revisited @BioengineerGM Guru Madhavan’s delightful “Think Like an Engineer“, and @DEHEEdgerton’s superb “the Shock of the Old“. Great books.

Norman Rockwell explains the secretive of being creative when you’re feeling blocked, using the “lamppost trick“.

Know that I, too, am proficient in Microsoft Word.

The paperback of The Data Detective was published on 1 February in the US and Canada. Title elsewhere: How To Make The World Add Up.

I’ve set up a storefront on Bookshop in the United States and the United Kingdom. Links to Bookshop and Amazon may generate referral fees.

How to stand up to your bullying inbox

Are you being bullied by your inbox? That was the question posed recently on the productivity podcast Change Your Game with GTD [Getting Things Done]. It hit a nerve, as on that particular day my email inbox had me in a corner, turning out my pockets for lunch money.

We do not often refer to the email inbox as a bully, but the metaphor is apt. Some people are unlucky enough to have an inbox full of abuse, but even those of us who do not must watch it warily, ever alert for some fresh demand. The inbox commands our attention because its insistent requests can be so unpredictable. The consequences of non-compliance are unknown and unsettling, and there are always new demands on their way, leaving you feeling permanently on the back foot.

This year I have resolved to no longer be bullied by my inbox. It is time to step forward and fight back. But how? My plan has two components: attack and defence. By defence, I mean keeping the inbox itself under control. By attack, I mean having something better to do than keep checking it.

Defence is fundamental, but difficult. By design, email is an open standard. Every day, people send me emails from all over the world. Some are complimentary, some abusive, many are marketing pitches and a surprisingly large number are requests for me to devote a few unpaid hours acting as a consultant, editor or research assistant for a total stranger. Some of this stuff can be blocked — on irrelevant marketing I operate a “one strike and you’re out” policy — but much of it I couldn’t block and wouldn’t necessarily want to.

The real difficulty, however, is that mixed in with all this are important and often urgent requests from my boss, my colleagues or my wife. If the people who matter to you are using email to get your attention, anyone else who wants your attention can saunter in there too. This can feel hopeless. You can’t ignore your boss or your spouse, so you can’t ignore your inbox. That would be like trying to hide from unwelcome visitors to your house by turning off the lights and not answering the door, then realising that you’ve booked a dozen deliveries.

Nor do the obvious tricks and hacks work terribly well. Switch off notifications? Sure, good idea. But if the inbox is central to your working life, that’s like taking the batteries out of your doorbell but going to check every 90 seconds to see if someone’s there.

The ideal strategy, if you can manage it, is to make sure your boss, your colleagues and your spouse don’t use the same inbox as random cold callers. This probably works fine for the Pope and the president, but I’ve never found that it works for me.

Cal Newport, author of A World Without Email, proposes setting up alternative workflows, such as regular “office hours” rather than coordinating meetings through endless rounds of email, or a process where project documents are uploaded to a shared drive by a given time each week. It is more like an assembly line for knowledge work than endless improvisation through the inbox. If you can manage this, you can ignore your inbox for most of the working day. But I have never found my work life predictable enough to make this seem feasible.

The alternative is what one might call the “scupper” approach. Scuppers are holes in the sides of boats or other structures that tend to get soaked. The scupper lets water in, of course, but it also lets water out again. This works better for me: the endless flow into the inbox is manageable if there is a sufficiently rapid flow out again.

I delete most emails immediately after scanning them if they don’t require any action. If a polite “Thanks!” is required, I tend to schedule it to be delivered the next day. Instant replies risk provoking a back-and-forth conversation. Nobody wants that.

The trick here is to make sure that getting emails out of the inbox is an effortless process. I have three subsidiary folders: one for things to do, one for things I’m waiting for someone else to do and a dumping ground for things I might want to read later (but which generally I don’t). I can mostly clear incoming mail in the blink of an eye. The tidal wave of email washes in and washes out again; what remains is neatly bottled into a few specific tasks.

That, then, is how I defend myself against bullying from the email inbox. But I have come to realise that attack is the best form of defence. Attacking means having a clear plan for what I intend to do — with the week, with the day, with the next half-hour. When I know what I am trying to achieve, I tend to check email less often. I am also less likely to abandon that goal in favour of someone else’s agenda popping up in the inbox.

Bullies tend to shy away from those who exude confidence, and this is true for email too. When I am anxious, the urge to self-distract with email is powerful. When I lack a clear plan, noodling around in the inbox becomes the substitute. But if I’m on a mission, I check email far less often.

Have I persuaded you? I’ve almost persuaded myself. Or perhaps I am just whistling in the dark. Let us see whether I can stand up to the bully in 2022.

Written for and first published in the Financial Times on 6 January 2022.

The paperback of The Data Detective is out on 1 February in the US and Canada. Title elsewhere: How To Make The World Add Up.

I’ve set up a storefront on Bookshop in the United States and the United Kingdom. Links to Bookshop and Amazon may generate referral fees.

The ten best books for thinking clearly about statistics

To celebrate paperback publication day of “The Data Detective”, and in no particular order…

The Art of Statistics by David Spiegelhalter @D_Spiegel is a wide-ranging guide to how statistics work, full of vivid and humane examples by the world’s greatest living statistical communicator.

The Signal and the Noise by @NateSilver538 is a great guide to the power and limits of statistical thinking.

The Biggest Bluff by @MKonnikova is, of course, a brilliant, riveting book about poker rather than statistics. Except that of course, it’s really a book about how to think clearly about statistics.

The Tiger that Isn’t by Andrew Dilnot and Michael Blastland is the closest thing you’ll find to a @BBCMoreOrLess bible. So full of wisdom; hugely recommended.

Naked Statistics by @CharlesWheelan is very clear, fun book about all the important ideas in statistics. A great preparation for any course requiring statistical skills.

Invisible Women by @CCriadoPerez is a truly eye-opening book, exploring the consequences of data-gathering with the assumption that the default human is male. A must read.

Weapons of Math Destruction by Cathy O’Neil (@mathbabedotorg) explains what happens when we place too much in statistical algorithms to make high-stakes decisions.

Innumeracy by @JohnAllenPaulos is a hugely influential exploration of what happens when people (particularly journalists) can’t think statistically.

Bad Science by @BenGoldacre – a hilarious, razor-sharp critique of media foolishness about stats, science and medicine.

And if you’ve made it this far you probably already know about my own book, which is called How To Make The World Add Up in the UK, India, Ireland, Australia, South Africa, New Zealand…

…and The Data Detective in the US and Canada. It’s so good they named twice. Now in paperback everywhere…

I’ve set up a storefront on Bookshop in the United States and the United Kingdom. Links to Bookshop and Amazon may generate referral fees.

The stripper, the Congressman, and why we undervalue statistical bedrock

At 2am on October 9th, 1974, police in the picturesque tidal basin area of Washington DC spotted a speeding car, weaving around, headlights off. They pulled it over, and out jumped a flamboyantly dressed woman, yelling in both English and Spanish.

She promptly threw herself into the water, and had to be hauled out again. In the car was an elderly man, glasses broken. He was steaming drunk. More to the point, he was a senior congressman, Wilbur Mills, and she was a celebrated erotic dancer, “Fanne Foxe”.

It was the beginning of the third most notorious sex scandal in American politics. But to my fellow nerds, the disgracing of Congressman Mills set in train a game of political musical chairs. This game had a welcome side-effect…

…the appointment of Alice Rivlin as the head of a brand new body: the Congressional Budget Office. She was the candidate every sensible person would have wanted for the job, but she’d been blocked by a congressional dinosaur who thought it was no place for a woman.

Once the dinosaur had been reshuffled in the wake of Wilbur Mills’s departure, Dr Rivlin would do much to establish the reputation of the CBO. Still, forty years later she noted “I owed my job to Fanne Foxe.”

It’s a silly slice of political trivia in some ways. But perhaps it typifies the way that we take serious independent analysis for granted. Do we put the best person in the job, or not? In Alice Rivlin’s case, all depended on a rather juicy quirk of history.

I think we undervalue the presence of high-quality, non-partisan organisations such as the CBO, the Census Bureau, and in the UK, the Office for National Statistics. There is very little glamour in what they do, but it serves as a kind of statistical bedrock.

When such organisations do their jobs well, we all benefit: the state, companies and individual citizens all have a vastly clearer view of the challenges around them. When such data-gathering and data-analysing bodies stumble, we suffer.

And where such institutions do not exist, or are subject to partisan control, we simply cannot see the road ahead of us. We’re as helpless as a drunk Congressman, glasses smashed, trying to drive with the headlights off.

I’ve much more to say about this – and other ways in which we under-rate the numbers all around us, in my latest book. It is out on tomorrow in paperback in the US and Canada with the title “The Data Detective”.

Elsewhere it’s called “How To Make The World Add Up” – it was top of the Sunday Times Business Bestseller lists in 2021, although I think it’s much more than a business book.

If you’ve read the book and enjoyed it – thank you! Please spread the word. If not, now would be a wonderful time to pick up a copy.

How to Truth with Statistics

“The crooks already know these tricks; honest men must learn them in self-defense.” – so wrote nerd legend Darrell Huff, in his million-selling “How To Lie With Statistics” (1954).

It’s a delightful little work, full of deceptive graphics, spurious correlations, biased samples and all sorts of other statistical crimes, playfully illustrated. As a teenager I loved reading it.

If you read How to Lie with Statistics and you will be more sceptical about the way numbers can deceive you.

But there’s a darker side to Huff; he sold out to the tobacco lobby, worked on a sequel “How To Lie With Smoking Statistics” (thankfully never published), and testified in Congress in an attempt to blow statistical smoke in the eyes of legislators.

It’s sad and infuriating. Now just because Huff turned bad doesn’t make “How To Lie With Statistics” a bad book. (It isn’t. It’s excellent.) Yet the book does have a serious weakness, and Huff’s personal failings are relevant here.

That weakness is Huff’s tendency to make statistics seem like a game, a stage magician’s trick, all good fun but never to be trusted. I worry that we’re starting to trust nobody; we’re starting to believe that lying with statistics is all anyone ever does. Huff does not help.

Scepticism is all very well, but not if it curdles into cynicism. Statistics can be used to deceive but they are also a vital tool in our quest to understand the world around us, like a telescope for an astronomer.

So alongside a healthy scepticism, we need the skills to distinguish statistical truth from lies, and the confidence to realise that statistics are often the only window we have onto a large and complex world.

A few months ago, I published my own antidote to Huff. (Cory @Doctorow suggested it should have been titled “How To Truth With Statistics”. Alas I missed that idea.)

That book is out on 1 Feb in paperback in the US and Canada with the title “The Data Detective”.

Elsewhere it’s called “How To Make The World Add Up” – it was top of the Sunday Times Business Bestseller lists in 2021, although I think it’s much more than a business book.

If you’ve read the book and enjoyed it – thank you! Please spread the word. If not, now would be a wonderful time to pick up a copy.