The real risks of cherry picking scientific data

My Times column is on the dangers of omitting
inconvenient results:

Perhaps it should be called Tamiflugate. Yet the
doubts reported by the House of Commons Public Accounts Committee
last week go well beyond the possible waste of nearly half a
billion pounds on a flu drug that might not be much better than
paracetamol. All sorts of science are contaminated with the problem
of cherry-picked data.

The Tamiflu tale is that some years ago the pharmaceutical
company Roche produced evidence that persuaded the World Health
Organisation that Tamiflu was effective against flu, and
governments such as ours began stockpiling the drug in readiness
for a pandemic. But then a Japanese scientist pointed out that most
of the clinical trials on the drug had not been published. It
appears that the unpublished ones generally showed less impressive
results than the published ones.

Roche has now ensured that all 77 trials are in the public
domain, so a true assessment of whether Tamiflu works will be made
by the Cochrane Collaboration, a non-profit research group. The
person who did most to draw the world’s attention to this problem
was Ben Goldacre, a doctor and writer, whose book Bad Pharma accused the industry of
often omitting publication of clinical trials with negative
results. Others took up the issue, notably the charity Sense About
Science, the editor of the British Medical Journal
, Fiona Godlee, and the Conservative MP Sarah Wollaston. The
industry’s reaction, says Goldacre, began with “outright denials
and reassurance, before a slow erosion to more serious
engagement”.

The pressure these people exerted led to the hard-hitting PAC report last week, which found
that discussions “have been hampered because important information
about clinical trials is routinely and legally withheld from
doctors and researchers by manufacturers”.

The problem seems to be widespread. A paper in the BMJ in 2012 reported
that only one fifth of clinical trials financed by the US National
Institutes of Health released summaries of their results within the
required one year of completion and one third were still
unpublished after 51 months.

The industry protests that it would never hide evidence that a
drug is dangerous or completely useless, and this is probably so:
that would risk commercial suicide. Goldacre’s riposte is that it
is also vital to know if one drug is better than another, say,
saving eight lives per hundred patients rather than six. He puts it this way: “If there are eight people
tied to a railway track, with a very slow shunter crushing them one
by one, and I only untie the first six before stopping and awarding
myself a point, you would rightly think that I had harmed two
people. Medicine is no different.”

Imbued as we are with an instinctive tendency to read meaning
into nature, we find it counter-intuitive that many experiments get
significant results by chance and that the way to check if this has
happened is to repeat the experiment and publish the result. When
the drug company Amgen tried to replicate 53 key studies of cancer,
they got the same result in just six cases. All too often
scientists publish chance results, or “false positives”, like
gamblers or fund managers who tell you about winners they
backed.

Outside medicine, we popular science authors are probably guilty
of too often finding startling results in the scientific literature
and drawing lessons from them without waiting for them to be
replicated. Or as Christopher Chabris, of Union College in
Schenectady, New York, harshly put it about the pop-psychology author Malcolm
Gladwell: cherry-picking studies to back his just-so stories. Dr
Chabris points out that a key 2007 experiment cited by Gladwell in
his latest book, which found that people did better on a problem if
it was written in hard-to-read script, had been later repeated in a
much larger sample of students with negative results.

To illustrate how far this problem reaches, a few years ago
there was a scientific scandal with remarkable similarities, in
respect of the non-publishing of negative data, to the Tamiflu
scandal. A relentless, independent scientific auditor in Canada
named Stephen McIntyre grew suspicious of a graph being promoted by
governments to portray today’s global temperatures as warming far
faster than any in the past 1,400 years — the famous “hockey stick”
graph. When he dug into the data behind the graph, to the fury of
its authors, especially Michael Mann, he found not only problems
with the data and the analysis of it but a whole directory of
results labelled “CENSORED”.

This proved to contain five calculations of what the graph
would have looked like without any tree-ring samples from
bristlecone pine trees. None of the five graphs showed a hockey
stick upturn in the late 20th century: “This shows about as vividly
as one could imagine that the hockey stick is made out of
bristlecone pine,” wrote Mr McIntyre drily. (The bristlecone pine
was well known to have grown larger tree rings in recent years for
non-climate reasons: goats tearing the bark, which regrew rapidly,
and extra carbon dioxide making trees grow faster.)

Mr McIntyre later unearthed the same problem when the hockey
stick graph was relaunched to overcome his critique, with Siberian
larch trees instead of bristlecones. This time the lead author,
Keith Briffa, of the University of East Anglia, had used only a small sample of 12 larch trees
for recent years, ignoring a much larger data set of the same age
from the same region. If the analysis was repeated with all the
larch trees there was no hockey-stick shape to the graph.
Explanations for the omission were unconvincing.

Given that these were the most prominent and recognisable graphs
used to show evidence of unprecedented climate change in recent
decades, and to justify unusual energy policies that hit poor
people especially hard, this case of cherry-picked publication was
just as potentially shocking and costly as Tamiflugate. Omission of
inconvenient data is a sin in government science as well as in the
private sector.

 

Post-script:

This column is not mainly about climate change, but about the
ubiquitous problem of selective citation of data. As I said, we all
do it to some extent, but it is still a sin against statistics.
However, as usual when publishing anything that touches on climate
change, there has been an immediate and highly misleading attempt
to rubbish the work I reported and to imply that I am evil for even
reporting others' opinions that climate scientists, unlike all
other scientists, might not all be infallible. This touchiness is
quite striking and not reassuring. For those who wish to know the
full story of bristlecones and Siberian larch, please follow the
links in the piece, one of which is to a Guardian article, and
please note that this article reports what one of Professor
Briffa's colleagues had to say in an email not intended for
publication:

"In October last year, Briffa's old boss at CRU, Tom Wigley,
said in an email to Briffa's current boss Phil Jones: "Keith
does seem to have got himself into a mess." Wigley felt Briffa
had not answered McIntyre's charges fully. "How does Keith explain
the McIntyre plot that compares Yamal-12 with Yamal-all? And how
does he explain the apparent 'selection' of the less
well-replicated chronology rather than the later (better
replicated) chronology?... The trouble is that withholding data
looks like hiding something, and hiding something means (in some
eyes) that it is bogus science that is being hidden."

There has been a subsequent attempt to justify the selectivity
of the larch data, but it is not very convincing, and I recommend
McIntyre's ripostes to it here, here and here. The rich irony is that one of Briffa's
justifications for ignoring one of the larger samples is that it
contains "root collar" tree samples. This is exactly equivalent to
the strip-bark problem that leads McIntyre and the National Academy
of Sciences to reject the inclusion of “strip-bark” bristlecones in
temperature reconstructions - that they give a false signal. They
cannot have it both ways.

Leave the last word on this business to McIntyre, discussing
Briffa’s explanation for why we should all ignore the fact that
tree rings inconveniently show a decline in temperatures in recent
years, or “hide the decline”:

“You’ll probably roll your eyes at the following Briffa-ism used
to rationalize Hiding the Decline:

‘In the absence of a substantiated explanation for the decline,
we make the assumption that it is likely to be a response to some
kind of recent anthropogenic forcing. On the basis of this
assumption, the pre-twentieth century part of the reconstructions
can be considered to be free from similar events and thus
accurately represent past temperature variability.’

When I first encountered this, I could not believe that
credentialed scientists could either write such bilge. That the
authors of such bilge should be among the most respected members of
the field was even more unbelievable.”

There is an even more shocking story of data omission in the
manufactured attempt to claim a 97% consensus among scientists on
dangerous man-made climate change. As Jo Nova details here, the conclusion was based
on just 0.3% of the data:

"Of nearly 12,000 abstracts analyzed, there were only 64 papers
in category 1 (which explicitly endorsed man-made global warming).
Of those only 41 (0.3%) actually endorsed
the quantitative hypothesis as defined by Cook in
the introduction."

Post-script 2:

The Times published a letter from the UK chief scientist and the
head of the UK Met Office, which rather misrepresented what I said,
while conceding my main point - that climate science had not been
sufficiently transparent:

 

Sir,

Matt Ridley falls into his own trap in his Opinion column (
Jan 6), though the title “Roll up: cherry pick your research
results here” is apposite, because that is exactly what Ridley does
with respect to the research evidence for global warming.

There can be no sensible arguments against making available the
results of properly conducted research for open scrutiny. The
arguments for this have been rehearsed very effectively in health —
and, in general, the biomedical research community has accepted
these arguments. Indeed UK scientists pioneered the controlled
clinical trial and the Cochrane Collaboration led the way in the
rigorous meta-analysis of all sources of evidence to reach the most
reliable conclusions allowing the implementation of
“evidence-based” medicine. The pharmaceutical industry, which can
certainly be criticised for past practices in not revealing the
results of all clinical studies of new drugs, is now moving towards
greater transparency, and drug regulators, such as the EMEA, are
rightly pressing hard. Iain Chalmers, Ben Goldacre and others
deserve much credit for their campaigning for openness.

The same can be said of the climate science community. Following
the controversy over leaked University of East Anglia emails there
have been substantial efforts in making source data openly
available. It is partly through this openness and replicablility of
findings by researchers in different institutes the Berkeley Earth
Island Institute analysis published last year is a case in point—
that drew the Intergovernmental Panel on Climate Change to the
unassailable conclusion in its most recent report that “warming of
the climate system is unequivocal”.

This report was a consensus led by 259 scientists, from 39
countries, which assessed the findings of all of the relevant,
peer-reviewed scientific literature published between the previous
report in 2007 and March of last year. Would that Matt Ridley
applied the same rigour when it comes to evidence about the
anthropogenic contribution to climate change. The “hockey stick”
graphs, prominent as they were at the time, are just one small part
of a massive global research effort that provides consistent and
overwhelming evidence.

Sir Mark Walport, Chief Scientific Adviser to HM Government

Professor Stephen Belcher, Head of Met Office Hadley Centre

My reply was as follows:

Dear Mark and dear Professor Belcher,

I am glad to see you recognising in your letter to the Times the
need for science, as well as industry, to clean up its act
with respect to transparency and data withholding. As for
the argument relating to the hockey stick that
"following the controversy over leaked University of East Anglia
emails, there have been substantial efforts to make source data
openly available", it is good that you acknowledge the role that
Climategate played in sparking this improved transparency.
Indeed, the surmise by Stephen McIntyre of Climate Audit that the
University of East Anglia had failed to report a Yamal regional
chronology that did not have a Hockey Stick shape was an important
issue leading into Climategate. Yet it was not investigated by
any of the East Anglia inquiries. As McIntyre says, "The existence
of this unreported adverse result was only revealed by subsequent
Freedom of Information requests - requests that were fiercely
resisted by the University." It was wrong that
those interested in understanding the hockey sticks had
to resort to freedom-of-information requests to get publicly funded
data that should have been freely published and wrong that the
requests were resisted.

You might be interested in McIntyre's account (to me) of what
has happened since: "In 2013, four years after the Climate Audit
criticisms, Briffa and coauthors published a re-stated version of
their Yamal chronology with a much diminished blade from the
previous superstick.  Rather than "discrediting" the earlier
criticisms, the re-statement implicitly conceded the validity of
the earlier criticism, as shown by the measures taken by Briffa and
coauthors to avoid repetition of the earlier mistakes.  While
they have avoided some of their earlier errors, their new
attenuated chronology still contains important methodological
defects and errors, as discussed at Climate Audit. Nor should
much weight be given to findings of the Muir Russell panel. Muir
Russell did not even attend the only interview with CRU academics
on the Hockey stick.  Nor did the panel interview CRU critics.
Nor did the Muir Russell panel even ask Briffa and Jones about
their destruction of documents to evade FOI requests."

You then go on to say that global warming is unequivocal, with
which I entirely agree (if we take a 30-50 year period) though it
is the evidence, not the number of scientists who have put their
name to a report, that convinces me. (It is equally unequivocal
that warming has been slower than the models forecast.) But this is
a straw man. My article did not claim that the hockey stick was
necessary to prove the warming unequivocal. "Unequivocal" is not
the same as "Unprecedented", which was the claim made by the
hockey-stick graphs. So I did not "fall into my own
trap". May I urge you in future to address the actual
arguments of sceptics rather than the almost entirely mythical
claim that they think climate change does not happen.

Best wishes

Matt