Evidence-Based Policy: A Practical Guide to Doing It Better

by Nancy Cartwright, Jeremy Hardie

Over the last twenty or so years, it has become standard to require policy makers to base their recommendations on evidence. That is now uncontroversial to the point of triviality--of course, policy should be based on the facts. But are the methods that policy makers rely on to gather and analyze evidence the right ones? In Evidence-Based Policy , Nancy Cartwright, an eminent scholar, and Jeremy Hardie, who has had a long and successful career in both business and the economy, explain that the dominant methods which are in use now--broadly speaking, methods that imitate standard practices in medicine like randomized control trials--do not work. They fail, Cartwright and Hardie contend, because they do not enhance our ability to predict if policies will be effective.

The prevailing methods fall short not just because social science, which operates within the domain of real-world politics and deals with people, differs so much from the natural science milieu of the lab. Rather, there are principled reasons why the advice for crafting and implementing policy now on offer will lead to bad results. Current guides in use tend to rank scientific methods according to the degree of trustworthiness of the evidence they produce. That is valuable in certain respects, but such approaches offer little advice about how to think about putting such evidence to use. Evidence-Based Policy focuses on showing policymakers how to effectively use evidence, explaining what types of information are most necessary for making reliable policy, and offers lessons on how to organize that information.

Genres: Politics, Philosophy, Nonfiction, Economics, Political Science, Public Policy

196 pages, Paperback

First published August 23, 2012

Reviews

[Steven Peterson · Rating 4 out of 5]

A fascinating book. . . . We often hear that, in terms of using evidence to determine what policy to adopt, we ought to privilege RCT studies (randomized control trials, otherwise called experiments). This book argues that using evidence to determine if a policy might work needs to be far more nuanced.

RCT studies are often very situation specific. In one state or locality, an experiment worked. Does that mean that it would work in another venue? The authors suggest not. The authors note early on (Page 7) that: "A great deal of this book is devoted to the significance of 'it worked there' if you are trying to bet whether 'it will work here'. . . ."

Essentially, one cannot a prior assume that a study based on RCT on one setting will automatically work in another setting. One has to under stand, as it were, "the facts on the ground."

Sometimes, this book can be a bit hard to follow, but it makes a good point. I teach a graduate course in policy analysis and I am seriously thinking of adopting this text. . . .

[Bernise Laurent · Rating 4 out of 5]

Oof, analytic philosophers do be writing absolutely obtuse books.

Anyway... overall it is a really good book and can really open up some good discussions about how we relate scientific methodology (particularly randomized control trials or RCTs) to the broader tendencies in policy implementation. Specifically, it is a call to be both more skeptical, but still optimistic, about our ability to use RCTs effectively in policy-making.

Cartwright starts out the book trying to formulate a little bit what the core process of understanding the scientific process in a philosophy of science manner, and particularly the philosophical ground of causal networks, warranted credence, an introduction to the philosophy underpinnings of RCTs, and so on. While Chapter I is undoubtedly fascinating, it can be a daunting task to get through.

Specifically, some of the core take-aways are this:

First, there are an array of causal support factors which are present in any given sociological context that may or may not transfer from one context to another. An infant nutrition security program in Tamil Nadu may not transfer to Bangladesh as educating mothers is fitting into the wrong section of the causal chain leading to infant nutrition, specifically as mothers-in-law are the primary determinants of food distribution in the household instead of mothers. Because of the "cake" which is present at this level, the transfer may not work. Getting similar results in a wide variety of contexts can increase warranted credence in their further application, but always be aware of the dangers. Keep in mind: focusing too deeply on trying to fully systematize all the support factors in a particular system is an impractical and impossible task, however identifying the causal relationships that lead to a particular policy outcome is far more important.

Second, RCTs are useful, powerful tools that can be use in order to facilitate evidence-based policy, but must be used with caution. Specifically, this has to be done with the far more qualitative process of identifying causal pathways, identifying the levels and support factors which would need to be present for the causal framework to work, so on. Specifically, Cartwright goes over 4 major strategies for doing this qualitative work in section III.B, however they can all be used to go about this process and need to be applied accordingly.

Third, collecting and ranking RCTs can be genuinely useful... but the RCTs always have to be evaluated given the context in which they are collected, and honest judgment in re-implementing or evaluating the transfer of policies from one context to another. Creating databases of high-quality RCT studies with rich information about the context in which they were studied is good, but using them requires a significant amount of judgment in line with the first two points.

Finally, Cartwright calls the reader to be sensitive towards the balance of a rules-based methodology, and being too reliant on judgment. You need both judgment and high-quality RCTs to make meaningfully warranted implementations of evidence for policy-making. Creating a rule book sometimes can be useful for reducing the cognitive load and freeing up deliberation space for higher quality qualitative study, but it can also act as a sort of sledgehammer that create limitations for the process. Don't be afraid to take it sufficiently slow.

All in all, the book is a fascinating read, and I can suggest it for anyone who has even the slightest passing interest in evidence-based policy making. Just wish it was written to be a little more readable...

[Rob Moore · Rating 2 out of 5]

This book has some interesting ideas, but is a little too theoretical to be helpful to practitioners and not quite rigorous enough to be theoretically compelling.

[Drew Lovett · Rating 5 out of 5]

<3

[Natalie · Rating 3 out of 5]

This was a bit hard to get through. I got to page 40 and almost gave up, but trudged through a few more pages - and then it got much better. I appreciate the authors wanting to make their points in various ways, but the book was very repetitive. Still, I may be able to take out bits and pieces for a public policy class I’m teaching. We’ll see how it goes...

[Simon Gant · Rating 1 out of 5]

Here's a direct quote to give you a flavour of the writing style: "Sometimes you may be interested in the efficacy of the policy in S because you think of implementing the same policy again in S, keeping the causal principles there fixed, or of implementing it somewhere else where the same principles obtain as obtain in S, but this time implementing it in a different way, perhaps in a way that keeps all other causal factors fixed or improves their values"

[Jimmy Volmink · Rating 2 out of 5]

Frustrating experience

This book uses 196 pages when 10 would have sufficed. Its overall message is that good decisions require reliable evidence as well as judgement (hardly a novel idea!). The creativity of the author in making the same point repeatedly throughout the book is indeed impressive. I kept on looking for more but alas...

[Scott Catey · Rating 3 out of 5]

Useful, but limited, especially in thinking about Federal policy implementation and evaluation.