Calling Bullshit: The Art of Skepticism in a Data-Driven World

by Carl T. Bergstrom

Achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways. If no one knew that you were going to evaluate schools by looking at test scores of students, test scores might provide a reasonable way of measuring schools’ effectiveness. But once teachers and administrators recognize that test scores will be used to evaluate their effectiveness, they have every incentive to find ways of raising their students’ scores—even at the expense of the quality of their education. They may “teach to the test,” for example, rather than teaching critical thinking. The consequences are twofold. Once this happens, test scores lose much of their value as a means of assessing a school’s performance. The process of measurement may even actively undermine the quality of education, as time and effort are shifted away from valuable activity and toward the kind of memorization that improves test scores but is nearly useless from an educational perspective. The same problem occurs in the sciences as well. A few years back, Jevin penned a commentary about the dangers of using quantitative metrics to measure the quality of scientific research. Getting the wrong answer is not the worst thing a metric can do. It’s much worse to provide peop...