Measuring Up: What Educational Testing Really Tells Us

by Daniel Koretz

The list of threats to the conclusions commonly based on test scores—threats to validity—is long. Some of the big ones: There is measurement error, to start, which creates a band of uncertainty around each student’s score. When we are concerned with aggregates, such as the average score or percent proficient in a school, there is sampling error as well, which causes meaningless fluctuations in scores from one group of students to another and from one year to the next. This is a particularly serious problem for small groups—for example, when tracking the performance of small schools or, even more problematic, the performance of groups of students within a school. The results we rely on are sometimes specific to a given test. Different choices of content, different methods of scoring, different item formats, even different mathematical methods for scaling a test can produce somewhat different patterns of scores. Different ways of reporting performance do not always paint the same picture. This is a particular concern in the light of the current reliance on standards-based reporting, which is one of the worst ways to report performance on tests and is sometimes simply misleading. Potential bias should always be a concern, especially when assessing certain groups of students, such as those with disabilities or limited proficiency in English. The current ubiquitous and intense pressure to raise scores creates the potential for seriously inflated scores.