Measuring Up: What Educational Testing Really Tells Us

by Daniel Koretz

After the first round of ratings, the procedure used by NAEP and many other testing programs does introduce some actual data about performance, called impact data. Panelists are given the actual percentage of students who answered each item correctly—the percentage of all students, not the percentages of students in the imagined groups just above each of the standards, which have not yet been set. This adds a norm-referenced element to the standards because the impact data are in fact normative data about performance. NAEP also provides panelists with data about the variation in ratings among the panelists. With this additional information, the panelists rated the items a second time. They then were given another round of feedback and rated the items a third time.