We’re Still ‘Lying to Our Children’
Saturday, March 6, 2010
Setting a national goal of having all students proficient by 2014 while letting states create their own tests and set their own cut scores has produced a mess.
The No Child Left Behind Act of 2001 (NCLB) set a national goal that all children be proficient in reading and math by 2014. The law left both the design of the tests and the setting of proficiency “cut scores” to the states.
The outcome of this choice is well-documented: many states set their cut scores so low that large numbers of students are judged proficient, even though they lack basic skills (see here and here). The National Assessment of Educational Progress’s (NAEP’s) recent Trial Urban District Assessment (TUDA), covering math in 18 cities, presents another window into this disconnect. Here I focus on fourth grade math, as results for the eighth grade are so similar that there is no need for separate analysis.
As international comparisons show, our students are lagging our major competitors, so what may be ‘aspirational’ for our nation is about basic for many other nations.
Consider Figure 1. The purple bars represent the percent of students in each district that have been classified as proficient in math based on state tests and state cut points. We might conclude that most students in these districts are doing reasonably well. In six districts, state results show that more than three-quarters of pupils are proficient and in four more districts more than two-thirds are proficient.
The orange bars show the percent proficient in these same districts based on the 2009 TUDA results. TUDA and state results describe two different worlds.
Figure 2 organizes these districts by the size of the gap between percent proficient on state assessments and NAEP. The gaps range from near 70 points in Baltimore and Detroit, and leaving Boston aside, drops to about 30 points in San Diego, Jefferson County, and the District of Columbia. (Massachusetts’s standards are among the most rigorous in the country—it has not taken the easy path that so many other states have.)
Measuring Progress Toward Proficiency
One of America’s biggest challenges is to make progress on the number of students who are proficient in math (and, of course, in reading). While Figures 1 and 2 show a disconnect between the percentage of proficient students at a single point in time, do NAEP and state measures paint a similar picture in terms of progress?
We can calculate change in the percent of students proficient on both NAEP and the state assessments for the 11 districts that took part in TUDA in 2007 as well as 2009. See Figure 3. The good news: in nine of those districts, at least the direction of the change is the same on both assessments (Atlanta and Austin are the exceptions). In terms of the magnitude of change, however, in seven of the 11 districts more growth was made on state assessments than on NAEP—and in Cleveland the decline in the percent proficient was smaller on the state test than on NAEP.
As a rough indication of how state assessments and NAEP differ in measuring progress, the average gain in percent of students achieving proficiency across these 11 districts on state assessments was 10 percent over that two-year period, while the average gain on NAEP was only 4 percent. Moreover, if we rank the districts on the size of their gains on each assessment, the correlation is only 0.20 and not statistically significant.
In short, we’re using different yardsticks to measure progress—and find only limited agreement between them.
We’re using different yardsticks to measure progress—and find only limited agreement between them.
Setting a national goal of having all students proficient by 2014 while letting states create their own tests and set their own cut scores has produced a mess. Clearly, state assessments and NAEP do not have identical purposes, but can these differences account for the wide variation in estimates of percent proficient? Moreover, NAEP’s proficiency standards are often dismissed as “aspirational,” setting a very high bar. But as international comparisons show, our students are lagging our major competitors, so what may be “aspirational” for our nation is about basic for many other nations.
The Obama administration recognizes this problem, and Secretary of Education Arne Duncan has gone so far as to say that states are “lying to children” in how they define proficiency. Indeed, as part of their emerging proposals for the reauthorization of the Elementary and Secondary Education Act, the administration is pushing to require all states to “adopt and certify that they have college- and career-ready standards.” But NCLB called on states to adopt “challenging” standards—and we saw the outcome of that mandate.
Faced with the tensions between setting a common cut score and deferring to states, the political pressures to allow the states to do so will be intense.
This evolving debate highlights the stakes involved in the “Common Core State Standards Initiative" now being pursued by the National Governors Association and the Council of Chief State School Officers, with the encouragement of the administration. This effort is fraught with challenges—and, as my analysis shows, one of the most fundamental will be who gets to set the cut scores on any assessments that are used to define “college- and career-ready standards.”
Faced with the tensions between setting a common cut score and deferring to states, the political pressures to allow the states to do so will be intense. The disconnect between state and TUDA results show just how consequential the stakes are in this decision.
Mark Schneider is a visiting scholar at the American Enterprise Institute and vice president at the American Institutes for Research. This is a revised version of a commentary that appeared in Education Week.
FURTHER READING: Earlier, Schneider detailed how “Obama’s Education Hopes Face Achievement Realities.” He has explained “Where Does All That Tuition Go?” in rising college costs, decried the weaknesses of “Math in American High Schools,” and examined “How Much Is That Bachelor’s Degree Really Worth?”
Image by Rob Green/Bergman Group.