Tuesday, July 28, 2015

A test is only as good as its graders

A recent article in the New York Times raises yet another concern about the new Common Core tests: who exactly is grading them? This concern stems from the fact that these tests "put less stock in rote learning and memorization" and therefore require fewer multiple choice questions

and far more writing on topics like this one posed to elementary school students: Read a passage from a novel written in the first person, and a poem written in the third person, and describe how the poem might change if it were written in the first person.
The two big Common Core testing companies are Pearson and PARCC. For Pearson, according to the Times:
About three-quarters of the scorers work from home. Pearson recruited them through its own website, personal referrals, job fairs, Internet job search engines, local newspaper classified ads and even Craigslist and Facebook. About half of those who go through training do not ultimately get the job.
As for PARCC:
Parcc said that more than three-quarters of the scorers have at least one year of teaching experience, but that it does not have data on how many are currently working as classroom teachers. Some are retired teachers with extensive classroom experience, but one scorer in San Antonio, for example, had one year of teaching experience, 45 years ago.  
Compare this to the AP requirements:
For exams like the Advanced Placement tests given by the College Board, scorers must be current college professors or high school teachers who have at least three years of experience teaching the subject they are scoring.
Of course, much smaller numbers of students take the AP tests, thus fewer graders are needed, thus higher standards are possible.

But if the Common Core test questions really live up to claims about the high-level thought processes they measure, then surely we need measurably high-level thinkers grading them:
The new tests are much more complicated and nuanced than previous exams and require more from the scorers, said James W. Pellegrino, a professor of psychology at the University of Illinois at Chicago who serves on advisory boards for Parcc and Smarter Balanced. “You’re asking people still, even with the best of rubrics and evidence and training, to make judgments about complex forms of cognition,” Mr. Pellegrino said. “The more we go towards the kinds of interesting thinking and problems and situations that tend to be more about open-ended answers, the harder it is to get objective agreement in scoring.”
I've written earlier about the virtues of well-constructed multiple choice tests. This article highlights one of them.

1 comment:

Auntie Ann said...

Poor grading also leaves a lot of room for fudging and bias.