Friday, August 14, 2009

No Child's Critical Thinking Left Behind

The week's Education Week reports that U.S. Secretary of Education Arne Duncan plans to set aside $350 million of the $4.35 billion in discretionary aid in the Race to the Top Fund to improve student assessments:

Testing experts say that money could serve as a down payment for scaling up tests that would better measure students’ critical-thinking skills and improve teacher and student engagement in the assessment process.
Many education experts would like replace the multiple-choice tests that dominate today's No Child Left Behind Testing. Paraphrasing Randy Bennett, a scholar at the Educational Testing Service, Education Week notes:
Such tests... are not ideal for identifying whether students can take multiple pieces of domain-specific knowledge and analyze, integrate, and apply them in unfamiliar contexts..
Researchers familiar with international benchmarking argue that those critical-thinking skills are precisely the type that will be in demand as the global economy becomes increasingly knowledge-oriented.
Education Weekly cites two examples of such tests. First, there's the College and Work Readiness Assessment, a computer-based test used by private high schools:
A typical ... question might present examinees with a dossier of materials relating to a child who had a roller-skating accident at school. The materials could include newspaper articles, technical reports about the skates, data about competitors’ products, sales figures, medical reports, and the number of documented accidents. Then, the student would be asked to analyze those materials and write a memo about whether the skates are truly dangerous, and to justify his or her conclusions drawing from the information.
The second example is recently piloted subset of the 2009 National Assessment of Educational Progress in science, which used "interactive computer tasks" to prompt students:
to engage in the entire process of scientific inquiry, in which they must participate in a simulated experiment, record data, and defend or critique a hypothesis.
While such tests have typically been costly, because they must be scored by humans, Education Week cites experts as saying that advancements in technology could help score these tests:
The high costs of scoring such a complicated assessment with an almost unlimited number of answers... could be mitigated by advancements in natural-language-processing software­—essentially programming that proponents claim can judge written essays as accurately as human readers and reduce, though not eliminate, the need for costly human evaluation.
Even with what is still pie-in-the-sky technology (I've worked in Natural Language Processing!), the proposed new measurements sound dangerously subjective to me, and also highly language-intensive in ways that will disfavor bright, analytically-minded kids with language delays.

Also, wouldn't it be cheaper just to make the multiple choice questions (which, in many states, are notoriously simple) more challenging? Well-crafted multiple choice question can indeed measure higher-level thinking skills, as they do on standardized aptitude tests like the SATs.


Mrs. C said...

Arg! But why the test at all? Why the funding (OUR money, remember) being tied to tests at all?

Beth said...

Mrs. C., boy do I agree with you. NCLB has been a disaster for all kids, right-brain, left-brain, whatever. Gifted students get ignored because they're already at the 99th percentile, so they can't measurably improve. Enormous amounts of school time is wasted on test prep and test administration. Honestly, in my daughter's last year at public school, she lost at least 3 weeks out of the year to standardized tests. And even aside from those 3 weeks, the entire year is tainted by the teacher's fear of these tests.

Tweaking the tests is not the answer. Scrap the whole thing and start over. How about asking parents how they would like to improve their kids' schools, and then taking the answer seriously? Ask the teachers too. For the older grades, ask the students. Then work toward a consensus, or even better, try to offer as much variety as possible to serve all the different types of kids and their needs.

By the way, the SATs are no longer described even by the company that makes them as "aptitude tests". They've admitted that it's about achievement. When I took the SATs, the official line was that it was impossible to study for them, because they measured innate aptitude! And although I was one of those students with a transcript in the toilet, and the only bright spot in my application was my SAT score, I would be thrilled to ditch them as well. There have got to be better ways to show a student's strengths.

Katharine Beals said...

I agree with Mrs. C. and Beth-- ideally, chucking the tests seems best. And I Beth's alternatives for holding schools accountable. On the other hand, the book Education Myths has convinced me that high-stakes tests have had some empirically good effects on school performance--but only inasmuch as they provide some sort of objective measure. To that end, a better multiple choice test seems far better than what Duncan et al are proposing.

As for whether the SAT is a measure of aptitude (vs. achievement), it seems that, while the College Board no longer wishes to say so (concerned with public relations?), there is evidence that it does. Cf Dunlan 1984, and:

Beth said...

In the hyper-achieving, competitive suburbs where I live, it has now become standard to study for the SATs and do all kinds of tricks (for instance, practice tests) to bring your scores higher. Our babysitter even told us that at her very expensive private school, the rumor going around is that if you take Ritalin, you're guaranteed an extra 200 points on the SATs! In this way the SATs wind up increasing the gap between the advantaged and the disadvantaged. Advantaged kids have access to all these tricks, and manage to inflate their scores.