Monday, January 7, 2013

Educational malpractice for the sake of Reform Math

...Thus ends, a few days ago now, my Favorite Comments of '12. And it seems fitting to follow these, and belatedly usher in the year 2013, with a digest of one the most compelling email messages I've received in a very long time. It comes from James Milgram, an emeritus Professor of Mathematics at Stanford University.

Professor Milgram is known in the education world for his comprehensive critique of a study done by Jo Boaler, an education professor at Stanford, and Megan Staples, then an education professor at Purdue. Boaler and Staples' paper, preprinted in 2005 and published in 2008, is entitled Transforming Students’ Lives through an Equitable Mathematics Approach: The Case of Railside School. Focusing on three California schools, it compares cohorts of students who used either a traditional algebra curriculum, or the Reform Math algebra curriculum The College Preparatory Mathematics (CPM). According to Boaler and Staple's paper, the Reform Math cohort achieved substantially greater mathematical success than the traditional math cohorts.

In early 2005 a high ranking official from the U.S. Department of Education asked Professor Milgram to evaluate Boaler and Staples' study. The reason for her request? She was concerned that, if Boaler and Staples' conclusions were correct, the U.S. department of education would be obliged, in Milgram's words, "to begin to reconsider much if not all of what they were doing in mathematics education." This would entail an even stronger push by the U.S. educational establishment to implement the Constructivist Reform Math curricula throughout K12 education.

Milgram's evaluation of Boaler and Staples' study resulted in a paper, co-authored with mathematician Wayne Bishop and statistician Paul Clopton, entitled A close examination of Jo Boaler's Railside Report. The paper was accepted for publication in peer-reviewed journal Education Next, but statements made to Milgram by some of his math education colleagues caused him to become concerned that the paper's publication would, in Milgram's words, make it "impossible for me to work with the community of math educators in this country"--involved as he then was in a number of other math education-related projects. Milgram instead posted the paper to his Stanford website.

This past October a bullet-point response to Milgram's paper, entitled "When Academic Disagreement Becomes Harassment and Persecution," appeared on Boaler's Stanford website. A month ago, Milgram posted his response and alerted me to it. I have his permission to share parts of it here.

Entitled Private Data - The Real Story: A Huge Problem with Education Research, this second paper reviews Milgram et al's earlier critiques and adds several compelling updates. Together, the two papers make a series of highly significant points, all of them backed up with transparent references to data of the sort that Boaler and Staple's own paper completely lacks.

Indeed, among Milgram et al's points is precisely this lack of transparency. Boaler and Staples refuse to divulge their data, in particular data regarding which schools they studied, claiming that agreements with the schools and FERPA (Family Educational Rights and Privacy Act) rules disallow this. But FERPA only involves protecting the school records of individual students; not those of whole schools. More importantly, refusals to divulge such data violate the federal Freedom of Information Act. Boaler's refusal also violates the policies of Stanford University, specifically its stated "commitment to openness in research" and its prohibitions of secrecy, "including limitations on publishability of results."

Second, Milgram et al's examination of the actual data, once they were able to track it down via California's education records, shows that it was distorted in multiple ways.

1. Boaler and Staple's chosen cohorts aren't comparable:

It appears, from state data, that the cohort at Railside [the pseudonym of the Reform Math school] was comprised of students in the top half of the class in mathematics. For Greendale, it appears that the students were grouped between the 35th and 70th percentiles, and that the students at Hilltop were grouped between the 40th and 80th percentiles. [Excerpted from Milgram; boldface mine]
2. Boaler and Staple's testing instruments are flawed:
Our analysis shows that they contain numerous mathematical errors, even more serious imprecisions, and also that the two most important post-tests were at least 3 years below their expected grade levels.  [Excerpted from Milgram; boldface mine]
3. The data comparing test scores on California's standardized tests (STAR) comes from a comparison of test scores from students not involved in Boaler and Staple's study:
The students in the cohorts Boaler was studying should have been in 11th grade, not ninth in 2003! So [this] is not data for the population studied in [Boaler and Staple's paper]. This 2003 ninth grade algebra data is the only time where the Railside students clearly outperformed the students at the other two schools during this period. There is a possibility that they picked the unique data that might strengthen their assertions, rather than make use of the data relevant to their treatment groups.   [Excerpted from Milgram; boldface mine]
4. The most relevant actual data yields the opposite conclusion about the Reform Math cohort's mathematical success relative that of the traditional math cohorts:
o The most telling data we find is that the mathematics remediation rate for the cohort of Railside students that Boaler was following who entered the California State University system was 61%
o This was much higher than the state average of 37%
o Greendale's remediation rate was 35% o and Hilltop's was 29%.
5. School officials at "Railside" report that the results of the reform math curriculum are even worse than Milgram et al had originally indicated:
A high official in the district where Railside is located called and updated me on the situation there in May, 2010. One of that person's remarks is especially relevant. It was stated that as bad as [Milgram et al's original paper] indicated the situation was at Railside, the school district's internal data actually showed it was even worse. Consequently, they had to step in and change the math curriculum at Railside to a more traditional approach.

Changing the curriculum seems to have had some effect. This year (2012) there was a very large (27 point) increase in Railside's API score and an even larger (28 point) increase for socioeconomically disadvantaged students, where the target had been 7 points in each case.
6. Boaler’s responses to Milgram et al provide no substantiated refutations of any of their key points

In response to comments on an article on Boaler's critique of Milgram, Boaler states:
"I see in some of the comments people criticizing me for not addressing the detailed criticisms from Milgram/Bishop. I am more than happy to this. [...] I will write my detailed response today and post it to my site."
However, as Milgram notes in his December paper:
As I write this, nearly two months have passed since Boaler's rebuttal was promised, but it has not appeared. Nor is it likely to. The basic reason is that there is every reason to believe [Milgram et al's paper] is not only accurate but, in fact, understates the situation at "Railside" from 2000 - 2005.
In a nutshell: under the mantle of purported FERPA protection, we have hidden and distorted data supporting a continued revolution in K12 math education--a revolution that actual data show to be resulting, among other things, in substantially increased mathematics remediation rates among college students. Ever lower mathematical preparedness; ever greater college debt. Just what our country needs.

Nor is Boaler's Reform Math-supporting "research" unique in its lack of transparency, in its lack of independent verification, and in its unwarranted impact on K12 math practices. As Milgram notes,
This seems to be a very common occurrence within education circles.

For example, the results of a number of papers with enormous effects on curriculum and teaching, such as [Diane Briars and Lauren Resnick's paper "Standards, assessments -- and what else? The essential elements of Standards-based school improvement"] and [J. Riordan and P. Noyce's paper, "The impact of two standards-based mathematics curricula on student achievement in Massachusetts"] have never been independently verified.

Yet, [Briars and Resnick's paper] was the only independent research that demonstrated significant positive results for the Everyday Math program for a number of years. During this period district curriculum developers relied on [Briars and Resnick's paper] to justify choosing the program, and, today, EM is used by almost 20% of our students. Likewise [Riordan and Noyce's paper] was the only research accepted by [the U.S. Department of Education's] What Works Clearinghouse in their initial reports that showed positive effects for the elementary school program ``Investigations in Number, Data, and Space,'' which today is used by almost 10% of our students.
As Milgram notes:
Between one quarter and 30% of our elementary school students is a huge data set. Consequently, if these programs were capable of significantly improving our K-12 student outcomes, we would surely have seen evidence by now.
And to pretend that such evidence exists when it doesn't is nothing short of educational malpractice.

18 comments:

Anonymous said...

Also very much worth noting is what Bishop and Milgram wrote specifically to respond to Boaler's claim that they were harassing her: http://math.stanford.edu/~milgram/Jo-Boaler-reveals-attacks-AccusationsResponse-trans.html

Anonymous said...

Thank you -- a powerful post. And yes to the comment concerning Bishop's & Milgram's response to the harassment claims. This is altogether a sad and puzzling situation. How can it be that scholarship on education has fallen so low?

Anonymous said...

It can be because the Education Reform racket is populated by grifters and cheats. Neil Bush, Michelle Rhee, etc. There's no useless expensive textbook whose failure can't be covered up by pencil-whipping the tests. The "education researchers" aren't trying to find the truth, they're trying to get in on the grift.

kcab said...

Thanks for posting the information and link to Milgram's response. I didn't see that earlier, perhaps I wasn't looking at the right time or place.

Anonymous said...

I think this is more than just malpractice. This kind of research is motivated by a real desire among educators to use education as a tool of upward mobility and 'social justice" for traditionally poorly performing groups. In math, that largely means minorities and girls. The constructivist approach of multiability groupings probably does improve the performance of the lowest quartile of students. Unfortunately, it does so at the expense of the highest quartile who will never achieve their potential. The idea is to drive all students toward the mean, as a method of obtaining equality of results and "social justice". The end result is that the best students from constructivist schools will never be competitive with the best from traditional schools and can't begin to compete with students from high performing foreign countries. Someone should explain to the education community that leaving high potential students with a sub par education is not justice, and the idea of social engineering through education is creepy and antisocial. Public education is no longer providing opportunity for the disadvantaged as it did for my immigrant family 90 years ago. Social justice advocates need a lot less John Dewey and a lot more ED Hirsch.

Jen said...

Anon at 3:53 said:
"The constructivist approach of multiability groupings probably does improve the performance of the lowest quartile of students. Unfortunately, it does so at the expense of the highest quartile who will never achieve their potential."

Actually, I'd argue that the constructivist approach is MOST harmful to the lowest quartile of students. Those are the students who need the most explicit instruction, the most guided practice and the most repetition.

High performing kids often do okay with the more constructivist approach because they *can* pick up a concept that was only mentioned once or only mentioned in passing. Two or three repetitions of practice may be enough for the very brightest and most motivated students. And, kids who have involved parents, likely to be your average or above scorers, are also most likely to be taught at home as needed or to get tutoring.

Taking students who come to school behind their peers and providing little to no real instruction and assuming that the activities they do in class will somehow coalesce in their heads without any sort of road map is the real horror. These are kids who are behind at the start and are never given the type of instruction which would catch them up.

That's not to say that this is the best way to teach the highest scorers, just that they are the ones most likely to be successful.

AC said...

It's all about The Gap. If you're not aware, The Gap is the persistent under-performance of blacks, and more lately, Hispanics in school. Constructivist approaches hurt everyone in absolute terms, but in relative terms they hurt the middle most of all: bottom's already at the bottom, top will always find a way. The middle, though, can be pushed toward the bottom with poor instruction. This compresses the range and where there used to be The Gap, there are now two, largely overlapping low-performing groups.

Shannon Severance said...

Anonymous said, How can it be that scholarship on education has fallen so low?

Has it fallen far? Where did it start at to have fallen?

AC said, [the] top [children] will always find a way emphasis added.

No, not always. The top will need luck to find that there is a "way". (In my case the big break was TI discontinuing their TI 99/4a computer and my uncle giving my family one. Something he could do because of the lower closeout price.) And then the child will need luck or be high in other attributes, autodidacticism is not always for the faint of heart.

Jen said, High performing kids often do okay with the more constructivist approach because they *can* pick up a concept that was only mentioned once or only mentioned in passing

In almost all cases doing okay will be well short of their potential.

Anonymous said...

At Harrison Bergeron High, students scoring in the top decile are given remediation to lower their scores. This results in the lowest Gap in the district, a resounding success.

AC said...

No offense, but if luck is a factor, we're not talking about the "top."

By top I mean the top 2-8% in a nation-wide sampling. These are the people that are going to be vacuumed up by Google & Apple, or Microsoft & Intel before them, or HP & IBM before them. These are the people that invent the things upon which fortunes are made.

"autodidacticism is not always for the faint of heart" Um, OK, but that's pretty much how the top do it.

The middle, those are subject to luck whether they get a break. At the high end they go into good, solid fields that make our nation richer and a more pleasant place to live: Fortune 500 companies, civil & mechanical engineering at all levels, teachers, etc. Believe me, we want to help them. But it's a relative measure and few people are able to notice a rampant under-achieving mediocrity, and it takes a long time for the rot to set in.

Anonymous said...

With the goal of driving all students toward the mean, 50% will be working below potential. Only about 30% of students go on to college, so the system is working to lower educational standards and achievement for virtually all of the college bound population. Hence the explosion of remedial courses in colleges and the growing drop out rate. This country is loosing it's edge in education to Asia and Europe and this is likely to accelerate in the next 10 years as more and more kids come out of constructivist primary and secondary programs.

Leigh Lieberman said...

Scientists and engineers (S-E) like me would never tolerate grossly shallow programs such as Everyday Math in our homes or classrooms for core elementary school math instruction.
When the R&D facility where I worked for over 20 years as an S-E was slated for closure by the DoD to reduce the federal deficit in the early '90's, I, along with many of my colleagues, took advantage of the opportunities for career change to acquire teaching certification. Although dozens of us took all the required pedagogical courses for math and other certifications, not one of us wanted to teach math in schools that were wedded to these types of reform practices, so inherent in the prevailing, dreadfully weak state standards and certification system. A number of us looked into charter school possibilities which were just beginning to evolve; we got as far as finding an urban school principal who was ready to come out of retirement to support such a venture. To the best of my knowledge, none of us applied for any math classroom teaching assignments in any of the local public schools, in spite of the fact that teaching was the field that offered the most opportunities to avoid relocation.

Doug1943 said...

(1) I think it dramatically overstates the case to say the constructivists are consciously seeking to engage in downward levelling, in order to achieve social equality. Daniel WIllingham, a psychologist who studies learning and who is absolutely in favor of evidence-based practice, put it well in this blog post: http://www.danielwillingham.com/1/post/2012/10/your-edupolicy-adversaries-theyre-not-evil-and-you-dont-know-what-they-think.html
(Incidentally, his blog is well worth following.)

(2) The real problem is weak students and weak teachers. Discover-it-for-yourself methods of learning absolutely have their place with intelligent and motivated students and an intelligent and motivated teacher. (You learn to do proofs by doing them, not by memorizing them.) But at the bottom of the ladder, they are disastrous. This problem is not going to be solved in a hurry, if at all. The changes required to get fatherless children from an inner-city housing project, taught by semi-literate and anumerate EdSchool graduates, to acquire a real education, are just too great.

(3) It CAN be done, though. Watch this video (it's transferred from film done in the 1960s -- the first minute or two of sound is poor quality but it improves): http://vimeo.com/23659310

(4) Of course, the education establishment has a long history of blithely ignoring evidence that would disconfirm its prejudices. See the report on "Project Follow-Through", a massive study of differing approaches to teaching that was carried out forty years ago: http://darkwing.uoregon.edu/~adiep/ft/151toc.htm

(5) But things are not as bad as they seem. Compared to other countries, the United States doesn't do too badly: our Caucausians are about average, compared to European Caucasians; our Africans and Hispanics are better than African Africans and Latin American Hispanics; and even our Asian students are better than Asian Asians. See the evidence here: http://super-economy.blogspot.co.uk/2010/12/amazing-truth-about-pisa-scores-usa.html

Barry Garelick said...

Discover-it-for-yourself methods of learning absolutely have their place with intelligent and motivated students and an intelligent and motivated teacher.

It also has to do with the domain knowledge of the students and how the discovery process is implemented--is it guided, or merely facilitated with low expectations such that the process trumps the content and the outcome? Discovery works well at The Harkness table at Exeter, but there are high expectations and a lot of work required of the students at home. Ill-implemented discovery programs hurt those with high IQ's as well as average and low IQ's.

momof4 said...

I can see a HS-level discovery/seminar class working, IF and ONLY IF the kids have really solid background knowledge in the field AND are willing and able to work independently, outside of class, on preparation and the written report of the content discovered. In other words, only a small fraction of the American school population. In most cases, particularly in ES, it amounts to the kids pooling their ignorance, and should therefore be avoided.

Particularly when working with disadvantaged kids, who enter school significantly behind their advantaged agemates, the educational priority should be EFFICIENCY. Time is precious and should never be wasted. Of course, I'd like to see the elimination of advantaged third-graders (only the girls) spending 45"(each) on acting out a scene from a book (with friends and costumes); just write the book report, already! Down with dioramas!(it's a wonder the kids and I don't panic at the sight of a shoebox)

Barry Garelick said...

When my daughter was in 8th grade, she brought home the directions from the teacher about their book report "project". They were reading "Lord of the Flies" in class, and the project offered various options for presenting a book report: A t-shirt, a book jacket blurb, a comic book, some other ludicrous options, and finally, a report. My daughter said "Can I just write a report?" and I enthusiastically agreed.

momoffive said...

Cathy Williams, a Vista School math head, has been insisting that Jo Boaler is the best in math for students. Cathy has dumbed down math at my son's Vista , Ca middle school claiming that middle-schoolers are just not ready for difficult math. She believes in Integrated Math, and with Common Core now in full swing she has got Int. Math adopted in our district. My 7th grader's math is sooooo bad!!! No skills at all, they move slowly and he is so bored. I started researching Cathy and her pride and joy, Jo B. and I find this article. OMG! This explains EVERYTHING that is going on at Vista. Please help me figure out the bottom line in this piece (I'm no mathematician). I need help to bring back regular math to middle school and high school. This is tragic!

Anonymous said...

Momoffive, your Cathy is like the psychic who predicts a murder and then commits it herself. Yes, middle-schoolers will be unready for difficult math if they've only been given thin constructivist gruel for years. But the solution isn't to stop giving them difficult math in middle school, it's to prepare them adequately for it.

Math gets more fun and more interesting as it gets more difficult, but if you never really mastered arithmetic, algebra will be impossible and confusing.

There's no getting around the difficulty of memorizing times tables. The greatest disservice of constructivist math is in driving middle-class parents to extra-curricular math programs, leaving children of parents without those resources at an even greater comparative disadvantage.