Category Archives: Accountability

Scott Milfred’s column in the Sunday May 17, 2009 Wisconsin State Journal is a textbook example of the kind of foolishness people indulge in when they are dazzled by data that seems to support their favored policies.

Mr. Milfred likes charter schools. One piece of the recently released test score data for the Madison charter school Nuestro Mundo was positive, Mr Milfred seizes on this single piece of data, giving it no examination or context, then asserting that the entire state needs more charter schools.

In a previous post on the Nuestro Mundo WKCE results, I wrote:

All good, but really too little change or information to justify any action or inaction. Data should inform actions, but not drive them. Do we really want a system where decisions are made based on how 1 or 2 or 3 kids test on a given day?

Apparently Mr.Milfred does, and he wants it for the entire state.

The post quoted above explained that the rise in test scores being hailed amounted to about four kids in fourth grade improving. Further, it noted that mobility issues make it difficult to be certain of even this much. That post also noted that scores in other subjects still lag and that this year’s third grade scores are pretty bad.

Mr. Milfred’s money quote is:

Eighty-one percent of the Nuestro Mundo fourth-graders tested proficient or advanced in English reading on the most recent tests. That’s better than the fourth-graders in the district as a whole, most of whom speak only English.

Getting at the limited utility of this statement requires looking behind the numbers some more.

First, I don’t know where the 81% comes from, DPI/WNSS says 80% and since there are exactly 0 students from the Nuestro Mundo fourth grade whose results are reported, a percentage of 81 is impossible. The district advanced/proficient for fourth graders is 75.9%

If one fewer Nuestro Mundo student had hit the advanced/proficient cut score, the school percent would be 77.5%; two fewer, 75%and below the district average. So that’s the one, or two or three kids whose scores Milfred wants us to base policy on.

Let’s look at some demographics. Nuestro Mundo’s fourth grade class has a 4o% poverty rate; the rate for MMSD’s tested fourth graders is 44.6%. As more than one researcher has noted “…Standardized Achievement Tests are [more] Sensitive to Socioeconomic Status Rather than Instruction…”

Nuestro Mundo’s fourth grade has one tested student with disabilities (2.5%); MMSD’s fourth grade as a whole has 251 tested students with disabilities, or 15.7%. Only 47.4% of the fourth graders with disabilities in Madison tested advanced/proficient (note, the testing of some disabled students amounts to a torturous game of asking someone to do what they clearly cannot; the use of these results to judge or punish schools and districts is a sick practice…I have similar feelings about the use of tests in this way with non-English speakers).

You can do the Math, adjust for demographics and Nuestro Mundo’s fourth grade achievement starts to look a lot like the average fourth grade in Madison.

I don’t want to dwell on it much further, but fourth grade math scores at Nuestro Mundo aren’t good, especially for low income or ELL students. Here are a couple of graphs:

Only 33.3%  of ELL students scoring advanced proficient on Math gives one pause.

Unlike Mr. Milfred, I’m not simple-minded enough to urge any drastic policy based on these scores. They deserve considered and continued attention, further investigation and perhaps some remediation, but even though they are much more negative than Mr. Milfred’s vaunted reading scores are positive, I don’t think there is enough information here to say (for example) “shut the school down.”

Unfortunately, as long as we have standardized tests substituting for knowledge and learning — in both classrooms and policy debates — people like Mr. Milfred will latch onto some simplistic cherry picked results to push this reform or that.

It isn’t all that different from the mentality that created this economic crisis. In that case, investors wanted to bypass investigation and thought, and boil risk down to a single number. They could check their brains and pretend all was well because “the numbers were good.” Never mind that what was behind those numbers was a mystery to the investors and a house of cards that was doomed to fall.

In education there is a politician’s and polemicist’s desire for easy transparency and accountability. Test scores must be published, but few look at the tests or even the cut scores and scales. The complexities of the tests and scores are swept aside and almost completely forgotten are all things not on the tests and all those things that can’t be boiled down to a single number.

Things like “value added” have the potential to make this worse by removing “the number” further from comprehension by most.

People like Scott Milfred then use things — a number(s) — that they don’t understand, to spout off about other things — education policy — they don’t understand.

Don’t listen to them. Look behind the numbers and the polemics, read and learn and contribute. Education is too important to be hijacked by the lazy likes of Scott Milfred.

One last note. If I read Mr. Milfred’s column correctly, he is a parent of a Nuestro Mundo student. This information should be prominently disclosed when Mr.Milfred, or the Editorial page he is in charge of, opines on the topic. It has not been.

Thomas J. Mertz