Professor Longhair, “Bald Head” (click to listen or download).
Educational “accountability” is in the news and on the agenda again this week. It seems it is always in the news and on the agenda these days. I have many problems with most conceptions of educational “accountability,” especially those that are based largely on standardized tests (a visit to the National Center for Fair and Open Testing is in order if you don’t agree, or even if you do) and are proudly dubbed “data driven,” (the link takes you to old AMPS posts, Esther Quintero has an important post up on the topic this week at the Shanker Blog: “The Data-Driven Education Movement,” read it). I’m not going to take on the big concepts here and now, but instead say a few things about the new Wisconsin Report Cards and offer some thoughts about imposing some accountability on those concocting and implementing educational “Accountability” systems, about cutting the barbers’ hair.
The new Wisconsin Report Cards are the product of the “School and District Accountability Design Team” led by Governor Scott Walker, State Superintendent Tony Evers, Senator Luther Olsen, and Rep. Steve Kestell and featuring a decided over-representation of privatizers and deformers (those friends of education at Wisconsin Manufactures and Commerce had a seat), and an under-representation of educators (one teacher, no union reps). The final version is a centerpiece of Wisconsin’s successful effort to garner a waiver of NCLB strictures from Arne Duncan.
A school rating system like this should do three things. First it should with some accuracy and transparency rate school quality. Second, it should honestly and effectively communicate what the rating means and doesn’t mean to policy-makers, educators, parents, and citizens. Last — and assuming that the ratings are accurate — it should direct appropriate resources to those schools that need improvement. The Wisconsin system does none of these well. In fact, because of the complexities of assessing school quality, I don’t think it is possible to do all of these well and know that it is very difficult to do any of them well. The whole enterprise is in many ways a fool’s errand.
A recent must-read post by Gene V. Glass for the Washington Post captures some, but not all, of the problems (I’ve touched on the use of NAEP cut scores previously, will be saying more about some other things below and will be writing more on the waiver, the abuse of NAEP cut scores, “accountability,” and “educator effectiveness” issues in the future; as I was writing this another fine critique came my way, this one from Steve Strieker, called “Another Distractor: School Report Cards,” it is a must read also).
In the introduction to Glass’s piece Valerie Strauss calls the Report Cards “another cockamamie way to grade schools for “accountability” purposes.” Glass refers to the Report Cards as “a dog’s breakfast of numbers,” and writes:
The report card for Wisconsin K-12 schools currently making the rounds is a particularly opaque attempt to grade the quality of education that Wisconsin’s children are receiving at the hands of their teachers and administrators. It is as though the Department of Public Instruction has decided to weigh cattle by placing them on a scale to get their weight in pounds then combining that with the wealth of the farmer who raised them, the number of acres of the farm, and the make of car the farmer drives.
The Report Cards combine multiple and often complicated measures in complicated ways. It takes 62 pages to explain how it is all done. If in order to understand the choices made you want to dig deeper into the nature of standardized test construction (hint, they are designed to sort students, not measure skills, knowledge or ability), or the controversies over graduation rate calculations, or the limitations of the Student Growth Percentiles ( the link takes you to Bruce Baker posts on SGP and related things) used in the “growth” calculations, or any of the other concepts and tools employed , you are probably looking at at least the equivalent of a graduate school seminar’s worth of work. The system fails the transparency test.
All this information is interesting, but what it means for any particular school or district is far from clear, even after the graduate seminar and that’s how it should be, that’s reality…all the test score data, and graduation rate data, and attendance data in the world isn’t going give you a full and true picture of schools and districts. That’s the first way it fails the accuracy test, a little more below.
With “accountability” the order of the day, the “accountability” mavens know that people want something easily swallowed (if not digested), so the Wisconsin team has given each school a score, based on those calculations that take 62 pages to introduce. That score is what everyone looks at, everyone remembers and everyone seems to think has some profound meaning. What you really have is a Rube Goldberg machine of black boxes inside black boxes that spits out a number. That number hides all the questionable choices in the measures and manipulations, as well as all unmeasured and unmeasurable things that contribute to or detract from school quality. Some in Wisconsin were proud that we didn’t assign letter grades like Florida has, but the number is just as bad, or even worse because superficially something like 66.7% seems to have more scientific accuracy., an a B-. It doesn’t. Superintendent Tony Evers and others have said many of the appropriate things about over-interpreting the scores given schools, but they put the score there and because of the inclusion of the score, the system fails the communication test.
This failure reminds me of the misuse of NAEP cut scores that is central to the accountability system, used for sorting individual students and in the growth scores sores that only recognize movement between NAEP based levels, not within them. This is what the National Academy of Sciences publication, “Grading the Nation’s Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress,” says about these cut score in chapter 5, “Setting Reasonable and Useful Performance Standards (I’ve quoted this before here, “The news from Lake Gonetowoe“):
Although standards-based reporting offers much of potential value, there are also possible negative consequences as well. The public may be misled if they infer a different meaning from the achievement-level descriptions than is intended. (For example, for performance at the advanced level, the public and policy makers could infer a meaning based on other uses of the label “advanced,” such as advanced placement, that implies a different standard. That is, reporting that 10 percent of grade 12 students are performing at an “advanced” level on NAEP does not bear any relation to the percentage of students performing successfully in advanced placement courses, although we have noted instances in which this inference has been drawn.) In addition, the public may misread the degree of consensus that actually exists about the performance standards and thus have undue confidence in the meaning of the results. Similarly, audiences for NAEP reports may not understand the judgmental basis underlying the standards. All of these false impressions could lead the public and policy makers to erroneous conclusions about the status and progress of education in this country. (Emphasis added)
The NAE-based cuts scores (WKCE scores “mapped” to NAEP are also being used with the results of individual students. Here’s what the people at NAEP say about that:
Does this mapping method allow us to link student scores received on state test to the NAEP scale? If not, why not?
No, student scores cannot be linked to the NAEP scale because the NAEP does not generate reliable scores at the individual student level, only average scores for groups of students (e.g. males, females).
I would hope that at least the DPI staff working on the “Accountability” system knew this. If they didn’t, that’s a problem; if they did and went ahead anyway, that’s a bigger problem.
In terms of accuracy, the Report Cards do one thing well, they sort schools by their relative poverty. Here is what Gene V. Glass wrote on this:
What emerges from this dog’s breakfast of numbers? A measure of the wealth of the community in which the school is located. The correlation between the OAI and the “% Economically Disadvantaged” in the school is nearly -.70. That means that the poorer the children in the school, the lower is the school’s number on the Overall Accountability Index; and the relationship is close. In fact, a correlation of .70 is even tighter than the relationship of adults’ height to their weight, and both measure a person’s size. So what the DPI has created is a handy measure of a community’s wealth (SES, Socio-Economic Status) without ever having to ask anyone their income.
Steve Strieker observes that this isn’t news to many of us:
DPI’s own school report card data proves what Social Context Reformers have been trying to highlight for years: Poverty is the eight ball for public education.
Even an amateur’s analysis of the state’s school report card data is telling.
- A supermajority of Wisconsin’s public schools with over 70% economically disadvantaged students were graded “Failed to Meet Expectations.”
- Almost all below-standard schools had at least 45% economically disadvantaged students.
- In contrast, almost all graded schools with less than 10% economically disadvantaged students were considered by DPI’s measurement to surpassed expectations.
Social Context Reformers must not be shouted down by the “no excuses” reformers who will surely shame Wisconsin schools graded below expectation by showcasing the few schools with high poverty rates and high-test scores.
Given this pattern and what we know from 1,000 sources, the remedy should be to provide additional, appropriate help to high poverty schools. We didn’t need the Report Cards to tell us that.
Unfortunately the new system fails this test too. Most of “help” under the new system is directed to Title I schools. In theory, Title I schools are high poverty schools, but not all high poverty schools are Title I. In Madison and some other districts, for reasons I’ve never understood, only pre-K-5 schools are Title I, which means that no matter how high poverty (or low scoring) middle and high schools are left out.
In this case, that is probably for the best, because the “help” being offered appears to be more of a diversion of resources than an addition. No extra resources will be provided and some of the scarce resources available must be reallocated to questionable purposes.
The “Schools Below Expectations, and Significantly Below Expectations” will be required “to submit a plan detailing the extended learning opportunities for eligible students.” And:
[S]chools must participate in an online district-directed diagnostic review of the current core reading and math curriculum including interventions for struggling students. The school must develop an improvement plan based on the diagnostic review, and implement RtI, working closely with the Wisconsin RtI Center. Specific interventions in the plan must address identified problem areas. The plan must be approved by DPI…o DPI will conduct electronic reviews of each school’s progress and monitor throughout the year.
So extended learning, an online review, with an online plan, and online monitoring.
For “Schools Persistently Failing to Meet Expectations” extended learning is also mandated, the diagnostic review will be onsite, and also must result in an approved plan. But there is a kicker, and the name of that kicker is privatization: “Schools must contract with a state-approved turnaround expert/vendor to implement reform plans aligned to the diagnostic review.” In other words, schools have to take money from the classrooms and give it to the likes of Paul Vallas and hope for the best (here is a selection of posts on “turnarounds” from Diane Ravitch, read them to understand my skepticism). And when the turnaround fails, as they almost always do, here is what happens:
o For public schools that do not participate in the diagnostic review, improvement planning and interventions with turnaround experts, they will close.
o For schools that do participate but fail to show demonstrable improvement after three years, the State Superintendent will intervene. Pending legislation, in the case of schools participating in the Parental Choice Program, the state will remove the school from the program. In the case of charter schools, the authorizer must revoke the charter.
Arne Duncan has always liked school closings. I think it is safe to say that the system fails the “direct appropriate resources to those schools that need improvement” test also.
Superintendent Tony Evers also has some school finance proposals that he has been touting. Unfortunately, his Fair Funding for Our Future plan does not include directing any extra resources to high poverty schools or even those identified as in need by this accountability system. The Fair Funding plan does some good things, but addressing poverty is not one of them. For years the only state program directing resources to classrooms based on poverty is the Student Achievement Guarantee in Education or SAGE), which only targets the early grades and in budget cutting moves over the last few years, done under the rhetoric of “flexibility” has been eroded by larger allowed classes and new allowances concerning the grades covered. There does not seem to be any desire to change that, either from DPI or the legislature.
Fair Funding claims that it “Accounts for family income and poverty.” In sense it does, but via tax relief for property owners, not by giving schools serving students in poverty the resources they need to meet their challenges. Under Fair Funding student poverty levels will be factored into calculations of state aid, but revenue limits will not have a poverty bump and there is no new categorical aid for students in poverty. So property taxpayers in districts with higher poverty will have lower taxes and the schools will not have an extra penny (btw –the Penny for Kids proposal from the Wisconsin Alliance for Excellent Schools/ Opportunity to Learn Wisconsin includes a poverty based categorical aid). So the Widow Hendricks of “divide and conquer” fame who owns property in multiple high poverty districts gets a tax break and the students of Beloit and Janesville get nothing.
Back to the titular questions, who cuts the barber’s hair?; who holds the people behind this mess created in the name of “accountability” accountable? We all need to.
Start at the top. For Arne Duncan, join the thousands who have signed the “Dump Duncan” petition. There is also an election on November 6th and Duncan’s boss Barack Obama is up for re-election. Diane Ravitch has made a case that “as bad as the Obama education policies are, they are tolerable in comparison to what Mitt Romney plans.” Others concerned with education, especially those not in swing states, should take a good look at Jill Stein.
In Wisconsin, for Senate and the House, more-or-less the same situation exists. Some version of NCLB/ESEA will certainly be before Congress, and for that all of the Democrats on the ballot are better than the Republicans, but none have distinguished themselves on Education issues the way Russ Fiengold did. Still, I’ll be voting for Tammy Baldwin and Mark Pocan and urge you to do the same. I’ve already warned Mark that I’ll be contacting him regularly on Education and other issues and calling on him be more of a progressive champion on this blog, just as I have when he was my State Rep.
That’s another version of accountability. It starts at the ballot box, but it doesn’t end there. Our elected officials need to here from us, all the time. They need to know — as we sing at the Solidarity Sing Along — “We’re not going away.”
At the state level, we don’t get another crack at Scott Walker this year, but there are State Senate and Assembly races. Again, the rule is Democrats better than Republicans, but there are also some Democrats who are not only better than Republicans, but are real supporters of education. The two I’d like to point to are Melissa Sargent (who is a good friend) and Mandela Barnes (who I have admired from afar). By all accounts the Senate is the key this time around. The key races where your support ($$$ and time) may make difference appear to be Susan Sommer, Jessica King,and Dave Hansen. Keeping the Senate is the best way to keep Walker in check. With all this, it is important to remember that the “Accountability” system has been presented as a work in progress and there is some legislative power to dictate changes in some areas (the Report Card portion did not require legislative action, but other parts of the waiver did), and that any changes to school funding — good or bad — have to go through the legislature. With the State Legislature, this time around accountability means at minimum limiting the power of the Walker allies who aided in the creation of the “Accountability” system.
I’ve saved Superintendent Tony Evers for last. He is up for re-election in April 2013 and as with all elected officials, the best place to assert accountability is at the ballot box. It is also likely that come April, Evers will be the better choice (I supported Todd Price in the Primary last time and Evers in the General Election against Rose Fernandez). Also as with all elected officials, imposing accountability includes making sure Evers hears from the voters throughout his term, both positive and negative, and I hold some hope he may listen and adjust his course.
There is much I like and admire about Evers, but as the above indicates there are many things he has pushed that I think are bad, wrong or at very least should be better. I understand that most of this was done in the context of a state in the control of the Fitzwalker gang and a federal policies set by Arne Duncan. Given the circumstances, it is impossible to tell which things he truly believes are good for our state and our students and which are pragmatic choices made in order to keep a seat at the table and maybe deflect even worse policies (one example where I believe he did this was the mandated grade retention that Walker initially wanted in the Read to Lead legislation). This situation keeps bringing to mind something Anthony Cody wrote recently about teacher leaders:
How can we make sure that we are not being used as tokens? For this, we have to look at why we are being asked to join the conversation. What are the power dynamics at play? Do we have a vote when decisions are to be made? Will we find allies around the table to help us have some influence? Do we have any real cards to play? This gets us closer to defining what real leadership is all about. Real leadership is not just the ability to speak with clarity and authority based on our experience in the classroom. It also involves a relationship to other teachers, and to some level of political power in these situations.
The bottom line is that we do not have the money to buy influence. We have to get it the old-fashioned way. We have to organize for positive change at our school sites. We have to join with others at our union meetings, and as our colleagues in Chicago showed, we may need to go on strike. We have to build strong relationships with our colleagues, with parents, with allies in other unions and social movements, and with reporters, and use this strength as the basis for our ability to speak for ourselves. We have to organize and build our strength from the ground up, because the strength that comes from the top down is like the strings on a marionette.
Thomas J. Mertz
AMPS 