Monthly Archives: August 2009

Timely Material — “Accountability, Rigor, and Detracking: Achievement Effects of Embracing a Challenging Curriculum As a Universal Good for All Students”

Vodpod videos no longer available.

posted with vodpod

An article and video from the Teachers College Record on Detracking seem timely with the MMSD Board vote on a very slightly revised Talented and Gifted Education plan scheduled for Monday August 17 (public comment at 6:00 PM).

The interview is with Professor Kevin G. Welner who had a great essay “Obama’s Dalliance with Truthiness” in TCR earlier this month.

Needless to say, the reforms that Welner describes are very different than what MMSD is poised to enact.  Madison is moving toward increased ability grouping.  I do not believe the  unrepresentative Advisory Committee ever considered Detracking.  They certainly did not place that option before the Board or community.

I hope to have more on the TAG plan posted before Monday, but travel plans may make that impossible.  Meanwhile more of my concerns are expressed in this post.

Thomas J. Mertz

Leave a comment

Filed under Accountability, Best Practices, education, Equity, Local News, Uncategorized

Sherman Dorn Asks THE QUESTION and Offers Some Answers

riddler45cover1Longtime readers should know that Sherman Dorn is one of my favorite people in the edusphere. His  recent “How can we use bad measures in decisionmaking?” is a fine example of why I value his contributions so much.

His titular question is THE QUESTION at the heart of so much ed policy action these days.  Nobody who isn’t seeking profits or losing their mind likes the tests being used — not Arne Duncan, not Barack Obama, not the people in Madison poised to build a Gifted Education house of cards on them — but almost nobody wants to give up on the tests and many want to expand their use (Arne Duncan, Barack Obama, those house of card builders in Madison).

Everyone talks of better tests, multimodal assessments, new ways of looking at data….  All this can be good, however we aren’t there yet and the simple-minded attraction of letting the flawed data “drive” education policy is strong (the current draft of the MMSD Strategic Plan has both reasonable  data ” inform[ed]”  and frightening “data driven” language).    Additionally, at least three truths often get lost when better assessments and data are discussed (Dorn hits most of all of these).

  1. All assessments and data are of limited utility.  They are snapshots at best; they are only designed to measure specific things; standard deviations and confidence intervals recognize some of the limits, but are rarely part of “accountability” discussions.  the temptation to use assessments for things they are not designed for is always there.
  2. Because better assessments should mean assessing more things in more ways,fulfilling this promise will result in more time and resources devoted to assessment and analysis and less to teaching and learning.
  3. Employing multiple assessments or sophisticated data analysis (ie Value Added) moves away from transparency in accountability. It already clear that few policy makers, much less members of the public, understand the nature of current assessments and accountability practices.  When you employ Value Added techniques all but the most statistically adept are shut out (some Value Added methods are proprietary and even those who commission the analysis are kept in the dark about the nature of that analysis; others are open, but beyond the understanding of most people).   Combining multiple assessments, including qualitative approaches, produces similar issues.   The MMSD Gifted plan is a perfect illustration.  They promise to identify potential and achievement with referrals and multiple assessments over five domains (academic, creative, leadership, visual and performing arts) and then decide who gets the extra services based on “percentile scores.”  Does anyone think that the promised “transparency” of this exercise will be meaningful to parents and Board members?

This was supposed to be about Sherman Dorn’s post, so back to that (although I think the above — especially the local stuff — is a salient context for what Dorn wrote).

After much good introductory material (including a link to the relatively recent, must read Broader, Bolder Approach Accountability Paper), Dorn explores a variety of positions relative to the problems  of “data that cover too little,” and “data of questionable trustworthiness.”  His presentation of their strengths and weaknesses is insightful and informative.

Dorn himself rejects both the “don’t worry” and “toss” extremes and seeks to extend (begin?) the conversation in pragmatic directions.  Here is how he closes:

Even if you haven’t read Accountability Frankenstein or other entries on this blog, you have probably already sussed out my view that both “don’t worry” and “toss” are poor choices in addressing messy data. All other options should be on the table, usable for different circumstances and in different ways. Least explored? The last idea, modeling trustworthiness problems as formal uncertainty. I’m going to part from measurement researchers and say that the modeling should go beyond standard errors and measurement errors, or rather head in a different direction. There is no way to use standard errors or measurement errors to address issues of trustworthiness that go beyond sampling and reliability issues, or to structure a process to balance the inherently value-laden and political issues involved here.

The difficulty in looking coldly at messy and mediocre data generally revolve around the human tendency to prefer impressions of confidence and certainty over uncertainty, even when a rational examination and background knowledge should lead one to recognize the problems in trusting a set of data. One side of that coin is an emphasis on point estimates and firmly-drawn classification lines. The other side is to decide that one should entirely ignore messy and mediocre data because of the flaws. Neither is an appropriate response to the problem.

I probably don’t do justice to his post.  Read the whole thing.

The reality is that bad data is being used and that the uses are expanding.  I am not as sanguine as Sherman Dorn about the potential for better data and better ways of using it (I’m guessing he’d object to the word sanguine here, and he’d be right because it does not capture where I think he is coming from.  Take it not as an absolute but only as a comparison with me), but I do know that explicit discussions of the issues involved like Dorn’s post are necessary to progress.

Thanks Sherman for the questions and answers.

Thomas J. Mertz

Leave a comment

Filed under Accountability, Arne Duncan, Best Practices, education, Local News, National News, nclb, No Child Left Behind, Uncategorized

Big News out of Milwaukee (Updated)

Mural Milwaukee SkylineGovernor Jim Doyle (or his reps) , Mayor Tom Barrett (or his reps), and others (maybe Arne Duncan’s reps) are holding secret meetings to hijack the MPS Innovation and Improvement Advisory Committee for a Mayoral Control proposal.  MPS Board President Michael Bonds has resigned from the Committee in Protest.

Lisa Kaiser has the full story, including excerpts from Bonds’ letter and reactions from the Mayor’s office.  Milwaukee Talkee is looking for action to stop this and there is an online petition here.

Jim Doyle likes his secret meetings, Arne Duncan likes his Mayoral control, lots of elections to be considered with the expectation that an MPS shakeup would buy Doyle and Barrett some time; the Race to the Top beauty contest is part of this too.

Notice how none of this has to do with educating the students.  Notice also that allowing Doyle and Barrett to say, “give the reforms a chance” and the Race to the Top funding are only short term remedies.  At some point the chickens do come home to roost.

In a related note, The New Teacher Project gave Wisconsin’s chances for Race to the Top funding a very low rating.  Mayoral control could change that.  That said, I’m more than wary of making big changes in order to buy a lottery ticket in what is likely a rigged game (that goes for the use of bad student tests for teacher compensation decisions too).

Update:  The Journal-Sentinel has more this morning, including an endorsement of the Doyle/Barrett plan from State Superintendent Tony Evers.  Mayoral control was not included in the “Milwaukee Public Schools – An Agenda for Transformation” Evers campaigned on; his opponent — Rose Fernandez — pushed for dissolving the Milwaukee Board of Education and replacing it with a team appointed by the Mayor, the County Executive and the State Superintendent.

Thomas J. Mertz

2 Comments

Filed under "education finance", Accountability, Arne Duncan, Budget, education, Elections, finance, Gimme Some Truth, Local News, School Finance, Take Action

Following the Money — MMSD Budget Updates (part 1)

segal-600sub

Clyde McPhatter & the Drifters-“Money Honey” (click to listen or download).

Busy, busy time at the Madison Board of Education.

At the Monday, August 10th meetings, in addition to the Talented and Gifted Program Plan, new limitations on the transportation of homeless students, middle school report card evaluation (more linked here), Singapore Math, and much, much else; there are big items involving the Budget Gap created by the State Budget and the use of American Recovery and Reinvestment Act Title I and IDEA funds.

The Budget Gap document is interesting and intriguing, though not terribly detailed. For those who need a reminder of the situation the district finds itself in, here is how Supt. Dan Nerad describes their circumstances:

The amount of revenue the district is projected to lose amounts to $2,810,851 for the 2009-10 school year compared to the preliminary budget approved by the board of education. This amount is due to the decrease in numerous categorical aids the school district receives annually and the reduction of the per pupil increase from $275 per child to $200 per child.

The amount of state aid the school district is projected to lose is in 2009-10 is approximately $9.2 million. Under current revenue limit laws, for every dollar of state aid lost, the school district would have the ability to increase taxes by that same amount. Over the past month, administration has worked to mitigate the tax impact due to the loss in state aid.

There are two documents attached as initial recommendations to the Board for discussion. The first is described as utilizing “four specific budget areas to apply a decrease in funding without a reduction or elimination of programs and services.” There are only three listed.

These three are the fund for contingencies and re-budgeting the Elementary Teacher and substitute projections “based on historical” reviews and analyses (I’d like to see these!). These adjustments total $2,810,851.

This all seems good. The re-budgets are related to issues around the Fund Balance and “Salary Savings.” It gets a little complicated, but I think it is worth at least sketching (if you aren’t interested in a little history, please skip down because there is important current stuff below).

“Salary Savings” is an accounting concept that some entities use (some abuse it too). It is based on the reality that allocations for staff are often unfilled — temporarily or permanently — or filled at costs below the allocation. A classic example is when a teacher goes on leave. If for some portion of that period a substitute is employed, the difference between the salary and benefits of the teacher and the substitute would be a (salary) savings. Many public entities budget salary savings at between 1% and 2% of their personnel budgets.

In the minds of some, MMSD abused Salary Savings for a few years. In order to limit programmatic cuts they used unrealistic projections. This led to spending down the Fund Balance.

As the 2007-8 budget was being prepared and considered the district fell under the sway of a fiscal conservative over-reaction. Salary Savings had recently been budgeted in the range of $5 million to $7 million (if I remember correctly) and the reality had been about $2 million or $3 million less (again, if I remember correctly…the figures might be off, but the trends are right). The Fund Balance was shrinking and some Board members were making political hay on this.

When the initial 2007-8 budget was prepared, Salary Savings were projected at $1 million. You may remember that budget. The 2007-8 fiscal year was the year that MMSD almost closed schools; did decimate locally funded class size reductions, did reconfigure special education; did initiate the Athletic Director mess; did reduce allocations for psychologists, social workers, reading specialists and English Language Learning; did eliminate yellow buses for Wright and Spring Harbor, did initiate the private school busing mess.

That budget produced a Fund 10 surplus of $4.3 million. The 2008-9 budget also under-budgeted Salary Savings and a surplus is anticipated.

Another way of phrasing this is that these budgets over budgeted salaries and benefits. That is the way Asst. Supt. Erik Kass likes to think of it.

Erik Kass doesn’t like the concept of salary savings and is working toward more accurate initial allocations. That’s the basis of these re-budgets. The history is why I think it is sound policy.

The third part is the “Tax Impact Solution,” and I am not 100% clear on how to read this, but can at least point out some things. Below is an unclear image file, click here for the full document pdf.

tax impact

The biggest news here is that MMSD is proposing not to tax to their maximum revenue limit in 2009-10. Of $245,140,473 in taxing authority, MMSD proposes to use $237,625, 752; they have authority to increase projected taxes on a $250,000 home by $162.50 and the proposal is to raise taxes on that home by $82.50 (this is from all funds).

As Governor Jim Doyle and others continue to pretend that the revenue limits are all that is wrong with school funding in Wisconsin and pass budgets and propound schemes (and here) that shift more of the burden for educational investments to local property taxes, the fact that a prosperous district in a community with a track record of valuing education does not have the political will to tax to the limit is hard evidence that the problems go well beyond the revenue limits and require a comprehensive solution. New Berlin and Appleton are among the other districts that are not taxing to the max. This is a trend I will be posting on in the future.

Technically, MMSD is proposing to tax to the max with the Fund 10/General Fund and is cutting the taxation for debt service and for Fund 80/Community Services. I believe this is related to state laws that take away revenue authority if it is not used in consecutive years. The way they are doing things should provide more flexibility to not tax to the max in 2010-11.

Whatever happens with the 2010-11 budget/tax decisions, things will be ugly that year and the year following. The projections in this document are for a $110 tax increase on a $250,000 home in 2010-11. The pressure against that increase will be high. 2010-11 will also bring more cuts in categorical aids, as well as more cuts due to the structural gap between allowed revenues and costs. 2011-12 will bring the end of stimulus funding. The maintenance referendum runs out some time in here too. Repeat after me: “We need comprehensive school finance reform.” Send that message to the powers that be.

The document is not clear on how the debt service will be paid. My guess is that the Fund Balance will be used.

The second part of the document promises “no reductions in programs or services,” but the third part reduces the projected levy for Fund 10 from the preliminary budget by $900,00 (the preliminary budget had reduced that levy by $1.5 million already and used the Fund 10 Fund Balance to make up most or all of that amount). There may be some cuts here, or there may be more use of Fund Balances.

Stay tuned for more as things develop.

This post is titled “part 1”; part 2 will be on the ARRA plans. I don’t know when I will get to that. Here is a link to the document. For now, a few highlights/observations based on a first read:

  • Proposals for these monies include some 4 Year-Old Kindergarten start up costs, some Ready, Set, Goals costs in Title I schools that may not be within the guidelines. Communication with DPI is ongoing.
  • Culturally Relevant programming/development/staffing is included.
  • Dual-Immersion implementation is included.
  • There is some parent/community outreach development funding.
  • Millions in technology purchases are included .
  • This money will go away in two years!
  • Some of the needs addressed will not go away.
  • This will mean that cuts must be found here or elsewhere.
  • We need comprehensive school finance reform.

Thomas J. Mertz

1 Comment

Filed under "education finance", Best Practices, Budget, education, Local News, School Finance, Take Action, Uncategorized

MMSD “Gifted” Program Plan — No Thank You

roy-lichtenstein-no-thank-you

Roy Lichtenstein, "No Thank You."

The Madison Board of Education will be considering a “Talented and Gifted Program Plan” at their Monday, August 10, Student Achievement and Performance Monitoring Committee meeting. It is on the agenda as an action item for the August 17 regular Board meeting. This plan was developed outside of the public eye — no noticed meetings of the Advisory Committee, no published minutes — and the first look any interested citizens have had was when it was posted on Friday. August 7.*

Although there are some things I like in the plan, my initial overall reaction is “no thank you.”

My number one reason for urging rejection is the lack of attention given to demographics in the plan. The word “minority” only appears in reference to the Minority Student Achievement Network, the phrase “low income” does not appear. There is nice language about increasing the “identification of students from underrepresented populations,” and researching “additional assessment tools that are non·biased, multi·cultural,” but at the time of implementation (September, 2010), the screening and identification measures that will be in place (the WKCE, the Primary Math Assessment, the Primary Language Arts Assessment, TOMAGS, Writing Samples, Middle School Math Assessment, referrals, and even the Torrance Test of Creative Thinking) do not have great track records in these areas (there is also a mention of MAP tests, but the meaning of that mention is not clear). Additionally, with the exception of the Torrance (and maybe TOMAGS) tests, these are all achievement, not ability, measures. They are not designed to be employed in the manner contemplated by the plan.

I said there are some things I like. These have some potential. I’m going to touch on three now (there are more — like the consistency throughout the district — that I don’t have time to delve into).

The first one is the requirement that the evaluation use data disaggregated by demographics (it also calls for annual reports, but as the case of the Equity Policy indicates, don’t hold your breath waiting for that). This is good. It would have been much, much better if the committee had instead begun their work by assessing the the current demographic inequalities in Talented and Gifted and advanced opportunities in MMSD.

The second is a guideline from the National Association for Gifted Children. In general, I am uncomfortable with the cut-and-paste inclusion of materials from a lobbying/advocacy organization in the plan, but this one is good:

Gifted programs must establish and use an advisory committee that reflects the cultural and socio-economic diversity of the school or school district’s total student population, and includes parents, community members, students, and school staff members.

This is clearly not the case with the current committee (is there a single member whose family income would qualify for free or reduced lunch? Do the families of 44.6% of the members?). Unfortunately, the locally produced “action steps” call for the current committee to continue and a sub-committee to be formed from current membership. This needs to be changed.

I also like the “Differentiated Education Plans,” but limiting these to “Talented and Gifted Students” is wrong. The one thing from the Strategic Plan draft that I can endorse without reservation is the policy of Individual Learning Plans for all students. This would be a wonderful but expensive reform. Only individualizing for those perceived as “Talented and Gifted,” is offensively inequitable.

Back to the things I don’t like (as you can tell, the lines are a bit blurry). Just two more that I have the time to address.

The plan asks the district to commit to “cluster grouping” in classroom assignments. The research on the benefits of “cluster grouping” is thin, the applicability of this research to Madison is questionable, and the potential for harm is great.

Since the Advisory Committee did not deign to provide the Board or the public with an extended exploration of what “cluster grouping” is, I’ll offer a little more here (contrast with this literature review provided to the Clayton, Mo. Board and note that if MMSD wants to do its own version there will be no extant research on that version).

According the Cluster Grouping Handbook, the practice requires dividing students into quintiles based on “local criteria” (that means those tests and referrals that have been so successful in bringing diversity to Talented and Gifted programs in Madison and elsewhere). The five groups are labeled Gifted, High Achieving, Average, Below Average, and Far Below Average. In classroom assignments, the Gifted are isolated from the Far Below Average and the High Achieving; the students are otherwise mixed. I repeat, the main “innovation” is to keep the “Far Below Average” and the “High Achievers” away from the “Gifted.”

Every student in Madison will be slotted into one of these categories and decisions concerning their education will be made based on incredibly imperfect assessment tools. We are talking about five and six year olds.**

As I said, the research on the supposed educational benefits of this is thin (in contrast, the research on the flaws of these assessment and referral practices, and on the harm done by labeling, is voluminous). The authors of the Handbook cite exactly two empirical studies. One of these is the unpublished Doctoral dissertation of Handbook co-author Dina Brulles. I could not find any peer-reviewed or non peer-reviewed publications of Dr. Brulles research. The other is a widely cited (but also not peer-reviewed) study by Marcia Gentry. My extensive searches of databases turned up other empirically based publications by Gentry, but none by other authors. I would welcome any citations.

Doctor Gentry conducted her research in schools that were less than 1% minority. Given the history of grouping practices and the demographics of Madison, I think extreme caution should be used in asserting the reproducibility of Gentry’s results in our district.

Additionally, the lack of diversity in her study means that important issues such as segregative and unequal impacts were not examined. At the very least, prior to implementation, the Board of Education and the public should be provided with assessments of segregation based on a trial run of the cluster grouping scheme. If the practice is implemented, these factors should also be included in all evaluations.

One of the asserted advantages of “cluster grouping” over tracking is that it allows for mobility among the quintiles. If cluster grouping is implemented, measures of this mobility in practice should also be part of evaluations. As Doctor Willis D. Hawley has noted, “Ability grouping often turns into tracking” (read the whole linked document for a fine introduction to the issues).

Cluster grouping may appear to be a politically attractive compromise. I am generally wary of both political compromises and grouping schemes in education policy, especially when the research basis for the desirability of the compromise is almost nonexistent. I urge the Board of Education to be wary also.

The other big area where I find the plan lacking is not about Talented and Gifted programing per se, but about advanced and honors programs and courses. The plan includes these action steps:

Develop a plan to increase participation of students in advanced courses, including support systems for transition to and completion of courses, and greater consistency in eligibility requirements across the District.

Review the design, implementation, and requirements for District embedded honors courses. Survey teachers, parents, and students to determine effectiveness and interest.

The idea is good, but do we really need a plan and a review? Here is what the Equity Task Force said in a recommendation that was never discussed in public by the Board of Education:

Open access to advanced programs, actively recruit students from historically underserved populations, and provide support for all students to be successful.

Pretty simple and it could be done tomorrow.

I’ll admit that some work would be needed to determine which courses have legitimate prerequisites, but with little effort, things like the “advanced biology placement test” and screening for 8th Grade Algebra could be tossed in the dustbin of history where they belong.

These entry level courses, the first rungs of the ladder, are the key to opening access; the barriers need to go. As long as the district puts these first rungs out of the reach of students who want to be challenged, inequalities will continue to be reproduced and upward mobility will be exception not the rule.

One related side note. These barriers are the reason I hate the “raise the bar” language that the Board of Education recently added to the Strategic Plan. We have enough bars keeping people out; the ambiguity invites more and higher bars.

Last thought is that the long-term costs — financial but also human — of this plan are not clearly explored in the document (there is offered a one year, prior to implementation figure of $83,000).

I’ll be offering some version of this critique in public testimony to the Board on Monday, August 10th. If you have thoughts — whether in agreement or disagreement with what I have written — please join me or email to comments@madison.k12.wi.us.

Thomas J. Mertz

*[Note: Because the Advisory Committee that produced the plan was not appointed by Board of Education, it is unclear if open meetings statutes are applicable, see: Staples Correspondence, February 10, 1981 and a 1991 memo to former MMSD Legal Counsel Clarence Sherrodd. If the Board approves this plan with the authorization of a continued Advisory Committee, that committee will be required to post meeting notices and publish minutes. Whatever the legality, it is not a good policy to have a plan which will effect every child in the district drafted without public scrutiny or input.]

**[Note: I am not clear what procedures are currently being used by MMSD although I have heard talk that some “cluster grouping” is going on. For sketchy information on current practices, see here and here.]

Leave a comment

Filed under Accountability, Best Practices, education, Equity, Local News, Take Action, Uncategorized

Race to the Bottom? – Quote of the Day

declineOverall, our results consistently indicate that the increased focus on individual teacher performance caused a sizable and statistically significant decline in student achievement. This decline in achievement is also much more pronounced in the case of national exams with an e ffect of up to 40% of a standard deviation. As in the different effects in terms of internal and external results, our triple-difference evidence also documents a significant increase of grade inflation. In addition, in support of a causal interpretation of our results, we also find that in almost all specifications and dependent variables there are no significant differences between the treatment and control groups over time before the introduction of merit-pay. Finally, the inclusion of different control variables or the consideration of different subsets of the data makes only very minor differences to the size of our estimates, as would be the case if assignment to treatment were random.

Graph and quote from Pedro S.  Martins, “Individual Teacher Incentives, Student Achievement and Grade Inflation,” Institute for the Study of Labor (2009).

In 2007 Portugal instituted a merit pay plan.  Azores and Madeira (the graph above) and private schools were excluded.   Using these as a control, the quoted study found that this merit pay plan resulted in a decline is student achievement.

Arne Duncan and Barack Obama have made incentive pay plans a centerpiece of their “Race to the Top” scheme.  It may be a path to the bottom.

More on the “Race to the Top” later this week.

Thomas J. Mertz

Leave a comment

Filed under Accountability, Arne Duncan, Best Practices, Contracts, education, Gimme Some Truth, National News, nclb, No Child Left Behind, Uncategorized