Update, 8/14/17, 1:14 PM: The Board just received an email that the IMA vote meeting will be rescheduled to 8/21/17.
Note: I began this post in January 2017, and then set it aside. As a result, some of the introductory material has been superseded. For better coverage of events since January (and some prior to January), see this post and this one. More recently I finished the sections on footnotes 5 & 7, added the section on footnote 27, and wrote the conclusion. I like the conclusion, so I am going to quote it here: “ If the people behind IMA can’t do better with their proposal, how can we trust them with our children and our money?“
Update (8/13/17): This week, the MMSD Board of Education will have at least one, and maybe two important meetings on Charter Schools. On Monday 8/14 the Operations Work Group will be discussing a new, much more “Charter-friendly” policy. It is being recommended that the Board approve this at the end of the month. The last time the Board revised the Charter policy, it took six months and many meeting. We may not want to take that long, but there are too many things — involving both what is there and what isn’t — that need examination and (Board and community) discussion to cover in two meetings, with a vote at the second.
The second meeting, for a vote on the IMACS proposal on Wednesday 8/16/17 does not appear on the legal notice (despite my request to add it). Board Members are now being told this may be rescheduled (to my knowledge no Board Members were asked about their availability before this meeting was scheduled by the Board President, and none have been asked about their availability in relation to a possible rescheduling). At this time we the Board do not have any update on a revised contract in response to the votes in July (the language of those votes is here). We have been told “The goal is to get a draft document out prior to the meeting,” but there is no guarantee that will happen (we have also been told by the President that no public comment will be scheduled). I am not clear why there is a rush. Our policy calls for contracts to be concluded about five months prior to the school opening, IMA will have about twelve months if approved. I do know that it would be a huge mistake to approve any contract without sufficient time for the Board and community to review. Stay tuned.
Foot(note) Stompin’ Post
Foot Stompin’ by the Flares (click to watch/listen)
During public discussions of the Isthmus Montessori Academy (IMA) proposal to open a charter school with the Madison Metropolitan School District (MMSD), I repeatedly stated that the matter before the Board of Education was the proposal submitted, not the role of charter schools, the value of Montessori education, or a host of other things. Other issues may contribute to our thinking, but as a Board we have an obligation to follow our policy (possible revisions of the MMSD Charter School Policy are scheduled for discussion and vote in May — Note: Now August, see update at the top) and state statutes, which means considering each proposal on its own strengths and weaknesses, according to set criteria (the required National Association of Charter School Authorizers Guidelines here). The majority of this post concerns weaknesses of the IMA proposal in the area of citation and footnotes, but first some background.
It is undisputed that the IMA proposal did not meet the established criteria (nor did the initial proposal meet the criteria to go to the next step). On this basis MMSD staff did not recommend approval (memo here, rubric here, minutes here). However by a vote of 6-1 (I dissented) the Board (emphasis added)
[A]pprove(d) the proposal for the Isthmus Montessori Academy CharterSchool and that the District enter into a contract, consistent with applicable Board policy and state law, to establish the Isthmus Montessori Academy Charter School as an instrumentality charter school for a period of five (5) years from July 1, 2018 until June 30, 2023. Such approval shall be contingent on the Isthmus Montessori Academy Charter School supplementing their current proposal to address the following areas of concern by no later than May 31,2017:● Student Body and Demographics● School Data● Curriculum, Assessment and Instruction● Financial Operations
With a second vote (6-1 also, I dissented again), the Board took care of some housekeeping requirements, and set July 1, 2017 as the contract deadline.
[See the update and the linked posts at the top for more accurate information on how events have unfolded] This is where things stand now, by May 31, 2017 MMSD staff — without a formal role for the Board — will decide whether the requested supplementary materials are sufficient to meet the requirements, if they are deemed sufficient, contract talks will begin and prior to July 1, 2017 that contract will come to the Board for approval (not amendment, this is a yes or no vote). Among my objections to this process was that many (perhaps all) of the criteria are subjective, and although I respect the professional opinions of our staff, with subjective criteria I believe the Board should decide.
The Board and the public have also been provided with only minimal information related to the staff analysis of the proposal (this is something I think we need to address in the looming policy revision). In addition to the memo and rubric linked above, a three-page “Financial Analysis” and a one-page (financial) “Key Factors” document were distributed. All total this comes to nine pages of analysis for a 121 page proposal (and given that the financials and rubric have much white space, that is a generous count).
The use of citations or footnotes is among the things not referenced in any of the materials provided. As a college instructor and (recovering) academic, I can’t help but look at and consider the references and how they are employed (part of subjectivity is that different things are important to different people). For me, footnotes serve two important purposes. First, they allow the reader to verify the information provided in the main text. Second, they provide some clues to the intellectual journey of the author(s).
On my second read of the final IMA proposal, I began making notes for myself on this topic. What follows is revised version of those notes.
The general format is a cut & paste of the passage from the proposal that contains the footnote (because of formatting issues, these are mostly images, not text), the footnote with a link to the source material, followed by a brief discussion of the source material and how it does or does not support or shed light the passage (there is also a digression on Lumin Education included for reasons that should be clear). Here we go.
Passage and Footnote #1
1 Efimova, V., and F. Ratner. “Integrating the educational principles of Maria Montessori in the process of pedagogical support for pupils with learning disabilities.” International Review of Management and Marketing. (2016).
Discussion
For many reasons, the small study — 30 students, 80 minutes a week, for three months — is not impressive.
The authors are Russian academics, and the journal appears on this list of “[p]otential, possible, or probable predatory scholarly open-access publishers” (see also the Calling Bullshit project). Although the Journal title indicates a marketing and management focus, they appear to publish in a wide array unrelated of fields.
In relation to the passage the footnote is applied to, there is nothing at all in the paper about “differentiated curriculum,” “well-rounded education,” “achievement gaps,” “socioeconomic status,” and very little about “self-initiated exploration,” or even “ability backgrounds” (the classification/definition of “learning disabilities” is limited to those without “any evident deviations in functioning of the nervous system, … no marked motor, sensory impairments or intellectual impairment”).
The “educational principles of Maria Montessori” used in the study are described here:
One section of the objective environment in Montessori approach to education was described as the exercises of practical life. The work of the child under the exercises of practical life program entails the fulfillment of various “household” affairs: Sweeping, washing hands, washing a table, shoe-polish and so forth. Any of these exercises mean the solution of direct and indirect tasks. Even if direct tasks (to learn to wash a table, to clasp buttons, to sweep away garbage from a table, to wash dishes, etc.) aren’t connected directly with educational activity, then indirect tasks solved in the course of a child’s activity, are important to the preparation for writing, reading and the development of other learning skills.
Among the conclusions of this “study” is that students with disabilities should be educated in “special centers.”:
The educational principles of Maria Montessori, in our view, fully meet the requirements of the new standards of education adopted in our country. However, the presence of the class-lesson system limits the possibilities for the application of the educational principles of Maria Montessori in a regular school. It seems to us that the process of pedagogical support for children with learning disabilities can be managed in a more flexible manner on the basis of special centers.
IMA has explicitly presented their version of Montesori education as inclusive for students with disabilities, (however it appears that some public Montessori schools in Milwaukee employ staff for “More Restrictive Placements,”examples here and here).
I can discern no good reason why IMA included a citation to this article.
Passage and Footnotes #2 and #6
2“[The Montessori] child-centered classrooms provide for a greater level of student engagement possibly due to more positive teacher-student relationships. It is also possible that students at child-centered classrooms participate at a greater proportion, complexity, and intensity in class work than students in more traditional educational settings do because alternative curricula provides more opportunities to develop higher levels of competency.” Franczak, 2016.
The full cite for Franczak appears elsewhere (the first use should include the full cite): Franczak, Iwona. “Comparative Analysis of Behavioral Engagement and Transferable Skills in Conventional and Montessori Schools.” 2016 NCUR (2016). The passage from the IMA proposal that cite annotates reads:
Discussion
At the time of the study Ms Franczak was an undergraduate student at Eastern Illinois University, Charleston (the journal publishes the proceedings of an undergraduate research conference held annually in Asheville, NC). As far as undergraduate research goes, it is fine work. There are, however, a number of issues.
First, the study says nothing directly about “test scores,”or “achievement gaps,” or even “achievement” (it does make a case – based on other work – that engagement and transferable skills increase academic achievement), very little about “opportunities to follow their own interest and develop deep understanding,” but much on “transferable skills.”
With the second passage, the biggest problem is that this is not a “longitudinal study,” but rather for three non-randomly selected schools, “[o]bservations were conducted at site in three thirty minute intervals during the first three hours of a school day for five days.” Referencing observations that took place over five days as “longitudinal” is a big problem.
It should be noted that the statements on “self control, team work, and problem solving” in the study were not based on anything “measured,” (the word used in the IMA proposal) in the research, but rather are presented as “additional qualitative data” (teacher surveys – no survey instrument is provided – are said to “confirm” these observations, but elsewhere it is noted that there were no differences found among the schools in the parent survey). These are also big problems.
The quality of the research is questionable and IMA mischaracterizes the findings.
Passage and Footnote #3
3 “[The] benefits of parental involvement in school… include: improved parent–teacher relationships, teacher morale and school climate; improved school attendance, attitudes, behavior and mental health of children; and, increased parental confidence, satisfaction and interest in their own education.” Hornby, Garry, and Rayleen Lafaele. “Barriers to parental involvement in education: An explanatory model.” Educational Review 63.1 (2011): 37-52.
Discussion
This is generally fine. The authors are recognized scholars, the journal is respectable, and the information being annotated is generally related to the source cited. The article itself says nothing about Montessori, so it is a little off, but there isn’t anything egregious with this citation.
Passage and Footnote #4
4 Abigail Jordan, Bailey Fern, Chelsea Morris, Rebecca Cross, and Smita Mathur,“Critical Thinking in the Elementary Classroom: Exploring Student Engagement in Elementary Science Classrooms through Case-Study Approach” Journal of Emerging Trends in Educational Research and Policy Studies 5(2014) 6: 673-678, 665.
Discussion
Undergraduate research once more, this time published by an organization on the ““[p]otential, possible, or probable predatory scholarly open-access publishers” list linked above.
Again it is a fine as undergraduate research, but overall not very impressive, and it has little or no relationship to the passage it supposedly supports. The words/phrases “Montessori,” “customized learning plans,” or “systems of support” do not appear. The article does endorse “Constructivist” education, and the relationship between Constructivism and Montessori is complicated and controversial (see here). Here is the conclusion
Through our research, we have found that male students in the elementary classroom often need multiple forms of engagement to learn. Specific examples of engagement include physically stimulating opportunities and moments that involve emotional engagement such as personal connections to material. It is important for teachers to remember that behavioral engagement does not always relate to cognitive success. For students to truly understand and connect to science material, a layered engagement approach must be administered.
I can discern no good reason to cite this article here.
Passage and Footnote # 5 and Uncited Statistics
5 See: www.public-montessori.org/sites/default/files/resources/EDCS%20Outcomes%20Charts%20and%20Graphs.pdf (note: The link from the application no longer works, the document has been archived here).
The “increasing research base” turns out to be two pages of a 2010 newsletter from an organization now called Lumin Education. There is a “research base” on Montessori (parts of which appear elsewhere in the IMA application), but that’s not what this is.
The same organization and statistics from the same two pages appear elsewhere (without a citation) in the IMA application to support of IMA’s projections for student achievement (I also covered this in an earlier post here). Here is that passage:
For the last thirty years a public Montessori Charter in East Dallas, where the area has a 50% high school drop out rate, has boasted third grade student math and reading scores in the top 36% nationwide, graduation rates of 94% of high schoolers, with 88% going on to college. The school has a higher than district-average percentage of ELL students and economically disadvantaged students, and a per-pupil expenditure 14% less than the district level. IMA believes in the power of the method to produce similar results for Madison. For these reasons, our academic achievement goals are ambitious.
Lumin began 38 years ago as a small private (Montessori) school; in 1999 they opened a pre-K to grade 3 charter school. There was no Charter School 30 years ago. They also operate two pre-natal to age three programs, one of which now has toddler and primary classrooms, and is slated to add elementary (as far as I can tell these are private, not charter or public).
Many more problems here. The first is that there is no source given for this data, no footnote, although it appears all of the figures are from the same 2010 newsletter cited above.. Next is the comparison of drop out rates with graduation rates, without defining either. There are many ways to calculate drop out rates and graduation rates, but only rarely do they lend themselves to simple comparisons between the two. This mistake appears to be the work of the IMA team, because the newsletter states both in terms of “graduation rates.” It isn’t clear from the source which schools or students are included (does this combine the private school with the charter, are any students who attended for any length of time included or only those who attended for a certain number of years), nor how data was obtained and for what percentage of students (neither school goes past 4th grade and it can be difficult to track outcomes for mobile students over time). All these things matter. One thing that distinguishes quality research from promotional materials is that quality research provides these sorts of details.
Whatever the sources or quality of data behind the assertion about graduation rates, the numbers involved appear to be very small. The private school currently has a total pre-K to 3d grade enrollment of 72, with only 9 3d graders. The public charter has 272 students with the vast majority in pre-K and Kindergarten and only 32 in grade 3 (the single tested grade).
The achievement statistics from the newsletter — which form the basis for IMA’s aspirational projections — come in two forms.
One is characterized as being based on “a ten-year study of standardized test scores (1993 – 2003),” with no further information given. Since the Charter school opened in 1999, at most 3 years of this unavailable study could include that school. Again, IMA misrepresents the figures they are citing.
The other statistics are clearly from the state TAKS tests, however there is also some confusion and misrepresentation with these. This graph from the newsletter does not indicate whether the data is from the Charter or the Private school, or a combination of the two:
Further investigation appears to show it is from from the Charter. In 2008-9 there were 20 students from the Charter tested in reading, so the number was very small. Further investigation also reveals how this graph is misleading. Note that the Charter students are all in grade 3, but the comparisons are to the “Sum of all grades tested.” If you only include 3d grade, in Region 10 for 2008-9, the percent meeting standard was 94%, or within 1% of the Charter school. This fits with a 2011 study by NWEA which showed that the TAKS cut scores were generally very low, and that 3d grade had some of the lowest. It is also worth noting that the newsletter says nothing about math scores, probably because only 76% of their students met the standard in 2008-9 (compared to 84% in Region 10). Add to this well-reported findings that “suggests they [the TAKS tests] are virtually useless at measuring the effects of classroom instruction,” and problems present become even greater.
It should be noted that those who “analyzed” the IMA materials for the Board displayed little or no awareness of any these issues with the single newsletter IMA referenced as an “increasing research base” (nor any of the issues raised here with any of the other citations) That’s a problem too.
Passage and Footnote #7
7. Sara Rimm-Kaufman and Lia Sandilos,“Improving Students’ Relationship with Teachers to Provide Essential Supports for Learning” American Psychological Association, (2011).
Discussion
This is another case of a perfectly fine presentation, by respected researchers, published on a respected forum, being presented by IMA as supporting assertions about Montessori, when the article says nothing about Montessori. Nothing.
As far as attribution and “increased achievement” goes, here is what the authors have to say “Solely improving students’ relationships with their teachers will not produce gains in achievement. However, those students who have close, positive and supportive relationships with their teachers will attain higher levels of achievement than those students with more conflict in their relationships.”
It would be fine if the authors of the IMA proposal took Rimm-Kaufman and Sandilos’ work and in a systematic way drew parallels between their findings and Montessori practices. Failing this, it would be acceptable to simply include something in the footnote like “for more on relationships and child-centered environments in general, and outside of Montessori, see:…). To imply that these authors “attributed” “increased achievement” to the “Montessori method,” is dishonest. See also the section on footnotes #2 & #6 above.
One More, Passage and Footnote #27
Note: When I was first drafting this post back in January, this is where I intended to stop (at some point you hit diminishing returns, and it becomes increasingly clear that you have put more work into critiquing the research cited than went into doing the cites in the first place). However, in recent weeks I found one more instance I would like to include.
27 Diamond, A., “The Evidence Base for Improving School Outcomes by Addressing the Whole Child and by Addressing Skills and Attitudes, Not Just Content.” Early Education and Development, 2: 780-793. (2010)
Discussion
On the surface this is simply (yet) another case of IMA placing a citation to a perfectly fine article as supporting evidence for assertions about Montessori that the article does not make (there are no mentions of Montessori in Dr. Diamond’s article). Certainly misleading, and to me dishonest. What makes this one special is that in the IMA proposal it comes with a “hot link,” that doesn’t take you to the article, but rather sends you to this (Walton funded) Public Montessori advocacy organization. That’s a whole new level of misleading.
Conclusion
As Mark Twain wrote, “Let us draw the curtain of charity over the rest.” After the citations examined here, I stopped my systematic researches. With the others I did look at, I found some that were fine, and many that were problematic. Regardless of what the scorecard for the entire proposal might be, the clear failings revealed in those covered here are beyond troubling. If the people behind IMA can’t do better with their proposal, how can we trust them with our children and our money?
AMPS 