Getting Behind the Veil of Effective Schools

Will Dobbie and Roland Fryer studied New York's Charter Schools.


They quote a number of authors (Rouse 1998, Ladd 2002, Krueger and Zhu 2004, Cullen, Jacob, Levitt 2005, 2006, Hastings, Kane, and Staiger 2006, Wolf et al. 2010, Belfield and Levin 2002, Hsieh and Urquiola 2006, Card, Dooley, and Payne 2010, Winters forthcoming) to suggest that:

• "Competition alone is unlikely to significantly increase the efficiency of the public school system." (p1)
• In school strategies that "are not correlated with school effectiveness" (p1) include:
• Class size
• Per pupil expenditure
• Teacher qualification
• Rigorous and complex lesson plans (this measure includes differentiated lesson plans and lesson plans aimed at a higher level os Bloom's taxonomy)

"In stark contrast, we show that an index of five policies suggested by over forty years of qualitative research – frequent teacher feedback, the use of data to guide instruction, high-dosage tutoring, increased instructional time, and high expectations – explains approximately 50 percent of the variation in school effectiveness." (p2)


I have several methodological concerns:

• The measure of effectiveness is based on Maths and 'ELA' (English Language Arts) tests. This is because these are standardised across the state so it is reasonable to use these measures. However, the measures may not be valid if you want to consider the education of the whole child including, for example, PE, Art, Music etc.
• Most of the measures they use are turned into binary measures with seemingly arbitrary watersheds. Thus they claim that giving frequent feedback to teachers based on lesson observations works; by frequent they mean 10 or more times per semester. Similarly, 'high dosage tutoring' means tutoring 4 or more times per week in groups that have "typically" 6 or fewer students. One wonders whether they chose the watersheds that gave them the results.
• They use three significance levels: 10%, 5% and 1%. In classic statistical based research only the 5% and 1% levels are regarded as significant. Conventional researchers would say that the 'frequent feedback to teachers' strategy is therefore not proven as an effective technique.
• Having found that some of their strategies are not significantly effective they then package all five measures into an index so it becomes statistically significant as a whole. One would have thought it better to discard the individual strategies that are not effective and to concentrate one's index on those measures that are individually significant. But then why have a package? Why not just say schools should do this, this and this? One suspects that they use the index to protect their non-significant sacred cows.

But let me put aside my suspicions. What do they say works?

• Using lesson observations to give frequent feedback to teachers. Frequent feedback means 10 or more times per semester; this is statistically significant only at the 0.1 level.
• "The typical teacher at a high achieving elementary school receives feedback [from observed lessons]16.41 times per semester, compared to 11.31 times at other charter schools. The typical teacher at a high achieving middle school receives feedback 13.42 times per semester, 6.35 more instances of feedback than teachers at other charter schools." (p8)
• Extra time also works. 
• "High achieving elementary schools provide about 26.68 percent more instructional hours per year than a typical NYC schools, while high achieving middle schools provide about 28.07 percent more" (p9)
• "Teachers at high achieving schools also work longer hours than teachers at other charter schools; an additional 7.75 hours per week at the elementary level and 10.29 hours per week at the middle school level. Despite this higher workload, the maximum salary of teachers at high achieving schools is the same or somewhat lower than other charter schools." (p8)
• Testing and tracking also works.
• "We attempt to understand how schools use data through the frequency of interim assessments,whether teachers meet with a school leader to discuss student data, how often teachers receive reports on student results, and how often data from interim assessments are used to adjust tutoring groups, assign remediation, modify instruction, or create individualized student goals." (p8)
• "High achieving schools use data more intensely than other charter schools in our sample. High achieving elementary schools test students 3.92 times per semester, compared to 2.42 times at other charter schools. Higher achieving middle schools test students 4.00 times, compared to 2.04 times at other charter middle schools in our sample. Higher achieving schools are also more likely to track students using data and utilize more differentiation strategies compared to low achieving schools." (p8)
• Reporting to parents works.
• "Parent outreach variables capture how often schools communicate with parents due to academic performance, due to behavioral issues, or to simply provide feedback. Summary statistics in Table 2 suggest that high achieving elementary and middle schools provide more feedback of all types to parents. Higher achieving schools provide academic feedback 3.00 more times per semester than other schools, behavioral feedback 9.20 more times per semester, and general feedback to parents 7.27 more times per semester." (p8)
• High dosage tutoring (groups of 6 or fewer students meeting 4 or more times per week) works. 
• "Thirty-three percent of high achieving elementary schools offer high-dosage tutoring  compared to ten percent of low achieving schools." (p9)

In effect they conclude that the controversial 'No excuses' schools work.
"'No Excuses' schools emphasize frequent testing, dramatically increased instructional time, parental pledges of involvement, aggressive human capital strategies, a 'broken windows' theory of discipline, and a relentless focus on math and reading achievement" (p2)






Reference


Dobbie W and Fryer R 2011 Getting Behind the Veil of Effective Schools: Evidence from New York Ciry available at http://www.economics.harvard.edu/faculty/fryer/files/effective_schools.pdf accessed 6th January 2012