Kevin Drum commented on a charter school study a couple of days ago, which made me go look through the report (available from this ultra-minimalist page— seriously, you can’t even be bothered to cut and paste some of your introductory boilerplate into an HTML file to give people an idea of what’s behind those PDF links?).
The summary message is kind of bleak. From Drum’s post, quoting the LA Times:
The study of charter schools in 15 states and the District of Columbia found that, nationally, only 17% of charter schools do better academically than their traditional counterparts, and more than a third “deliver learning results that are significantly worse than their student[s] would have realized had they remained in traditional public schools.”
Looking closer, there are a couple of weird things in the executive summary, most notably this:
Charter schools have different impacts on students based on their family backgrounds.
For Blacks and Hispanics, their learning gains are significantly worse than that of their
traditional school twins. However, charter schools are found to have better academic
growth results for students in poverty.
That seems like a really odd combination of results. Given that Blacks and Hispanics are disproportionately likely to be poor, this would seem to imply that middle-class Black and Hispanic students are absolutely crushed by charter schools, or that poor whites and Asians kick butt in charter schools. Or some linear combination of the two.
Of course, either of those seems pretty damn strange. So what’s really going on? Looking at the full report, my inclination is to say that this is one of those cases where averaging results together hurts more than it helps.
If you look at Tables 5 and 6, giving the lists of state-level effects for Black and Hispanic students, respectively, you’ll see that the results are kind of all over the map. Georgia and Texas are bad across the board, Missouri is good across the board, and the rest of the states jump around. Florida is bad for Blacks, Illinois, New Mexico, and Ohio are bad for Hispanics. Louisiana and Minnesota are good for Blacks, while Arkansas is good for math but indifferent for reading. California charter schools manage to have a statistically significant positive effect on reading scores for Blacks, but a statistically significant negative effect on math scores for Hispanics.
Turning to Table 7, giving the same information for poor students, we find that Missouri and Louisiana, whose charter schools did well by minorities, do a lousy job with poor kids. Meanwhile, Georgia and Texas, which were awful for minorities, are good for poor kids.
I think the technical term for this is “a hopeless muddle.”
I suspect that if this were broken down further (which it probably is in the state-by-state reports, but there are limits to how many PDF’s I’m willing to wade through), the school-by-school results would be as erratic as the state-by-state results. Which suggests that trying to report nationwide demographic trends is kind of foolish– I mean, sure, you can average everything together and get some sort of composite result, but it’s not clear that it means anything. The benefits or costs of charter schools appear to be highly contingent on local factors, which is not that surprising, as education is a complicated business.
Really, all that I’d be willing to say, looking at the report, is “Answer unclear, try again later.” There’s no clear, large positive benefit to charter schools, and there’s no clear, large negative effect, either. The averaging process leaves you with a smattering of results that appear statistically significant, but taken together, they don’t form any kind of coherent picture. Which means they’re probably just noise.
But, if you’re going to write a report, you need more than “It’s complicated…” So you get demographic breakdowns reported, whether they make sense or not.