I guess Ohio's A-F Report Cards weren't that great after all

When Ohio switched to a new A-F report card system in the 2012-2013 school year, much was made of how wonderful it was.

Here's what the Ohio Department of Education said about them, veritably gushing over the Education Commission of the States' awarding the system its highest marks:

"We released the new A-F report cards with the idea that they would be easier to understand, and provide more information to both parents and educators,” ODE spokesman John Charlton said. “This report kind of validates our efforts so far, as we continue to work on this report card. The real winners are the students in Ohio. They’re going to benefit from the information in that report card, whether it’s their parents using the data, educators using it, or whether the students are looking at it themselves.”

Folks from the education reform community claimed it created a more understandable system (because, apparently, parents couldn't decipher the words "excellent with distinction", "excellent", "effective", "academic watch" and "academic emergency"). Here's how the Fordham Institute described the new report cards:

"The Buckeye State’s new A-F report card is a wonderful opportunity for parents to gain a better appreciation of how their child’s school is doing, and to take action if necessary."

But I was concerned because the new A-F system was even more reliant on test scores than the previous system. And test scores are nearly perfectly correlated with poverty rates, not educational excellence. Here's what I said after the first batch of report cards came out:

"The new Report Card is based largely on standardized tests, which are tremendously influenced by demographics. Under this new system, a building and district's ratings are even more dependent upon their demographics than the prior system, which was pretty well dependent upon demographics as well."

So imagine my surprise when a new House Bill 591 appears and does away with Ohio's much-lauded A-F report card in favor of one less influenced by test scores. I should be excited, right? Not so fast. See, here's the thing.

It would keep the A-F report card for consequential actions -- the very things that make these A-F grades so high stakes, problematic and consequential for schools, districts, parents and kids.

What's "consequential" mean? Oh, whether a district is low performing enough that a charter school can come in. Or kids qualify for private school vouchers. Or whether a charter school closes. These -- and other things -- will all still be determined by a district or school's rating on the hated A-F report card.

And the bill eliminates several transparency reports about how schools spend money and how they educate kids. One of the things Ohio does better than nearly every other state is produce meaningful and copious data points.

Other than that, though ...

Look, I appreciate the effort to undo this misguided report card. As sponsor Mike Duffy, R-Columbus, put it, "nobody likes the current Ohio school district report card."

So why won't he just do a simple bill to get rid of the current system, replace it with a more rational system and go from there? Why would he let this system he admits everyone hates continue holding its dangerous grip on students and schools?

Again, these problems were all pretty predictable from the time the A-F report card was implemented. Yet folks couldn't stop fawning over the brilliance of turning five "grade" levels into the familiar A-F scale from the less familiar Excellent with Distinction, Excellent, Effective, Academic Watch and Academic Emergency system.

Now everyone hates it.

I'm not one to say, "I told you so."

But, well ... you know.

Ohio Urban Districts Perform Better Than Charters

One of the quirks in Ohio's Charter School landscape is the fact that many of the state's highest performing Charter Schools, while physically located in the state's urban cores, take substantial numbers of kids from outside those urban cores. However, the urban cores are always compared with these schools.

A more apt comparison would seem to be between Charters that take all their kids from the urban core vs. urban core traditional buildings. What's amazing is this: Of the 300 Charter Schools graded on the State Report Card (not including the state's 100 or so dropout recovery schools), only 84 took 95% or more of their kids from the state's big 8 urban districts (Akron, Canton, Cincinnati, Cleveland, Columbus, Dayton, Toledo and Youngstown) during the 2012-2013 school year (the most recent data I have). None of those 84 buildings are located in Canton. Of the 248 Charter School buildings located in the counties of the Big 8 urbans, about 1/3 take 95% or more of their kids from the Big 8 urban districts.

Franklin County's a perfect example of this phenomenon. Of the 72 Charter Schools located there, only 12 took 95% or more of their kids from Columbus City Schools in the 2012-2013 school year.

Now, is it fair to compare the performance of Charter Schools that take significant portions of kids from outside the Big 8 with the performance of schools in the Big 8 that take nearly 100% of their kids from the Big 8? No.

So I looked at the performance of Charter Schools that took 95% or more of their kids from the Big 8 with the performance of Big 8 urban buildings, and what I found is this: Ohio's urban buildings (minus Canton because no Charters take 95% or more of their kids from there) outperform their Charter School counterparts, even though Charter Schools remove as much as $1,000 per child in state money (depending on the district) from every kid in a Big 8 urban building.

Big 8 urbans outperform their Charter School counterparts on As and Bs and get smaller percentages of Ds and Fs. But perhaps the greatest disparity is in Performance Index Score -- the closest thing Ohio has to an overall school grade at this point.

The average urban building PI score was 79.05. The average Charter PI score was 63.45. To give you a sense of scale on what a 15.6-point difference in PI scores means, Wyoming City received the highest PI score of any district last year -- a 113.013. There are 429 (of 613) school districts between Wyoming and the district that is a bigger than 15.6-point drop.

In other words, a 15.6-point difference is enormous.

Again, this is as close an apples-to-apples comparison as can be done. Where do Ohio's urban kids do better, in district buildings or Charters? The answer is pretty clear -- district buildings.

It should also be made clear that urban district performance isn't any great shakes (more than 40% of their scores are Fs and barely more than 20% are As and Bs, after all), but even that performance is better than Charters, and in the case of Performance Index, much better.

Can we have a discussion of quality charters? Please? Our kids need it.

Charters were supposed to improve performance. Instead, they're performing worse, even in the communities where they were supposed to have the most profound effects. Sixteen years and $7.4 billion into this experiment, shouldn't we be getting better results?

Report Card: Charter Value Added Grades Not Much Better

One of the only areas where Charter Schools have been able to make an argument for their success has been on value-added measures (VAM). VAM is, essentially, a growth measure. It measures actual student academic growth versus expected student growth. I've always had questions about how the expected growth is calculated, and Ohio's VAM formula is notoriously proprietary (so it can't be re-produced). But that's the idea.

I'm more intrigued by VAM at the macro, district level because its calculation breaks down as you drill into classrooms, and especially teachers. For example, there have been many incidents around the country where a VAM determined a teacher was the worst teacher in the district one year, then the best the next year -- and vice versa. But at the district level, VAM holds more promise, is less swayed by demographics than raw test scores, and is better philosophically. Though it still needs a lot of work.

Anyway, I broke down Ohio's report card, which was released yesterday, on the VAM categories the state measures. Then compared Charter-District performance. Specifically, the grades for overall value added, lowest 20% (how well do the lowest performers grow), and disabled all receive VAM scores. The state also does gifted VAM, but only a handful of Charters have enough gifted kids to qualify. So the comparison is pretty meaningless.

The results are below. As you can see, Charters do a little bit better than their raw scores would indicate. But it's still nothing to write home about.

Districts still get higher percentages of As and Bs on all the value added categories. Meanwhile, Charters get higher percentages of Ds and Fs than districts do. So, on overall value added for example, a higher percentage of districts get As than Charters get As and Bs, even though the highest percentage of Charter As are in that category.

And even when Charters do get a higher percentage of As, as they do in the lowest 20% category, districts so outperform them on Bs that a higher percentage of districts still get As and Bs in that category. Meanwhile, Charters get higher percentages of Ds and Fs, even in this category.

Remember that every Ohio school district lost money and children to Charter Schools last year (only Ohio's tiny Lake Erie island districts did not). Even Charters in the urban core receive a significant number of kids from outside that urban core. The most famous example, perhaps, is the Columbus Preparatory Academy -- run by the for-profit Mosaica Education, Inc., which has among the highest performance index scores in the state. Yet about half the kids don't come from Columbus City Schools. So is it fair to judge Columbus based on this school's performance? Yet Columbus always is.

Overall, we know that about half of kids in Ohio Charter Schools do not come from the state's urban core districts -- the original site of Ohio's Charter School experiment. We spent $914 million on Charters last school year. And about all we can say positive is that in one value added category, they got 1% more As than districts. But we can also say that they fail at a significantly higher level in all these categories than the districts from which they receive their children and money.

And in perhaps the most bottom line measure there is -- graduation rates -- the difference is stunning:

What these value-added data demonstrate for me is this: Ohio's Charter Schools perform marginally better overall on them than more traditional measures. But the question I ask is this: does this marginal improvement justify kids in Columbus losing $1,063 every year because the Charter School deduction is so huge? Or kids in Boardman losing $1 million? Or kids in Brooklyn losing more than 60% of their state revenue?

Yes, there are a small handful of Charter Schools that are doing the innovative things Charter Schools were originally intended to do. These tiny pockets of success could be achieved with far less damage to the remaining 90% of kids in Ohio's school districts, where, overall, they attend higher performing options. Instead, the 90% of kids in those districts lose about 7% of their state revenue because Ohio's General Assembly pays more than double the state money to Charter Schools as they do to the child's district of residence and have cared little about the performance of these schools.

Charter School proponents over the next few weeks will probably be able to drill down the VAM enough to show that schools in some area outperform Districts in an area on a measure or two. But should it be this hard to show Charter School success after $7.4 billion spent since 1998? It's not that hard in other states. But it is here in Ohio. The Stanford CREDO study found that Ohio's Charters are one of only 4 states to see their performance slip between 2009 and 2013, while the average Ohio Charter student loses a full marking period in math and 1/3 of one in reading to their public school counterparts.

Given the amount the state has poured into Charter Schools, which is now more than the state spends in a year on kids in school districts through the state's funding formula, you would think the evidence would be overwhelming that Charter Schools are superior. But instead, we have to spend weeks with algorithms and sophisticated statistical tests to find some permutation that shows Charters may be slightly better at one tiny thing the state measures. On all the big, obvious measures, Ohio's Charter Schools just don't cut it overall.

Charter School quality, not quantity must fuel this debate from now on. Whether a school succeeds should be paramount. Not whether it simply exists. We're beyond the point of asking whether we should have school choice. Fifteen years and $7.4 billion in, it's safe to say that choice is here to stay. Now we need to ask a very simple question: "What should those choice options look like?" and, more importantly, "Should they be any good?"

Ohio Report Card: Urban Schools Face Bigger Challenges, Outperform Charters

In further analyzing the Ohio Report Card data released today, schools in Ohio's Big 8 urban centers (Akron, Canton, Cincinnati, Cleveland, Columbus, Dayton, Toledo and Youngstown) scored higher on their performance index score (the closest thing Ohio has to an overall performance assessment at this point) than Charter Schools, despite having substantially higher percentages of children who were economically disadvantaged. A staggering 51% of Big 8 urban buildings have more than 95% of their students designated as economically disadvantaged (the Ohio Department of Education only says buildings have ">95.0" if their economic disadvantaged number is higher than 95%).

So, despite having more than half their buildings with, for all intents and purposes, all their kids economically disadvantaged, Ohio's Big 8 urban buildings actually perform better, on average, than Ohio's Charter Schools, which were originally intended to "save" children from "failing" urban buildings.

I'm not saying I'm happy with the Big 8 results. What I am saying is the Big 8, despite their struggles, are doing better with more challenges than Ohio's Charter Schools. $914 million should buy us taxpayers more, I think.

Building Type	% Economically Disadvantaged Per Building	% of Schools >95% Economically Disadvantaged	Average Performance Index Score
Big 8	86.5%	51.0%	79.08
Charter	81.9%	38.2%	78.058

Ohio Report Card: Ohio Charters Get More Fs than As, Bs, and Cs COMBINED

The Ohio Report Cards are now all out, and the news is worse for Ohio's embattled Charter Schools than it was last year. Charter Schools received more Fs than As, Bs and Cs combined. Their percentage of Fs went up from about 41% last year to nearly 44% this year.

Meanwhile, Ohio's public school districts saw their As jump to the largest percentage and number of grades. This chart shows the breakdown:

As can be seen, Ohio's Charter Schools continue to struggle with their performance, with now nearly 62% of their report card grades registering D or F. Remember that Ohio school districts lost $914 million to these schools last school year.

Will Ohio's policy leaders take note?

New Report Card: State of State School Overrated

Apparently, Gov. John Kasich's choice of State of the State venue was overrated.

Kasich caused a tizzy when he chose to move the annual speech to Wells Academy in Steubenville, citing how it is doing great things despite budget challenges -- not so subtly suggesting that money doesn't matter as much to academic performance as commitment, vision and innovative adaptability to tough budget times.

However, Ohio's proposed waiver from No Child Left Behind contains a new Report Card system that State Superintendent Stan Heffner claims will give Ohioans a clearer indication of its schools' performance. And under that new system, Wells Academy goes from an A on the report card to a B.

Is this evaluation really more accurate? Or is it the result of a ham-handed evaluation tool that hurts schools like Wells Academy, which overcome demographic challenges to be considered great enough to host an important gubernatorial address?

The new Report Card is based largely on standardized tests, which are tremendously influenced by demographics. Under this new system, a building and district's ratings are even more dependant upon their demographics than the prior system, which was pretty well dependant upon demographics as well.

Note: According to an Excel regression analysis of ODE data on the new system at the district level, demographics (poverty, income, property valuation, teacher salaries, educational attainment levels, etc.) produce an R-squared value of .48 for the new system vs. an R-squared of .45 for the previous system. The closer to 1 (or -1), the stronger the correlation.

This could explain why Wells Academy now rates a B rather than an A because its demographics are not favorable. So if the evaluation system's more dependant on them, Wells will seem less successful under that evaluation. But is Wells, in fact, less successful than the Governor and nearly every other education observer in this state thought? And if the new system made a mistake on Wells, what about the other districts and buildings?

For the issue isn't just at Wells Academy. Of the 3,409 school buildings rated under the old system, more than 77 percent rate worse under the proposed system, according to ODE projections. And that's assuming that the Excellent with Distinction buildings under the old 5-point Report Card, which equates to an A+, would rate the same under the new 4-point system the department's assuming (which doesn't include A+, just an A). So it's probably an even higher percentage.

Only 40 buildings improve, which means that barely 1 percent of buildings were underrated by the old system. Meanwhile, more than three-quarters of buildings were overrated. Can that even be possible?

Meanwhile, more than 83 percent of school districts were overrated, while none, that's right, not a single Ohio school district was underrated by the previous system.

I hope folks ask a simple question: "Was the old system that off?"

Roosevelt Elementary in Springfield (right in my backyard here near Akron) was an "Effective" building under the old system, meaning it rated a B. Under the new system, it's an F. Three ISUS Charter Schools in Dayton were rated Excellent under the old system, an A. Under the new, they all get Ds.

Meanwhile, the only schools that actually improve under the new system are 40 schools that improve from Academic Emergency under the old system -- an F, to a D under the new system. No building improved more than one step. And no building rated above an F in the old system improves under the new system.

One would think if you were creating a more accurate system, there would be corrections in both directions, certainly not all in one direction.

Charter Schools' rating changes are interesting. While 83 percent of school districts saw their grade levels drop, only 55 percent of Charters saw them drop (perhaps because more of them rated poorly under the old system and had less room to drop). Meanwhile, 45 percent of Charters stayed the same or improved under the new Report Card, though improvement was relegated to previously failing charters.

One side effect of this is that fewer Charters would be up for closing under the new system, assuming the same standards that applied under the old report card are transferred to the new one.

The performance differences remain stark between Charters and Districts. Only 10 percent of Charters rate B or higher on the new system (nearly 3 in 4 rate D or F), with some of the better thought of Charters slipping from As or A+s under the old system to Bs in the current one, like Wells Academy did on the Traditional side.

Meanwhile, two-thirds of school districts rate B or better on the new report card, with about 10 percent rating D or F.

Despite these clear questions about the new Report Card's methodology, all I really care about is this question: Now what?

What's the state's plan to improve these schools, since the Ohio Constitution and State Supreme Court have found education to be a state responsibility in Ohio? Will cutting more money from the state budget help districts be more innovative? Can the improvement happen with the same amount the state's spending, just with better, more focused programming? Will local taxpayers have to tax themselves at higher rates so districts have the necessary resources to meet the tougher standards certain to come down from the state? Will districts be able to pass levies now when they are considered B and C districts rather than A and B districts?

These are just some of the many questions the state and districts now face.

Again, I would like to see a system that rates districts not so much on their proficiency rates, which are so heavily influence by demographics, but upon their relative success in overcoming those barriers. So, for example, a Performance Index score of 82 may be phenomenal in some districts, but in the wealthier ones, that would be a terrible score. So the district where 82 is great should have that score weighted to account for their greater challenges.

In other words, Wells Academy should take its rightful place as a source of pride for the community and state, not relegated to the above average. It's difficult to understand how one of the best schools in the state is now merely one of the good ones.