NEW Cleveland Transformation Plan Report: Slow, Non-Uniform Progress, but Many Challenges Ahead

Today, we at Innovation Ohio released a comprehensive look at the so-called "Cleveland Plan". The initial look we took at the plan in 2012 led to a report that caused some much needed changes to the legislation. Our newest follow up was meant to re-examine the plan, how it's working, how it needs improved and, ultimately, some ways to fix it.

Complicating this whole thing is the specter of a teacher strike. The tensions around the union negotiations will, inevitably, color how the sides view our report. But I want to make something clear: This report was done to examine how the plan is working so that the people of Cleveland understand how important it is to pass the district's levy renewal this November. One of the major flaws with the Plan was the state never put additional investment into its components' success. So the local community has had to step up to a much greater degree than it should have had to. And it will still need to do that.

It is a credit to the people, businesses and philanthropies in Cleveland that they have stepped up for Cleveland's kids thus far. And while the labor issues pose a real threat to the Plan's future, the potential for the Plan to usher in a new era of better, more accurate teacher evaluation and pay structures should not be understated. There is great hope here, but the district and its teachers need to work this out. Quick. Because there's little question that if the levy fails in November, even the modest improvements the district has seen will quickly fall away.

We went out of our way to not use district or union data in the report (and we got plenty of it), sticking rather the state data so that we couldn't be accused of siding one way or another. While I certainly spoke with and interviewed both sides, I chose to use state-reported (and national) data to do the analysis. There will be things in the report both sides will like and not like. But I did my best to, on the data, stick with a neutral party. Though I know some will suggest otherwise. That comes with the territory.

Here's our summary of the report:

"For the first time in decades enrollment in Cleveland public schools has increased. Graduation rates have also increased and proficiency test scores have improved relative to other large urban school districts.

These are a few of the positive developments in the Cleveland schools, but many challenges still remain. Fourth grade reading and math scores have seen a slight uptick, but are still very low nationally. Excessive state funding cuts and prolonged tension between the districts' teachers and administrators have made the path forward difficult. 

Our latest report examines these successes and challenges of the Cleveland Plan for Transforming Schools."

Testing the Boundaries: Part II - What Can Outliers Teach Us?

We've spent 12 years obsessing over two things: tests and achievement gaps. Yet it appeared that the efforts have been for naught, as the achievement gap has continued to grow.

However, there is some good news. The state's poorest districts saw the biggest percentage improvements in their test scores over those 12 years. The bad news is that improvement, while slightly narrowing the raw score performance gap between rich and poor districts, widened the state's relative performance gap. Meaning, it wasn't big enough to make a difference, relatively speaking, because high performing districts also significantly improved their scores.

You can see the problem in the following charts. Pay attention to what happens between the Suburban, very low student poverty and Urban, very high student poverty categories. You'll see impressive improvement among the poor category, which includes Ohio's so-called "Big 8" urban districts -- Akron, Canton, Cincinnati, Cleveland, Columbus, Dayton, Toledo and Youngstown. These are the districts state lawmakers have targeted for years as "failing" kids. Though, looking at the raw score improvement, one can't be anything but impressed by the 12-year jump.

Here's the problem: All districts improved, even the wealthiest districts. So despite the state's poorest districts' impressive improvement, all the other districts' improvement was enough to widen the relative performance gap so that the wealthiest districts made up a significantly higher proportion of top 10% mini-PI scores in 2014-2015, and the poorest districts made up a significantly higher proportion of the bottom 10% mini-PI scores.

But it cannot be ignored that Ohio's maligned, urban districts improved their math scores by more than 21%, and the improvement among all districts with high or very high poverty saw scores jump by 20% or more. That's impressive. But it wasn't enough to improve their status among Ohio's districts. And, in fact, they dropped in comparison with their wealthier brethren.

So what gives? Why did a 20% improvement lead to the widening of the achievement gap? A major reason is the upper limit of the mini-PI score. The upper limit of the PI score is 120. That's the maximum score any district could receive. And while some districts get close, getting a perfect 120 is probably impossible. So in the 2003-2004 school year, the average PI score for the OGT math score in the state's wealthiest, suburban districts was 102.84 -- a mere 18.16 points from perfection. So the greatest percentage improvement they could have was a bit more than 17%. The average for the state's poorest, urban districts was 68.26 -- 51.84 points from perfection. So the greatest percentage improvement they could have was 76%. 

In the intervening years, the wealthiest districts went up to an average of 112.25 -- an amazingly close 7.75 points from perfection, or more than 1/2 of their maximum percentage improvement. And the state's urban districts went to 82.26 -- less than 1/3 of their maximum percentage improvement. 

So what this meant is more suburban districts performed at an elite level, while the state's urban districts, which performed much better, remained mired in the bottom 10% of mini-PI Scores.

But this doesn't mean there aren't outliers. Take Lakota in Sandusky County -- classified by the state as a rural, poor district. In 2004, their OGT Math score placed them 511 of 608 districts. Not good. But in 2015, their score jumped so much that they are now in the top 1/3 of districts. Or Maplewood in Trumbull County. In 2004, they were in the bottom 1/2 of Ohio districts on OGT math scores. Now, they're in the top 5%.

In fact, the greatest rank jumpers in the state were the rural districts. They made up almost 1/2 of the greatest rank improvements. 

Here's the other problem: They also made up about 40% of the greatest rank fallers too.

Tellingly, no urban district was in the top 10% of climbers or fallers.

Again, is there something interesting going on in the districts with the greatest improvements? Or something awful happening in the fallers? Or is it just statistical noise -- a true outlier? One would have to go to the district and see what, if any, change has led to the relative improvement. But given the predictive force of poverty on these scores, normal statistical variation would seem to explain much of this difference.

What does all this mean from a policy perspective? Well, it means that if improvement or performance slippage can be explained in large part by statistical variation, should we be granting kudos or shame to districts that grow or slip? It'd be easy to devise a method to reward districts that show dramatic improvement on scores, even if their relative performance to other districts remains low. Likewise, it can be easy to punish districts that slip. But should we, given what we know about statistical variation in these cases? There will always be outliers -- districts that out- or under-perform their demographics. But if there's one thing variation means it's that over or under performance could, in fact, be nothing more than yet another expected statistical result of standardized testing. Every district has their strong or weak classes pass through the system.

In other words, a district's score may not be indicative of the actual quality of their educational program. Yet, under our current accountability system, it is the dominant determinant of a school or district's quality.

But that doesn't mean these score changes mean nothing. Figuring out what they mean is a great challenge, but one that must be unlocked in order to understand how, if at all, they should inform our nation's education policy.

Tuesday
Testing the Boundaries
Part III - Finding Excellence through the Noise

Last Week Tonight Takes on Ohio Charters. I Have Flashbacks.

On John Oliver's Last Week Tonight show last night, the host spent a good 20 minutes talking about charter schools, and, of course, singled out Ohio's sad, sordid history on the subject -- a history I am proud to say is changing in no small measure due to the work at Know Your Charter and pro-quality charter advocates.

So when I saw the stories of double dealing at Richard Allen academies, or the repeated references to CREDO studies singling out Ohio's horrible past, or the many other scandals at Ohio charters, I suffered flashbacks I didn't enjoy. There was even a trip to the way back machine from 2000 showing the infamous David Brennan declaring that schools are a business and if they're run like a business, they'll run properly ... unless Brennan runs the schools, given that his routinely perform the worst of any big Ohio charter chain.

But I digress.

I was surprised that Oliver didn't get into the fact that Brennan and William Lager, who runs the infamous Electronic Classroom of Tomorrow -- the nation's largest, for-profit K-12 school that doesn't even graduate 4 out of 10 kids, have given more than $6 million to mostly Republican politicians since the charter program started in the late 1990s. And schools run by them have collected one out of every 4 charter dollars ever spent in this state.

While I welcome the continued scrutiny on Ohio's charter past, I would hope that Oliver returns to Ohio over the next year or so to examine how even the "Wild, Wild West" of charter schools can be tamed. But, as I've said before, we still have a ways to go before Oliver could do that. But I'm hopeful that for the first time in years, there's a realistic shot he may be forced to do just that.

Someday.

Testing the Boundaries: A Series on Ohio (and the Nation's) Achievement Gap. Part I: After 12 years, Ohio's Performance Disparity Between Rich and Poor Districts has Grown Worse. Now What?

One the worst traps we can fall in as analysts is to ignore the big picture. Lately, I've been guilty of this -- dealing with short term, juicy topics while eschewing the forest in front of my face.

So I decided to look at 12 years of Ohio Graduation Test data and see how different districts fared each year. The OGT results are listed separately on the Ohio Department of Education website.

The results blew me away; after more than a decade of test-focused reform, Ohio's achievement gap between its wealthiest and poorest districts has gotten worse, not better. What now? Well, I'm going to do my part: namely a blog series on exactly that: How do we determine academic success and why aren't we closing the gap?

The data that inspired this series comes from the OGT data put out by the Ohio Department of Education. I chose OGT data because that's been the least volatile state-administered test. ODE lists the percentage of students who scored advanced, accelerated, proficient, basic and limited for each year's testing. Then, using the state's Performance Index formula (I didn't include the new "Advanced Plus" category for the calculus. Again, for all you nerds out there.), I was able to crunch those four categories into a single, mini-Performance Index (PI) Score so I could more easily see how districts were improving.

I then looked at the districts' improvement on the raw mini-PI score and how they ranked each year among the 608 districts that could be compared each year. I didn't include the island districts or College Corner.

Then I looked at their typeology. While the typeology numbers and definitions have changed slightly over the years, the typeology tells you the kind of district based on community make up and poverty. Here's the most current typeology chart:

The typeology make up is interesting in and of itself. For example, you can see that about 2/3 of Ohio's school districts are in small towns or rural communities. Yet 2/3 of Ohio's school kids are in suburban and urban districts.

This explains Ohio's struggle with school funding to a great extent. Because ways you can make a formula work for rural districts will likely hurt suburban and urban areas, where most of the kids are.

But I digress.

Again, I used the typeology chart to determine which typeologies tended to score better than others in each year. They I looked to see how they improved (or didn't) between the 03-04 school year and the 14-15 school year.

The results aren't really surprising. The wealthiest categories (3,5 and 6) rated the best. The poorest (categories 1, 4, 7, and 8) did the worst. What is surprising is this: The achievement gap between the rich and poor districts is growing more pronounced after a dozen years of test obsession.

For example, on the 2003-2004 Math OGT, category 6 (very wealthy, suburban districts like Ottawa Hills) made up 46.7% of the top 10% mini-PI scores. In 2014-2015, that had jumped about 10 percentage points to 56.3%. Meanwhile, the poorest urban districts (category 7, which includes districts like Euclid and category 8) made up 38.3% of the bottom 10% scoring districts, and 6 of the 8 Big 8 urbans (category 8, which is Akron, Canton, Cincinnati, Cleveland, Columbus, Dayton, Toledo and Youngstown) scored in the bottom 10%. On the 2014-2015 OGT Math, all the Big 8 urban districts score in the bottom 10% and 56.6% of the bottom 10% of scores come from the state's poorest urban districts.

The same general breakdown and change has occurred on reading scores, though not quite as dramatically. But the disparity still has grown significantly between the wealthiest and poorest districts.

So even though urban districts only make up about 9% of all districts in the state, they make up nearly 60% of the 60 lowest scoring districts in the state. Likewise, the state's wealthiest suburban districts (category 6) account for 7.6% of all Ohio districts, but make up about the same 60% of the 60 highest performing districts.

It is equally telling that in neither 2003-2004, nor in 2014-2015 did a single urban district score in the top 10% on either OGT category. And only 1 wealthy suburban district scored in the bottom 10% of either test in either year (Gahanna on 2003-2004 Reading).

What does all this mean? Well, it appears that, generally speaking, 12 years of test-focused accountability has grown the achievement gap between the state's wealthiest and poorest districts, not shrunk it. But I want to ask a different question: Can this disparity ever shrink?

We've known for years about the powerful connection between test scores and poverty. And we've tried to mitigate the problem by using value-added scores, or some other statistical pretzel, but the fact remains that the data produced by test scores has as much (if not more) to do with poverty than classroom performance. 

This calls into question our whole test-based accountability scheme. For example, if no Big 8 Urban district scored outside the bottom 10% on this set of OGT scores, is it a failure of the Big 8, or merely a confirmation that the tax and census data showing the extreme poverty in these communities is accurate? And if that is so, is it fair to hold these districts, buildings, teachers and even communities liable for the district's performance?

And if these scores are measuring poverty rather than quality, should we be opening the doors to more and often poorer performing choice options in these districts?

And if test scores aren't cutting the mustard, what can? And at what cost?

These are all questions I'll be exploring over the next several days as I dig into this series. But I think it's important to recognize that the state and nation's poverty achievement gap, if we keep measuring it through tests, may never close. We may only get a true assessment of our nation's education system when we stop using subject-based standardized tests to measure achievement.

Monday: Testing the Boundaries
Part II - What Can Outliers Teach Us?

Donald Trump Acts Like the Last 30 years of Ed Reform Never Happened

I've deliberately tried to stay out of the wild and wacky 2016 presidential race here. Primarily because education policy has been largely buried underneath the pile of mess that has become the daily 2016 campaign grind. However, Education Week put out an interesting blow-by-blow of Donald Trump's most recent comments about education. The conclusion?

He's big on school choice.

In fact, he spoke in Milwaukee this week and clamored for more school choice -- in the city where school vouchers began. Milwaukee has had choice for two generations of students. And Trump didn't see any irony when he decried the struggles of Milwaukee's schools, despite two generations of choice.

But what really struck me was when Trump told the Wisconsin audience:

"On education, it is time to have school choice, merit pay for teachers, and to end the tenure policies that hurt good teachers and reward bad teachers. We are going to put students and parents first."

It's time we have school choice?

Huh?

We have had school choice in America since the late 1980s -- 1968 if you include the magnet school movement. We have a $37 billion charter school industry that makes its money off school choice. Charters are in 43 states and D.C. We even have an entire major urban school district -- New Orleans -- being nothing but a choice/charter district. We've had school choice so long in America that we're able to study the long-term economic impact they have on kids and communities -- which has been primarily a negative one. And the returns on charter school performance are not good, especially in Ohio.

What is Trump talking about?

And merit pay for teachers? We have that in several districts right now. And it's not working out so great, especially when the issue goes to voters. When Ohio Republicans passed Senate Bill 5 in 2011, teacher merit pay was the core of the issue. Ohioans voted against merit pay by nearly a 2-to-1 margin. The early returns on merit pay's successful improvement of student outcomes is also not good. And even the free market bastions at the Harvard Business Review say merit pay doesn't work, no matter the system, even in the private sector.

And eliminating teacher tenure? Again, not a new idea. Once again, there's little to no evidence that ending tenure improves student achievement. As the Brookings Institute pointed out in a 2014 study, the years prior to teachers getting tenure (usually 4-5 years depending on the state) tend to weed out the vast majority of struggling teachers. The researchers found that "in the fifth year of their careers, only 27 percent of the bottom quarter of teachers, in terms of their value-added, remained employed at the school where they began" -- the lowest retention rate of any teacher performance tier. We can debate the effectiveness of judging effective teachers by student test scores, but the Brookings study demonstrates what teachers already knew -- really struggling teachers tend to leave before they ever get tenure.

While I'm not shocked that Trump has taken these positions, I am shocked he's speaking like they're new, or we don't have them. He also said in Detroit that Detroit needed school choice. Detroit has abundant school choice. Yet Trump acts like there isn't any?

What's clear to me is that Trump probably doesn't know what he's talking about, but even if he does, the ideas he's articulated for education are nothing new, exciting or especially innovative.

National Bureau of Economic Research: Charter Schools Decrease Future Students' Earnings

In a devastating new paper by the National Bureau of Economic Research, the authors find that charter schools overall have a negative impact on students' test scores and future earnings. That means if you send your child to a charter, the data shows that decision will likely dim your child's economic future. And even the higher performing charters generally don't improve students' future earnings.

Not good news for the nation's charter school sector, which also was hit recently by the NAACP coming out against for-profit operators running these schools, while calling for a moratorium on new ones.

Here are the NBER authors' findings:

"We find that, at the mean, charter schools have no impact on test scores and a negative impact on earnings. No Excuses charter schools increase test scores and four-year college enrollment, but have a small and statistically insignificant impact on earnings, while regular charter schools decrease test scores, four-year college enrollment, and earnings. Using school-level estimates, we find that charter schools that decrease test scores also tend to decrease earnings, while charter schools that increase test scores have no discernible impact on earnings. In contrast, high school graduation effects are predictive of earnings effects for both low- and high-value added schools."

Worse, they conclude that it appears that in high-performing charter schools, the schools have focused so much on improving math and reading scores that they've de-emphasized arts, foreign languages and other "soft skills", which ends up hurting students in the long run because students need these skills to achieve in the market place. The authors theorize that this could be why high-performing charters do improve scores, but have zero impact on earnings.

"Much more troubling, it seems, is the possibility that what it takes to increase achievement among the poor in charter schools deprives them of other skills that are important for labor markets."

Far from being an attack on charters, the paper is a sober, scientific look at what many of us are seeking: a determination of whether education actually improves kids' lives in the long term. And it also helps bring greater clarity to the question of whether improving test scores will mean anything in children's adult lives.

I've always thought it silly to conclude that America is failing because our students score in the middle of the pack (or toward the bottom) on international tests. We still are the world's dominant economy and military force, not to mention its creative and innovative hub. To say our public schools didn't have a lot to do with our country's nearly 8 decades as the world's powerhouse is extraordinarily unfair, or at least intellectually dishonest.

What this report also tells me is that focusing on test score improvement does not necessarily mean we will be improving the economic lives of our citizens. For if we do it by leaving behind the subjects that have separated our educational system from the world for decades, we'll be improving our test scores by limiting our students' future.

It also tells me that three decades into the charter school experiment, charters may not be providing the long-term benefit to our society that many hoped they would. And it certainly shows (again) that investing $1 billion in nearly 400, as Ohio does, seems unjustified. Investing in our state's few high-quality charters and dumping the poor performers seems like the obvious solution.

But seemingly obvious solutions have eluded our state's leaders for so long on this topic that I would fully expect it to continue.

For First Time Ever, Fewer Ohio Kids Attend Charters than Previous Year. Yet Funding Still Grows. Here's How...

For the last two years, the stories about Ohio's mostly failing charter school system have been numerous and compelling. In addition, the scrutiny has meant that websites like Know Your Charter have shined a fresh light on Ohio's charter school performance issues. The result has been the state's first enrollment drop at charters since the 1998 bill creating them passed.

However, there was NOT a commensurate drop in overall funding to the state's charters. How could this be? Especially given many charter school advocates' pleas for a "money follows the child" system? If money truly "followed the child", shouldn't a reduction in the number of children create fewer, not more charter school resources?

Of course.

However, in Ohio, we don't have that system. Instead, we have a system that has Ohio's legislature and governor protect the funding for the state's charter schools, whose operators have given millions to re-elect them.


In the 2015 budget, Gov. John Kasich signed into law provisions that gave charter schools significantly more additional facilities funding (even for e-schools, which - by definition - don't have facilities), as well as bonuses for graduation and third-grade reading guarantees. The budget also deducted funding from charters for regional Educational Service Centers, just like ever other school district. (By the way, remember this fact the next time anyone says the only way to fairly compare charter performance is between charters and school buildings rather than districts, for they're paid and treated by the state as districts.)

The result was that charter schools' bottom lines were sufficiently bolstered by their 88% increase in add on funding to provide a modest increase in overall funding, as well as a 2.6% increase in per pupil state funding. So despite a significant drop in enrollment (and a far less significant drop in base funding to charters), the legislative add ons allowed for charter schools to keep their historically perfect record of annual, overall funding increases, regardless of macroeconomic realities.

The legislative add ons will continue for this next school year as well. So I would fully expect Ohio's charters to continue receiving more funding than previous years, even if their overall enrollment continues to drop.

Here is the chart showing how Ohio charter funding has continued growing, regardless of year or economy.

ECOT Parents Sue State. Will ODE Stand up for Parents of non-ECOT kids?

We're seeing the Electronic Classroom of Tomorrow fight for its life. Their latest gambit is using parents, who have joined the school's lawsuit to keep the Ohio Department of Education from determining whether the $100 million per year the state sends to ECOT is justified.

Don't think for a second that ECOT's fighting for parents of its kids. ECOT is fighting to keep the $100 million, mile-long gravy train rolling. All ODE wants to do is inspect that train's box cars to see if they're empty.

But this is a typical poor performing charter school/school choice Ohio tactic: run out the two or three parents who love the school and make it seem like closing the school is an affront to American decency, apple pie, Old Glory, or something equally offensive. However, remember, not even 4 in 10 students graduate the nation's largest for-profit school on time and nearly 9 in 10 students stay in ECOT less than 2 years.

So the data shows pretty clearly that the vast majority of ECOT parents do not love ECOT.

But in all these stories about ECOT, media never take the next step: What has ECOT's repeated failure meant for kids and parents of children who are not in ECOT?

For example, in the 2014-2015 school year, ECOT removed 90 students and $672,928 from Westerville -- John Kasich's hometown. Every dollar sent to ECOT went to a far worse performing school option. Yet that $7,402 per pupil that the state sent from Westerville to ECOT meant that for every kid that went to ECOT, Westerville parents had to fork over nearly $5,000 in property taxes. That's because they had to make up the difference between ECOT's $7,402 and the $2,526 per pupil the state would have sent to Westerville if the students had stayed in that district.

So just to make up for the ECOT payments to those 90 kids, the remaining 14,685 kids in Westerville lost $30 each in state funding, which their parents had to make up for with $464,455 in property taxes. The loss of just 90 of Westerville's 14,775 kids to ECOT caused reduced opportunities for every kid in Westerville, and added to the local property tax burden of every land owning parent.

To add insult to injury, because the performance difference between ECOT and Westerville is so vast, this ECOT funding actually punishes a far higher performing district. All so a far worse performing charter can make enough money to pay their teachers, give every kid a $2,000 laptop every year and still clear more than 45% profit.

Remember, ECOT doesn't have buildings, buses, lunch ladies, custodians, and a whole host of other services brick and mortar schools have.

This is why it is so important for ODE to stick to its guns and figure out if any kids are receiving the instruction they deserve at ECOT, or whether our tax dollars are just paying for ECOT founder William Lager's lavish lifestyle (not to mention his political largess). Remember, ECOT makes enough money from the state that it could withstand as much as an 85% cut in funding and still pay 602 of its 607 teachers' salaries.

It is up to ODE to fight for the millions of families whose children have seen their educational opportunities significantly reduced because of ECOT's taxpayer-funded boondoggle. It is these parents whose stories go untold amid the bombast of ECOT's well-paid lobbyists and lawyers. These parents and children are hurt indirectly by ECOT almost as much as the thousands whom ECOT directly fails every year.