Monday, August 22, 2016

Testing the Boundaries: Part II - What Can Outliers Teach Us?

We've spent 12 years obsessing over two things: tests and achievement gaps. Yet it appeared that the efforts have been for naught, as the achievement gap has continued to grow.

However, there is some good news. The state's poorest districts saw the biggest percentage improvements in their test scores over those 12 years. The bad news is that improvement, while slightly narrowing the raw score performance gap between rich and poor districts, widened the state's relative performance gap. Meaning, it wasn't big enough to make a difference, relatively speaking, because high performing districts also significantly improved their scores.

You can see the problem in the following charts. Pay attention to what happens between the Suburban, very low student poverty and Urban, very high student poverty categories. You'll see impressive improvement among the poor category, which includes Ohio's so-called "Big 8" urban districts -- Akron, Canton, Cincinnati, Cleveland, Columbus, Dayton, Toledo and Youngstown. These are the districts state lawmakers have targeted for years as "failing" kids. Though, looking at the raw score improvement, one can't be anything but impressed by the 12-year jump.




Here's the problem: All districts improved, even the wealthiest districts. So despite the state's poorest districts' impressive improvement, all the other districts' improvement was enough to widen the relative performance gap so that the wealthiest districts made up a significantly higher proportion of top 10% mini-PI scores in 2014-2015, and the poorest districts made up a significantly higher proportion of the bottom 10% mini-PI scores.

But it cannot be ignored that Ohio's maligned, urban districts improved their math scores by more than 21%, and the improvement among all districts with high or very high poverty saw scores jump by 20% or more. That's impressive. But it wasn't enough to improve their status among Ohio's districts. And, in fact, they dropped in comparison with their wealthier brethren.



So what gives? Why did a 20% improvement lead to the widening of the achievement gap? A major reason is the upper limit of the mini-PI score. The upper limit of the PI score is 120. That's the maximum score any district could receive. And while some districts get close, getting a perfect 120 is probably impossible. So in the 2003-2004 school year, the average PI score for the OGT math score in the state's wealthiest, suburban districts was 102.84 -- a mere 18.16 points from perfection. So the greatest percentage improvement they could have was a bit more than 17%. The average for the state's poorest, urban districts was 68.26 -- 51.84 points from perfection. So the greatest percentage improvement they could have was 76%. 

In the intervening years, the wealthiest districts went up to an average of 112.25 -- an amazingly close 7.75 points from perfection, or more than 1/2 of their maximum percentage improvement. And the state's urban districts went to 82.26 -- less than 1/3 of their maximum percentage improvement. 

So what this meant is more suburban districts performed at an elite level, while the state's urban districts, which performed much better, remained mired in the bottom 10% of mini-PI Scores.

But this doesn't mean there aren't outliers. Take Lakota in Sandusky County -- classified by the state as a rural, poor district. In 2004, their OGT Math score placed them 511 of 608 districts. Not good. But in 2015, their score jumped so much that they are now in the top 1/3 of districts. Or Maplewood in Trumbull County. In 2004, they were in the bottom 1/2 of Ohio districts on OGT math scores. Now, they're in the top 5%.

In fact, the greatest rank jumpers in the state were the rural districts. They made up almost 1/2 of the greatest rank improvements. 

Here's the other problem: They also made up about 40% of the greatest rank fallers too.

Tellingly, no urban district was in the top 10% of climbers or fallers.

Again, is there something interesting going on in the districts with the greatest improvements? Or something awful happening in the fallers? Or is it just statistical noise -- a true outlier? One would have to go to the district and see what, if any, change has led to the relative improvement. But given the predictive force of poverty on these scores, normal statistical variation would seem to explain much of this difference.

What does all this mean from a policy perspective? Well, it means that if improvement or performance slippage can be explained in large part by statistical variation, should we be granting kudos or shame to districts that grow or slip? It'd be easy to devise a method to reward districts that show dramatic improvement on scores, even if their relative performance to other districts remains low. Likewise, it can be easy to punish districts that slip. But should we, given what we know about statistical variation in these cases? There will always be outliers -- districts that out- or under-perform their demographics. But if there's one thing variation means it's that over or under performance could, in fact, be nothing more than yet another expected statistical result of standardized testing. Every district has their strong or weak classes pass through the system.

In other words, a district's score may not be indicative of the actual quality of their educational program. Yet, under our current accountability system, it is the dominant determinant of a school or district's quality.

But that doesn't mean these score changes mean nothing. Figuring out what they mean is a great challenge, but one that must be unlocked in order to understand how, if at all, they should inform our nation's education policy.


Tuesday
Testing the Boundaries
Part III - Finding Excellence through the Noise

48 comments:

  1. Wonderful post..!!Thanks for sharing your interesting blog of post.
    Android Training in Chennai | Android Course in Chennai

    ReplyDelete
  2. Applicants are happy to find out Duke University serves as a medical center and nursing school at the same time. Students can get valuable work experience without living their campus! Click top high schools in the nation to find more!

    ReplyDelete

  3. Excellent post! When you are going to update your next post, I really very excited to see your upcoming articles.
    Selenium Training Chennai|software testing selenium training

    ReplyDelete
  4. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information.
    Software testing training|Software training

    ReplyDelete
  5. Great information. I have got some important suggestions from it. I'm working in ERP Software Company in Chennai. | ERP Providers in Chennai.

    ReplyDelete
  6. Thanks for sharing this information. This is very useful. Thanks mate

    ReplyDelete
  7. Thank you for this valuable information. I have got some important suggestions from it. Get your business to the next level in simple steps.
    ERP Providers in Chennai

    ReplyDelete
  8. This is an extraordinary article. It will help to improve my knowledge. Thanks for sharing this useful blog.Software testing institutes in Chennai | Software testing Training Institute Chennai

    ReplyDelete
  9. Really nice information. It is very useful and informative post. Thanks for sharing.
    school parent communication app india

    ReplyDelete
  10. Outstanding blog thanks for sharing such wonderful blog with us ,after long time came across such knowlegeble blog. keep sharing such informative blog with us.
    Air Hostess Training in Chennai | Air Hostess Training Institute in Chennai | Air Hostess Academy in Chennai | Air Hostess Course in Chennai | Air Hostess Institute in Chennai

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. This blog is very much helpful to us. Thanks for your information

    empoweriasacademy
    Technology

    ReplyDelete
  13. This is really too useful and have more ideas and keep sharing many techniques. Eagerly waiting for your new blog keep doing more.
    Aws Training in Bangalore
    Aws Course in Bangalore
    Best Aws Training in Bangalore
    hadoop classes in bangalore

    ReplyDelete
  14. Such an excellent and interesting blog, do post like this more with more information, this was very useful, Thank you.
    Aviation Academy in Chennai
    Aviation Courses in Chennai
    best aviation academy in chennai
    aviation training in chennai

    ReplyDelete
  15. This information is impressive; I am inspired with your post. Keep posting like this, This is very useful.Thank you so much. Waiting for more blogs like this.
    Airport management courses in chennai
    airline management courses in chennai
    aircraft maintenance course in chennai
    diploma in airport management in chennai

    ReplyDelete
  16. These details is actually amazing; I'm influenced together with your publish. Maintain publishing such as this, This is helpful. Thanks a lot. Awaiting much more http://chicago.bibhelper.net/ such as this.

    ReplyDelete
  17. Specialists the showdown was already fought and even compensated. At present, you will click here the sharp main difference around these kinds of interpretations in Analytics.

    ReplyDelete
  18. That is a fantastic write-up. It helps to boost my own information go to site . Thank you regarding revealing this kind of beneficial website.

    ReplyDelete
  19. Thanks for information , This is very useful for me.
    Keep sharing Lean Six Sigma Green Belt Training Bangalore


    ReplyDelete
  20. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here.
    iosh course in chennai

    ReplyDelete
  21. Great blog post, Our life we have to take off enhancements in us by confirmation that http://www.mbapersonalstatement.org/how-to-write-a-perfect-gmat-waiver-letter-for-mba-application/ site we in like path need to careful people about preparing.

    ReplyDelete
  22. I appreciate to the author who shares wonderful info. This https://www.statementofpurposeformba.com/our-sop-for-mba-writing-service/statement-of-purpose-for-project-management-writing-service/ site is really helpful to the students to manage your any kind of work and progressively constructive choice in this field. A firefighter is working for the comprehensive network.

    ReplyDelete
  23. Wondeful Explanation with lots of useful information in this blog. Congrats and Keep Rocking.

    QTP Training in Chennai | SAP Training in Chennai | SAP ABAP Training in Chennai | SAP FICO Training in Chennai

    ReplyDelete

Note: Only a member of this blog may post a comment.