A response to the recently published report by Children At Risk:
A recent article in the San Antonio Express-News highlighted an annual school district ranking system published by Houston-based, non-profit organization Children at Risk. This organization created its ranking by using Texas Education Agency (TEA) data to construct a series of indices that claim to be unbiased to student demographics and that truly reflect the organizational effectiveness of the school districts.
I’m not convinced these rankings accurately depict a school district’s performance due to a serious limitation in the variables used to control for student demographics. After carefully examining the methods used in developing the ranking, my main criticism stems from the notion that student achievement is largely a product of structural inequalities in the Texas population. The Children at Risk rankings are merely reflecting those disparities rather than measuring organizational effectiveness. The rankings account for student disadvantage, but their variable fails to capture the multiple levels of disparities between the most advantaged students and their most vulnerable counterparts.
Children at Risk acknowledged that students’ economic standing biased the different components of their indices; thus, they statistically suppressed the effects of race/ethnicity and economic disadvantage by using TEA variables. Technically, their method allowed them to observe each district’s effectiveness independent of student demographics. However, the ranking system treated students within school districts as belonging to one of two categories (economically disadvantaged or not economically disadvantaged), when in fact there are multiple tiers of household incomes–each one more advantaged than the other. These are important considerations when ranking districts and campuses relative to each other because the effect of relative advantage will bias the rankings towards districts and campuses nested within geographic areas with high incomes.
As a way to demonstrate this point, P16Plus has taken U.S. Census income data and linked it to the Children at Risk rankings in an attempt to discover associations between a district’s position in the rankings and its corresponding income level. Income data from the Census Bureau provides a tiered view of income levels between districts that are absent in the economically disadvantaged variable provided by TEA. Consider Figures A and B.
Figure A illustrates income distributions of the top ten districts based on the Children at Risk rankings and Figure B illustrates the bottom ten district rankings. (Click on image to enlarge the graphic.)
The differences in income distribution in the rankings between the top ten and bottom ten districts are dramatic.
For example, Highland Park ISD, which received an “A” and is ranked 4th out of 953 school districts, has a median annual household income of $174,318 compared to Premont ISD, which is the lowest ranking district and has a median annual household income of $30,859. An example closer to home is Edgewood ISD with a median annual household income of $26,302. Edgewood scored an “F” and was ranked 910 out of 953 school districts in Texas.
Figure C is a scatter plot that shows the linear association between the districts’ median household income and their composite score on the ranking index. This positive association is statistically significant and provides evidence of the limitations in the methods used by Children At Risk to rank districts across Texas.
The bottom line is that the Children at Risk rankings do not accurately reflect a district’s performance in educating their student population. Instead, the rankings are a reflection of a systemic relative advantage that some student populations have over others. The student populations at Premont ISD or Edgewood ISD do not have the same level of resources as their peers at Highland Park ISD. Therein lies the core of educational inequalities across Texas, and that is what is truly illustrated in these rankings. Children across districts do not start off nor do they continue through their educational trajectories on a level plane. Instead, those children at Premont ISD and Edgewood ISD begin their educational paths with far less in their educational toolbox and continue to face mounting barriers as they move forward in the educational system. Similarly, middle income students also experience relative disadvantages when compared to students whose families have higher incomes.
If the Children at Risk rankings are to be improved, then a more robust control variable of socioeconomic disadvantage must be used instead of or in conjunction with the economically disadvantaged variable provided by TEA. District rankings like these inadvertently divert attention away from the big elephant in the room– that we live in a world where access to educational resources for all students is inherently unequal, and this is apparent by the stark differences in levels of household income across school districts. Additionally, there are multiple social, economic, psychological, health and political factors responsible for driving educational outcomes.
Unfortunately, the rankings provided by Children At Risk do not take into account any of these factors.
According to the nonprofit’s 2014 Texas Public School Rankings Methodology Page, “Children at Risk seeks to hold schools accountable for student performance on standardized testing in addition to other measures such as graduation rates and improvement over time.”
P16Plus agrees that campuses and school districts must be held accountable, but it must be done within the context of their geography, economics, and government. In doing so, quantitative evidence used for accountability must be held to the highest methodological standards and stand to be scrutinized by peer organizations like P16Plus.
This post has been published with permission from the P16Plus Council blog.
Related Stories:
The Spurs’ Gift to Bowden Elementary School
Sheriff Fighting Crime with A ‘Keep Kids in School’ Message
For Thirteenth Year, H-E-B Honors Top Public School Educators in Texas
The Culture of ‘You Can’: Michelle Obama to Celebrate College Week in San Antonio
This isn’t surprising. Every ranking system in existence correlates with family income. No Child Left Behind, based on the system H. Ross Perot set up in Texas long ago, rates every school based on test scores. It has led to the closure of many low-income schools. No rich ones.
Regardless of the reasoning for WHY their schools are performing poorly, they are still performing poorly. This article is simply another form of excuse-making that we have heard time and time again from SAISD (Dr. Perez is on the board for P16-Plus). Instead of correcting their issues, they take the time to attempt to justify their failures/deficiencies. No Excuses.
Regardless of the reasoning for WHY their schools are performing poorly, they are still performing poorly. This article is simply another form of excuse-making that we have heard time and time again from SAISD (Dr. Perez is on the board for P16-Plus). Instead of correcting their issues, they take the time to attempt to justify their failures/deficiencies. No Excuses.
There’s definitely no excuses, Brian, but I’d imagine the true, deep reason for low performance has nothing to do with the teachers or schools themselves – it starts at home, with the parents. If they don’t care to participate in their children’s education, then the stats are doomed for any given school.
I don’t read this piece as making excuses but asking us to seek a more complex answer to a complex problem. Here’s a quote “district rankings like these inadvertently divert attention away from the big elephant in the room– that we live in a world where access to educational resources for all students is inherently unequal, and this is apparent by the stark differences in levels of household income across school districts.” Point being, the greatest predictor of a child’s success is the mother’s level of education. Before you blame the mom, consider that as also a statement on a parent’s power to influence a child’s life in a positive way. Schools are only one component of a child’s life – there is a tremendous opportunity gap in addition to the achievement gap, and many factors, including health, influence education. What the article is arguing is that a more robust control for income would allow you to see which schools in fact are being most effective with low-income kids. I think a ranking like that would in fact be helpful, whereas this one just essentially reproduces the economic class status quo.
As a passionate educator in SAISD for 9 years now, I always get very annoyed when I hear excuses. Yes. The community we serve is more economically disadvantaged. They are. So…. That means we should expect lower academic performance? If that’s the case, why even try to better ourselves? A person’s ethnicity nor income has nothing to do with their ability to learn. These variables are not excuses. Do these factors make it more difficult for schools and teachers? Absolutely. And this is why SAISD needs the most talented, passionate and intelligent teachers there are. A good teacher can produce great results no matter what the circumstances a child is in. And I’m not just talking about test scores. They can produce innovative, bright, respectful kids- with or without the help of parents. It’s a huge obstacle to get around and not a task for the weary, but it’s doable and NOT something we can just surrender and give up on.
Jeanne,
“The bottom line is that the Children at Risk rankings do not accurately reflect a district’s performance in educating their student population.”
A direct quote from Syl Perez (SAISD Superintendent AND P16-Plus Board Trustee) on the original article (http://www.p16plus.org/p16plus-news/data-suggests-isd-rankings-need-closer-review/) goes even further to state,
“This article illustrates and reveals the lack of validity in the aforementioned ratings and results. Your deeper examination of the flaws in their study was refreshing. Coming from an independent organization like P-16 PLUS doesn’t portray defensiveness nor bias, but illustrates the misleading deficiencies of such a shallow methodology.
The damage by Children at Risk’s ratings to SAISD’s reputation & to our teacher’s as well as student’s morale was harmful. On behalf of our faculties, students, trustees and Community, THANK YOU for educating and clarifying this inaccurate portrayal of our students, schools and district.”
The rankings are what they are. My concern is that regardless of where these schools rank, based on whatever new algorithm you prefer to be thrown in the mix, our students are STILL FAILING. We continue to refresh these assessments and incorporate more ways to find out what we already know. Being poor sucks. Along with that the rule is that if you are born poor, you’ll most likely go to a underperforming school and will live a mediocre life. Done. Assessment complete. NOW, what are we going to do to change this from being the rule, to be the exception?
I know the city loves to stand on their reigning accomplishment of PreK4SA, but what happens when our kids from PreK-SA end up in a failing middle school? We look at the situation and say “Oh, we need a better ranking/assessment procedure to find out why our kids are poor”? There are way too many intelligent folks running the show (yourself included) for us to continue to be in this type of situation.
This article is another excuse produced to take the heat or “stigma” off of SAISD’s back. Can we get some implemented LONG-TERM solutions? Let me know when that article comes out.
I was at a training today (yes, a teacher voluntarily at a teaching training on her second day of summer vacation) and saw this slide on the Elmo. It shows a research publication written by a woman from Harvard I believe. I don’t know all the details, and I would like to read this study, but in short, as you can see by the illustration, it shows that a student from low home support can achieve just as high as a high home support if and only of they have high classroom support. ??. The book is called Unfulfilled Expectations. Just thought I’d share! I thought about this post as soon as I saw it.