A response to the recently published report by Children At Risk:
A recent article in the San Antonio Express-News highlighted an annual school district ranking system published by Houston-based, non-profit organization Children at Risk. This organization created its ranking by using Texas Education Agency (TEA) data to construct a series of indices that claim to be unbiased to student demographics and that truly reflect the organizational effectiveness of the school districts.
I’m not convinced these rankings accurately depict a school district’s performance due to a serious limitation in the variables used to control for student demographics. After carefully examining the methods used in developing the ranking, my main criticism stems from the notion that student achievement is largely a product of structural inequalities in the Texas population. The Children at Risk rankings are merely reflecting those disparities rather than measuring organizational effectiveness. The rankings account for student disadvantage, but their variable fails to capture the multiple levels of disparities between the most advantaged students and their most vulnerable counterparts.
Children at Risk acknowledged that students’ economic standing biased the different components of their indices; thus, they statistically suppressed the effects of race/ethnicity and economic disadvantage by using TEA variables. Technically, their method allowed them to observe each district’s effectiveness independent of student demographics. However, the ranking system treated students within school districts as belonging to one of two categories (economically disadvantaged or not economically disadvantaged), when in fact there are multiple tiers of household incomes–each one more advantaged than the other. These are important considerations when ranking districts and campuses relative to each other because the effect of relative advantage will bias the rankings towards districts and campuses nested within geographic areas with high incomes.
As a way to demonstrate this point, P16Plus has taken U.S. Census income data and linked it to the Children at Risk rankings in an attempt to discover associations between a district’s position in the rankings and its corresponding income level. Income data from the Census Bureau provides a tiered view of income levels between districts that are absent in the economically disadvantaged variable provided by TEA. Consider Figures A and B.
Figure A illustrates income distributions of the top ten districts based on the Children at Risk rankings and Figure B illustrates the bottom ten district rankings. (Click on image to enlarge the graphic.)
The differences in income distribution in the rankings between the top ten and bottom ten districts are dramatic.
For example, Highland Park ISD, which received an “A” and is ranked 4th out of 953 school districts, has a median annual household income of $174,318 compared to Premont ISD, which is the lowest ranking district and has a median annual household income of $30,859. An example closer to home is Edgewood ISD with a median annual household income of $26,302. Edgewood scored an “F” and was ranked 910 out of 953 school districts in Texas.
Figure C is a scatter plot that shows the linear association between the districts’ median household income and their composite score on the ranking index. This positive association is statistically significant and provides evidence of the limitations in the methods used by Children At Risk to rank districts across Texas.
The bottom line is that the Children at Risk rankings do not accurately reflect a district’s performance in educating their student population. Instead, the rankings are a reflection of a systemic relative advantage that some student populations have over others. The student populations at Premont ISD or Edgewood ISD do not have the same level of resources as their peers at Highland Park ISD. Therein lies the core of educational inequalities across Texas, and that is what is truly illustrated in these rankings. Children across districts do not start off nor do they continue through their educational trajectories on a level plane. Instead, those children at Premont ISD and Edgewood ISD begin their educational paths with far less in their educational toolbox and continue to face mounting barriers as they move forward in the educational system. Similarly, middle income students also experience relative disadvantages when compared to students whose families have higher incomes.
If the Children at Risk rankings are to be improved, then a more robust control variable of socioeconomic disadvantage must be used instead of or in conjunction with the economically disadvantaged variable provided by TEA. District rankings like these inadvertently divert attention away from the big elephant in the room– that we live in a world where access to educational resources for all students is inherently unequal, and this is apparent by the stark differences in levels of household income across school districts. Additionally, there are multiple social, economic, psychological, health and political factors responsible for driving educational outcomes.
Unfortunately, the rankings provided by Children At Risk do not take into account any of these factors.
According to the nonprofit’s 2014 Texas Public School Rankings Methodology Page, “Children at Risk seeks to hold schools accountable for student performance on standardized testing in addition to other measures such as graduation rates and improvement over time.”
P16Plus agrees that campuses and school districts must be held accountable, but it must be done within the context of their geography, economics, and government. In doing so, quantitative evidence used for accountability must be held to the highest methodological standards and stand to be scrutinized by peer organizations like P16Plus.
This post has been published with permission from the P16Plus Council blog.
Related Stories:
The Spurs’ Gift to Bowden Elementary School
Sheriff Fighting Crime with A ‘Keep Kids in School’ Message
For Thirteenth Year, H-E-B Honors Top Public School Educators in Texas
The Culture of ‘You Can’: Michelle Obama to Celebrate College Week in San Antonio
