September, 2014 - AZ Charter Schools Association

The Association believes education leads to opportunities in life, and we work hard to make data transparent and actionable. As we look at Arizona’s A-F accountability system, we are asking whether our students are in quality seats.

For our purposes, a student is enrolled in a quality seat if they attend a Tier I school (A or B rating). At the end of the day, we want to be able to measure the change in the number of quality public school seats available for students across Arizona — district and charter.

As the Association has pointed out in several of our publications, the relationship between poverty and measures of student achievement has long been a limitation of measurement of student achievement in education. The challenge for Arizona’s accountability model is to adequately control for the effect of poverty on the final school rating. As we noted in our publications, despite the inclusion of the student growth percentiles, the A-F model still has limited use in portraying school quality due to its failure to adequately control for the effects of poverty and fairly measure a school’s contribution to learning. However, all those issues notwithstanding, we still believe analyzing these data will provide insight into Arizona’s current accountability system.

A-F data published by the Arizona Department of Education in August indicates an overall increase in the percentage of A and B rated schools, with nearly 2 out of 3 schools achieving these results (62 percent of schools were rated A or B in the traditional model). To put this number in context, in 2012 54 percent of schools were rated A or B, in the traditional model. While it’s important to see the numbers of schools improving, we wanted to take a closer look at the number of students impacted by these improved letter grades. In order to do this we designated each school into one of three tiers based on their A-F letter grade. Tier I represents all schools receiving an A or B rating; Tier 2 are C rated schools, and Tier 3 are schools earning a D or F rating.

We analyzed the data from the past three years of A-F accountability results to evaluate the trends in the number of quality seats. The following tables present the data for all public schools — district and charter — in each of the past three years. The number of students and the percentage of their enrollment is reported annually. The data show, as expected given the overall increase in the number of A and B rated schools, a larger number of students are enrolled in a quality seat in 2014 (69 percent) as compared to 2012 (62 percent). As the number of Tier 1 seats increased, the number of Tier 2 and Tier 3 seats decreased, demonstrating an overall improvement in the number of seats available across Arizona. However, an alarming number of seats are not rated in 2014, more than double the number in 2013. It should be noted that the Arizona Department of Education has not yet finalized the “pending” schools which account for a large number of these seats without data.

These data present an alternative lens to evaluate the impact of Arizona’s school accountability measure. Certainly, these results are affected by countless school improvement decisions at the classroom, school, LEA and state level (including the work of the Arizona State Board for Charter Schools). No matter the decisions being made, these data make it clear that, at least according to our A-F results, more of our students are enrolled in a quality seat now than in 2012 when we started.

In 2012, the Association released School Ratings: Improving The Data In Data-Driven Decision Making where we examined the relationship between Arizona’s A-F school letter grades and the level of poverty in a school. We found that a school’s letter grade is highly and negatively correlated with the percentage of its students who qualify for free and reduced priced lunch. In other words, schools with high percentages of poor students received lower letter grades, on average, than schools with more affluent students. This relationship is due, in great part, to the way in which Student Growth Percentiles (SGPs) are used in the accountability formula. Specifically, we found that the number of “growth points” awarded in the formula is artificially capped by the calculation which uses the averages of multiple medians.

After broadly communicating these concerns to state-level policy makers, which resulted in no changes to this portion of the A-F formula, we wondered if this was still the case in 2013 and 2014. As we expected, schools continued to earn most of their letter grade from the percentage of students who are proficient on AIMS (composite points). As the percentage of poor students in a school increases, the percentage passing continued to decrease (see Tables 1, 2 and 3). A closer look at the SGP scores show, as they did in 2012, that growth points remain consistent across poverty levels, meaning that growth is not strongly related to student demographics.

As in the initial 2012 report, we again call into question the implementation of the current accountability model. State statute (ARS§ 15-241) clearly calls for equal weight to be assigned to student achievement and growth (50-50), what we continue to see is closer to 60-40, weighted toward AIMS passing scores, a measure highly related to poverty.

Evidence from three years of A-F results indicates that under this accountability model, schools with high levels of poverty are at a disadvantage. For example, the difference in composite points between a school where less than 25 percent of its students qualify for free and reduced priced lunch and a school where more than 75 percent qualify is 29 points, or the difference between an A and C letter grade. Given these consistent findings, we must ask ourselves “Is our school accountability model measuring school quality or student demographics?”

In the coming year, the Arizona Department of Education and the State Board of Education have the opportunity to develop a new school accountability model to reflect the change in the state’s new College and Career Ready Standards. We encourage our state policymakers to consider this research and develop accountability models that more accurately measure school quality, rather than student demographics, so that schools with the toughest challenges aren’t disadvantaged.