8 EDUCATION NEXT / F A L L 2 0 1 5 educationnext.org

PH

OT

OG

RA

PH

/CH

RIS

GR

AN

GE

R,T

HE

TIM

ES

-PIC

AY

UN

E,

LA

ND

OV

What happened to the New Orleans public schools following the tragic levee breeches after Hurricane Katrina is truly unprecedented. Within the span of one year, all public-school employees were fired, the teacher contract expired and was not replaced, and most attendance zones were eliminated. The state took control of almost all public schools and began holding them to relatively strict standards of academic achievement. Over time, the state turned all the schools under its authority over to charter management organizations (CMOs) that, in turn, dramatically reshaped the teacher workforce.

A few states and districts nationally have experimented with one or two of these reforms; many states have increased the number of charter schools, for example. But no city had gone as far on any one of these dimensions or considered trying all of them at once. New Orleans essentially erased its traditional school district and started over. In the process, the city has provided the first direct test of an alternative to the system that has dominated American public education for more than a century.

Dozens of districts around the country are citing the New Orleans experience to justify their own reforms. In addition to being hailed by Democratic president Barack Obama and Louisianas Republican governor, Bobby Jindal, parliamentary delegations from at least two countries have visited the city to learn about its schools.

by DOUGLAS N. HARRIS

feature

GOOD NEWS FOR NEW ORLEANS

Early evidence shows reforms lifting student achievement

EMBARGOED

educationnext.org F A L L 2 0 1 5 / EDUCATION NEXT 9

2002 2003 2004 2005 2007 20082006 2009 2010 2011 2012-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

New Orleans

Sca

le t

est

scor

e (

stan

dard

dev

iati

on)

Matched Comparison

Hurricane Katrina hits New Orleans

StateAverage

NOTES: Separate analyses show that New Orleans students returning after the storm, who could be studied only through 2009, also made gains relative to their own prior performance, but these differences were often not statistically significant. Scale scores are averaged across grades 3 through 8 and across English Language Arts, math, science, and social studies. Scale scores are standardized so that zero refers to the statewide average.

SOURCE: Authors calculations

A New Trajectory (Figure 1)Following Hurricane Katrina and the reforms enacted in its aftermath, the performance of New Orleans students rose steadily compared to that of students in other Louisiana districts that were affected by the hurricane.

EMBARGOED

10 EDUCATION NEXT / F A L L 2 0 1 5 educationnext.org

The unprecedented nature of the reforms and level of national and international attention by themselves make the New Orleans experience a worthy topic of analysis and debate. But also consider that the underlying principles are what many reformers have dreamed about for decadesthat schools would be freed from most district and union contract rules and allowed to innovate. They would be held accountable not for compliance but for results.

There is clearly a lot of hype. The question is, are the reforms living up to it? Specifically, how did the reforms affect school practices and student learning? My colleagues and I at the Education Research Alliance for New Orleans (ERA-New Orleans) at Tulane University have carried out a series of studies to answer these and other questions. Our work is motivated by the sheer scale of the Katrina tragedy and the goal of supporting students, educators, and city leaders in their efforts to make the citys schools part of the citys revitalization effort. The rest of the country wants to know how well the New Orleans school reforms have worked. But the residents of New Orleans deserve to know. Heres what we can tell them so far.

Before the StormAssessing the effects of this policy experiment involves com-

paring the effectiveness of New Orleans schools before and after the reforms. As in most districts, before Hurricane Katrina, an elected board set New Orleans district policies and selected superintendents, who hired principals to run schools. Principals hired teachers, who worked under a union contract. Students were assigned to schools based mainly on attendance zones.

The New Orleans public school district was highly dysfunc-tional. In 2003, a private investigator found that the district system, which had about 8,000 employees, inappropriately provided checks to nearly 4,000 people and health insurance to 2,000 people. In 2004, the Federal Bureau of Investigation (FBI) issued indictments against 11 people for criminal offenses against the district related to financial mismanage-ment. Eight superintendents served between 1998 and 2005, lasting on average just 11 months.

This dysfunction, combined with the socioeconomic back-ground of city residents83 percent of students were eligible for free or reduced-price lunchcontributed to poor academic results. In the 200405 school year, Orleans Parish public schools ranked 67th out of 68 Louisiana districts in math and reading test scores. The graduation rate was 56 percent, at least 10 percentage points below the state average.

As a result, some reforms were already under way when Katrina hit in August 2005. The state-run Recovery School District (RSD) had already been created to take over low-performing New Orleans schools. The state had appointed an emergency financial manager to handle the districts finances. There were some signs of improvement in student outcomes just before the storm, but, as we will see, these were relatively modest compared with what came next.

A Massive Experiment After Katrina, state leaders quickly moved almost all public

schools under the umbrella of the RSD, leaving the higher-per-forming ones under the Orleans Parish School Board (OPSB).

Gradually, the RSD turned schools over to charter operators, and the teacher workforce shifted toward alternatively prepared teachers from Teach for America and other programs. So new was the system that a new name was requiredlongtime education reformer Paul Hill called it the portfolio model.

Researchers often refer to such sudden changes as natu-ral experiments and study them using a technique called difference-in-differences. The idea is to first take the differ-ence between outcomes before and after the policy, in the place where it was implementedthe treatment group. This first dif-ference is insufficient, however, because other factors may have

Gradually, the RSD turned schools over to charter operators, and the teacher workforce shifted toward alternatively prepared teachers from Teach for America and other programs. PH

OT

OG

RA

PH

/ J

OH

N M

CC

US

KE

R,T

HE

TIM

ES

-PIC

AY

UN

E,L

AN

DO

V

EMBARGOED

educationnext.org F A L L 2 0 1 5 / EDUCATION NEXT 11

feature

NEW ORLEANS REFORMS HARRIS

affected the treatment group at the same time. This calls for making the same before-and-after comparison in a group that is identical, except for being unaffected by the treatment. Subtracting these twotaking the difference of the two differ-ences between the treatment and comparison groupsyields a credible estimate of the policy effect.

We have carried out two difference-in-differences strategies: 1) Returnees only. We study only those students who

returned to New Orleans after Hurricane Katrina. The advan-tage of this approach is that it compares the same students over time. One disadvantage is that it omits nonreturnees. Also, we can only study returnees over a short period of timeafter 2009, they no longer have measurable outcomes to study.

2) Different cohorts. We consider the achievement growth of different cohorts of students before and after the reformsfor example, students in 3rd grade in 2005 and students in 3rd grade in 2012. The advantages here are that we can include both returnees and nonreturnees, and we can use this strategy to study longer-term effects. But the students are no longer the same.

In both strategies, the New Orleans data set includes all pub-licly funded schools in the city, including those governed by the district (OPSB), since all public schools were influenced by the reforms. The main comparison group includes other districts in Louisiana that were affected by Hurricane Katrina, and by Hurricane Rita, which came soon afterward. This helps account for at least some of the trauma and disruption caused by the storms, the quality of schools students attended in other regions while their local schools were closed, and any changes in the state tests and state education policies that affected both groups.

Effects on Average AchievementFigure 1 shows the scores for each cohort, separately for

New Orleans and the matched comparison group. The scores cover grades 3 through 8, are averaged across subjects, and are standardized so that zero refers to the statewide mean. The first thing to notice is that before the reforms, students in New Orleans performed far below the Louisiana average, at about the 30th percentile statewide. Students from the comparison districts also lagged behind the rest of the state, but by a lesser amount. The New Orleans students and the comparison group

were moving in parallel before the reforms, however, suggest-ing that our matching process produced a comparison group that is more appropriate than the state as a whole.

The performance of New Orleans students shot upward after the reforms. In contrast, the comparison group largely continued its prior trajectory. Between 2005 and 2012, the performance gap between New Orleans and the comparison group closed and eventually reversed, indicating a positive effect of the reforms of about 0.4 standard deviations, enough to improve a typical students performance by 15 percentile points.

The estimates we obtain when we focus just on returnees are smaller and often not statistically significant, although the discrepancies are predictable: first, the returnees were probably more negatively affected by trauma and disruption; second, creating a new school system from scratch takes time, so we would expect any effects to be larger in later years; and third, the effects of the reforms seem more positive in early elementary grades, and the returnees were generally in middle school when they returned. Even so, the combination of analyses suggests effects of at least 0.2 standard deviations, or enough to improve a typical students performance by 8 percentage points.

But there is still the possibility that what appear to be reform effects are actually the result of other factors.

Addressing Additional ConcernsThe goal of any analysis like this is to rule out explana-

tions for the changes in outcomes other than the reforms themselves. Our main comparisons deal with many potential problems, such as changes in state tests and policies. Here we consider in more depth four specific factors that could bias the estimated effects on achievement: population change, interim school effects, hurricane-related trauma and disruption, and test-based accountability distortions.

Population change. Hurricane Katrina forced almost every-one to leave the city. Some returned and some did not. The most heavily flooded neighborhoods were (not coincidentally) those where family incomes were lowest, and people in these neighborhoods returned at much lower rates than people who lived in other parts of the city. Given the strong correlation between poverty and student outcomes, this could mean that higher test scores shown in Figure 1 are driven not by the

In the school year just prior to Hurricane Katrina, Orleans Parish public schools ranked 67th out of 68 Louisiana districts in math and reading test scores.

EMBARGOED

12 EDUCATION NEXT / F A L L 2 0 1 5 educationnext.org

reforms but by schools serving more-advantaged students.Observers have pointed out that the share of the student

population eligible for free or reduced-price lunch (FRL) actu-ally increased slightly in New Orleans after the storm. But there are many reasons not to trust FRL data. For example, they reflect crude yes/no measures and are unlikely to capture extreme pov-erty of the sort common in New Orleans. Also, what really mat-ters here is not whether poverty increased in New Orleans, but whether poverty increased more than in the comparison group. Therefore, in addition, we gathered data from the U.S. Census, which measures changes in income and the percentages of the population with various levels of education. We also carried out the difference-in-differences analysis in these demographic measures to understand the changes in New Orleans relative to the matched comparison group of hurricane-affected districts, and then simulated the effect of changes in family background characteristics on test scores using data from the federal Early Childhood Longitudinal Study.

We also examined pre-Katrina characteristics to see whether the returnees were different from nonreturnees and found that returnees did have slightly higher scores. In fact, we come to the same conclusion in both analyses: the expected increase in student outcomes after the hurricanes due to population change is no more than 0.02 to 0.06 standard deviations, or about 10 percent of the difference-in-differences estimates in Figure 1.

Interim school effects. Some of the changes in student learn-ing may reflect neither the prestorm nor poststorm quality of New Orleans schools, but the performance of schools that students briefly attended outside the city after the evacuation. Other research on these students by Dartmouth economist Bruce Sacerdote suggests that New Orleans evacuees experi-enced larger improvements in school quality than evacuees from other districts.

Trauma and disruption. Any benefit of having good interim schools might be offset by the trauma and disruption of the storm itself and its aftermath. The majority of New Orleans returnees probably knew someone among the nearly 2,000 people who died in the Katrina aftermath. Also, almost all stu-dents experienced significant disruption, moving to unfamiliar neighborhoods and schools for extended periods. Reports of post-traumatic stress disorder remain common.

It is difficult to isolate trauma and interim school effects,

but we can estimate the combination of the two. A study by the RAND Corporation of students from Louisiana districts affected by the hurricane suggests that these two factors had a short-term net negative effect on evacuees performance of 0.03 to 0.06 standard deviations. Our analysis suggests that the negative influence is even larger for New Orleans students, most likely because of the more extensive destruction in the city compared with most other areas along the states coast. Thus, at least in the years just after the reforms, the factors pushing student outcomes down were at least as large as the population changes pushing them up.

Test-based accountability distortions. One key part of the New Orleans reforms was the idea that the state would shut down schools within three to five years if they did not generate a high enough School Performance Score, a measure based on test scores and graduation rates. Prior research suggests that such intensive test-based accountability can lead to behav-iors, such as teaching to the test, that increase scores without improvements in underlying learning or through reduced learning in nontested subjects.

To address this problem, we estimate effects separately by subject, recognizing that the stakes attached to math and language scores were roughly double the stakes for science and social studies scores during the period under analysis. Also, the states social promotion policy raises the stakes for students in grades 4 and 8. We find no evidence that the size of effects varied systematically with the stakes attached to the subjects or grades. However, it is hard to rule out other potential test-based accountability distortions with our data.

As further evidence, we considered descriptive information on nontest outcomes. State government reports indicate that, relative to the state as a whole, the New Orleans high school graduation rate and college entry rate (among high school graduates) rose 10 and 14 percentage points, respectively.

So, in theory, there are many challenges to estimating the effects of the New Orleans package of school reforms. The combined effect of these alternative factors on long-term achievement gains appears small, however, especially when compared with our initial estimate of the reform effects.

There is a clear pattern across these methods. The estimates are consistently within the same range, and even the lower end of that range suggests large positive effects.

There was an upward change in the trajectory of student test scores in New Orleans after the reforms of about 0.2 to 0.4 standard deviations, enough to improve a typical students performance by 8 to 15 percentile points.

EMBARGOED

educationnext.org F A L L 2 0 1 5 / EDUCATION NEXT 13

feature

NEW ORLEANS REFORMS HARRIS

Equity of OutcomesPublic schools exist to ensure that all children have an

opportunity to succeed in life. Thus we consider not only the average effects of the reform package, but also whether the most-disadvantaged students benefited.

We first define equity in terms of how New Orleans, as an urban district, performed relative to districts serving more-advantaged students. Both before and after the reforms, at least 80 percent of New Orleans students were minority or eligible for FRL. It is therefore noteworthy that the reforms brought the citys students near to the state average on a wide range of academic outcomes (see Figure 1).

It is also important to consider the distribution of effects within the city, and here the results are more mixed. All major subgroups of studentsAfrican American, low-income, special education, and English Language Learners (ELL)were at least as well-off after the reforms, in terms of achievement. Critics of charter schools express concern about possible increases in racial isolation (some would say segregation). Among all of the various sub-groups we considered, only Hispanic students seem to have experienced increases in isolation.

There have also been concerns about schools unfairly targeting low-income and African American students in disciplinary decisions. While we have not yet studied whether any student groups have been specifically targeted, we can say that the number of suspensions and expulsions has dropped since the reforms, for African American students and others alike.

There are a few less-positive signs, how-ever. In our analysis of what families look for when choosing schools, we found that the lowest-income families place less weight on the School Performance Score than other families. Their circumstances may lead them to focus more on practical considerations such as distance to school and extended hours (to avoid extra child-care costs). Similarly, in our analysis of student mobility, we see that low-scoring students are less likely than high-scoring students to migrate toward schools with high scores. Finally, up until a few years ago, principals reported cherry-picking stu-dents by, for example, counseling out students deemed poor fits and holding invitation-only events to attract certain students.

Given the large improvements in average outcomes in a district that is almost entirely low-income and minority, and the mixed evidence on other equity indicators, it would

be hard to say the outcomes from the New Orleans reforms are inequitable relative to what came before them. That said, they were highly inequitable to start with, and there is clearly room for improvement.

What Really Changed? To help improve the schools going forward, it is important

to know how school practices and other intermediate outcomes changed. In a series of 15 ongoing studies, my collaborators at ERA-New Orleans and I have examined four main components of the reforms: choice and competition, teachers and leaders, charters and CMOs, and test-based accountability.

Some of the reform effect may be driven by parental choice

In terms of achievement, all major subgroups of students were at least as well-off after the reforms.PH

OT

OG

RA

PH

/ S

US

AN

PO

AG

,TH

E T

IME

S-P

ICA

YU

NE

,LA

ND

OV

EMBARGOED

14 EDUCATION NEXT / F A L L 2 0 1 5 educationnext.org

and competition. The supply of schools in New Orleans appears highly differentiated. Some schools specialize in math and sci-ence, others in the arts. Some schools offer language immersion programs, while other schools have fairly traditional curricula. Some schools have selective admissions, while others are open enrollment or seek diverse student bodies. We also find that New Orleans families diverge in their schooling preferences, so having this degree of differentiation in schooling options is likely to help match what families want with what schools offer (see The New Orleans OneApp, features, and Many Options in New Orleans Choice System, research, Fall 2015).

It is still unclear, however, whether these changes in the market have contributed to the improvements in student out-comes. Even supporters of the reform efforts sometimes bristle when I use the word market and competition to describe the new system. Instead, they point to two other parts of the reform package: the authority of the state to close schools and the authority schools have over their teaching staffs.

Sixteen New Orleans schools have been completely closed and another 30 have been taken over in some fashion by either the RSD or OPSBa large number in a city that has only about 90 public schools in total. Consistent with written state policies,

we find that the School Performance Score is the strongest mea-surable driver of closure and renewal decisions. Moreover, in finding CMOs to open new schools and take over old ones, the RSD has preferred those with a track record of academic success.

School leaders in New Orleans talk frequently about how critical flexibility in personnel management is to their overall school success. Free of state and local mandates and constraints from union contracts, leaders reopening schools after the storm could hire anyone they wanted, including uncertified teachers, and dismiss teachers relatively easily. As CMOs took over, more of the teacher workforce came from alternative preparation programs such as Teach for America and The New Teacher Project. Consistent with some other studies, analyses commissioned by the state suggest that graduates of these programs contribute more to student achievement than graduates of traditional preparation programs.

The combination of policies had two types of effects on the teacher workforce. First, the percentages of teachers with regular certification and with 20 or more years of experience dropped by about 20 points each. Also, due to both the short-term com-mitments of some alternatively certified teachers and school autonomy over personnel, the teacher turnover rate nearly

doubled. The fact that such large improvements in student learning could be achieved with these common metrics going in the wrong direction reinforces a common finding in education research: teacher credentials and turnover are not always good barometers of effectiveness.

Finally, we turn to a topic that is not typically thought of as part of the reform package but may be an essential com-ponent: costs and resources. Our analysis suggests that from 200405 to 201112, the same years covered by our achieve-ment analysis, total public schooling expenditures per student increased by $1,000 in New Orleans relative to other districts in the state. Some of the increase probably reflects one-time start-up costs of new schools, and we are working to understand what share falls in that category. Regardless, there is wide agreement that the reforms did not come cheap.

None of this really tells Relative to the state as a whole, the New Orleans high school graduation rate rose 10 percentage points after the New Orleans reforms. PH

OT

OG

RA

PH

/KA

TH

LE

EN

FL

YN

N,T

HE

TIM

ES

-PIC

AY

UN

E,L

AN

DO

V

EMBARGOED

feature

NEW ORLEANS REFORMS HARRIS

educationnext.org F A L L 2 0 1 5 / EDUCATION NEXT 15

us exactly which of the factors drove the improvements in student outcomesno doubt they are interconnectedbut it does provide some indication of how schools and families responded to the policy shift.

Implications for New OrleansThese findings have important implications for the New

Orleans public schools, the many other urban districts pur-suing the portfolio approach, and for the state and federal policiesespecially test-based and market-based account-abilityfrom which the New Orleans reforms emerged.

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

The effects are also large compared with other completely different strategies for school improvement, such as class-size reduction and intensive preschool. This seems true even after we account for the higher costs. While it might seem hard to compare such different strategies, the heart of the larger school-reform debate is between systemic reforms like the portfolio model and resource-oriented strategies.

With the possible exception of distortions from test-based accountability, which are harder to identify, the reforms man-aged to avoid most of the side effects that many feared. But our findings also suggest areas of potential improvement. While the reforms have been successful on some dimensions of equity, it seems necessary to do more to ensure that all groups within the city benefit. All types of public school systems struggle with providing equitable access to quality schools, and the New Orleans system is no exception.

Implications for the NationUnfortunately, the effects of even the most successful pro-

grams are often not replicated when tried elsewhere, and there are good reasons to think the conditions were especially ripe for success in New Orleans:

There was nowhere to go but up. Pre-Katrina, the New Orleans public school system was highly dysfunctional, and

student test scores made it the second-lowest-ranked district in the second-lowest-ranked state in the country.

New Orleans is an attractive city for young educators. The national response to the hurricane aftermath was heartening, and for many young people, contributing to the rebuilding effort became a calling. Later, as the reform effort took hold, New Orleans also became the nations epicenter of school reform, an ideal place for aspiring reform-minded educators. Because the city is smaller than many urban districts, school leaders could be very selective in choosing from the pool of educators who wanted to come and work there.

The effects might also be smaller, at least in the short run, if the reforms were adopted on a statewide basis, because the reform is dependent on a specific supply of teachers. It seems difficult enough attracting effective teachers and leaders to work long hours at modest salaries in New Orleans; doing it throughout Louisiana is unrealistic without a major change in the educator labor market. Nonetheless, it would be a mistake to dismiss the relevance of the New Orleans experience for others. It is relevant precisely because it is so unusual. The citys reforms force us to question basic assumptions about what K12 publicly funded education can and should look like.

There is more to the debate than we can cover here, including fundamental philosophical issues about whose objectives and values should count in making schooling decisions. But there is also wide agreement that the academic outcomes considered here are important, so learning how much the reforms contrib-ute to changes in academic measures should also be a key part of the conversation. Better understanding of all the elements of the reforms is something we owe to the city, its children, and everyone who suffered and perished in this terrible tragedy.

Douglas N. Harris is professor of economics at Tulane University and founder and director of the Education Research Alliance for New Orleans. The research cited here is coauthored with others on the ERA-New Orleans research staff (Paula Arce-Trigatti, Nathan Barrett, Lindsay Bell Weixler, Christian Buerger, Matthew Larsen, Jane Arnold Lincove, Whitney Ruble, Robert Santillano, and Jon Valant) and members of the ERA-New Orleans National Research Team (Huriya Jabbar, Jennifer Jennings, Spiro Maroulis, Katharine Strunk, Patrick Wolf, and Ron Zimmer). All errors are the authors.

The effects of the reforms in New Orleans are large compared with other completely different strategies for school improvement, such as class-size reduction and intensive preschool.

EMBARGOED

_GoBack