Results from a Randomized Trial of Two Algebra Sequences for Underprepared Freshmen

Ruth Curran NeildVaughan ByrnesRobert Balfanz

Motivation for the Study• States and districts increasingly

consider Algebra 1 to be the “default” course for all 9th graders

• AND YET – • In districts with “Algebra for All” policies,

course failure rates in Algebra have been high (Los Angeles, Milwaukee, Chicago)

Large percentages of ninth graders in low-performing districts are underprepared for Algebra

(Data from Philadelphia, adapted from Neild and Balfanz, 2005)

Where is “catch up” required?

• Intermediate mathematics skills: fractions, decimals, signed numbers

Complete the following statement with >, <, or =.56 _____ ¾

Where is “catch up” required?

• Pre-algebra skills, including:– Graphing– One- and two-step equations– Inequalities

• Mathematics reasoning skills– Does this answer/argument make sense? –What are the patterns?

“Double dosing” is a common ninth grade “catching up” strategy

Options for extended time• Provide an additional 45-50 minute

Algebra support course during the school day (Philadelphia, Chicago)

• Place students into a year-long Algebra course that meets ~90 minutes each day and allows time for catch up as needed

• Provide a structured catch-up course first semester, followed by Algebra 1 second semester (class meets ~90 minutes each day)

The ConditionsCatch Up Course Followed

by Algebra 1 Course• Year-long 80-90 minute

classes, taught by the same teacher

• First semester course is the Transition to Advanced Mathematics Curriculum, developed by Johns Hopkins

• Second semester is Algebra curriculum of the district’s choosing

Year-Long Algebra with Embedded Catch Up

• Year-long 80-90 minute class, taught by the same teacher

• Algebra curriculum is of the district’s choosing

• Conceptualized as the “business as usual” condition

Transition to Advanced Math (TAM)

Genesis of the curriculum was observation of a quandary: either fail most students or water down the content

Recognizes material and human capital challenges in low-resource schools through student consumables, tightly organized curriculum, and classroom kits of materials.

Transition to Advanced Math (TAM)

Five Units:1. Mathematical Reasoning, Data Analysis,

and Probability2. Numbers and Integers3. Rational Numbers4. Measurement5. Patterns and Functions & Introduction to

Algebra

• Two cohorts of districts: eight in 2008/09, five in 2009/10 (13 district in total) Implementation/data collection in each district for a single year

• Randomization within districts, at the school level

• Students who are 1-4 years below grade level in mathematics are eligible to participate

Study Details

Study districts, 2008/09 and 2009/10

Research QuestionsPrimary questions• Are there mid-year differences between

the conditions in growth in intermediate mathematics skills?

• Are there end-of-year differences between the conditions in Algebra proficiency?

• Are there differences between the conditions in mathematics grades during semester 1 and semester 2?

Instrumentation

ALGEBRA PROFICIENCY (Orleans-Hanna & CTB Algebra)

September End of first semester (~January)

September End of year (~May)

INTERMEDIATE MATH SKILLS (CTBS)

Other Instrumentation• Student and teacher surveys:

Beginning-of-year and end-of-year• Classroom observations, with

quantitative measures and narrative: once each semester

Basic Models

• Multi-level modeling (HLM)• 3-level model with students nested

within teachers, within schools• Uncontrolled (empty) model• Model with student, teacher, and

school controls

• (4,941 Students – 131 Teachers – 46 Schools)

Level ControlsSTUDENT Fall score (CTBS or Orleans-

Hanna)FemaleYear of birthDays absentWhiteHispanicOther

Level ControlsTEACHER # of prior years teaching

mathAny prior block scheduled teachingPrior experience with classroom coachPrior experience with Stretch AlgebraVolunteered to teach 9th grade algebraMajored in mathematicsFully certified

SCHOOL TreatmentStudent-Teacher Ratio% Free/Reduced Lunch

GROWTH IN INTERMEDIATE MATH:

Distribution of fall CTBS national percentiles

LocalPercentile

TAM/Alg 1

Year-Long Algebra

10th 7 725th 28 2350th 36 3675th 47 4790th 58 53Mean 36.1 34.7

Distribution of winter CTBS national percentiles

LocalPercentile

TAM/Algebra 1 Year-Long Algebra

Winter CTBS

Fall-Winter Gain

Winter CTBS

Fall-Winter Gain

10th 21 +14 15 +825th 30 +2 26 +350th 41 +5 38 +275th 52 +5 48 +190th 63 +5 59 +6Mean 41.8 +5.7 37.8 +3.1

Estimated treatment effect, Intermediate Mathematics

Controls Effect

P-value

Level 1 ES

Level 3 ES

Uncontrolled 4.45 .003* .26 .85Student, teacher, school controls

2.80 .001* .16 .53

Direction of Effects is Consistent Across Districts

C D F G E H B K A M L J I0123456789

10

Mean Fall-Winter Change in Mathematics Percentile,

by Condition and District

SEMESTER 1 MATH GRADES:

Estimated treatment effect, Semester 1 Math Grade, 2008/09

Controls Effect

P-value

Level 1 ES

Level 3 ES

Uncontrolled 1.00 .007* .29 .77Student, teacher, school controls

1.12 .000* .33 .86

Distribution of 1st Semester Math Grades

By Condition

A B C D F0%

10%

20%

30%

40%

50%

8%

19%

29%24%

20%

12%

26% 29%

20%13%

ALGEBRA PROFICIENCY:

Distribution of national percentiles

Local%ile

TAM/Algebra 1 Year-Long AlgebraOrleans-Hanna

CTB Algebra Diff

Orleans-Hanna

CTBAlgebra Diff

10th 1 1 0 1 1 025th 1 22 +21 7 22 +1550th 17 35 +18 19 35 +1675th 29 47 +18 32 44 +1290th 43 55 +12 43 55 +12Mean 19.0 34.2 +15.2 20.1 32.9 +12.8

Estimated treatment effect, Algebra Proficiency

Controls Effect

P-value

Level 1 ES

Level 3 ES

Uncontrolled 1.56 .380 .09 .27Student, teacher, school controls

0.38 .772 .02 .06

Direction of Effects Across Districts

L E G C J A K D I M B H F0

5

10

15

20

25

Mean Fall-Spring Change in Algebra Percentile,

by Condition and District

SEMESTER 2 MATH GRADES:

Estimated treatment effect, Semester 2 Math Grade

Controls Effect

P-value

Level 1 ES

Level 3 ES

Uncontrolled .47 .150 .14 .41Student, teacher, school controls

.51 .086 .15 .45

Distribution of 2nd Semester Math Grades

By Condition

A B C D F0%

10%

20%

30%

40%

50%

7%

17%

28% 25% 24%

9%

20%26% 23% 23%

Robust ResultsResults for all achievement outcomes were

consistent when re-tested using several supplementary methods, Including:

• Best-Fit Models for each outcome• Analyses with only complete case data• Analyses with only those targeted students,

1-4 GE behind• Testing for interaction between treatment

effect and prior achievement levels

INTERVENING FACTORS AND INTERMEDIATE OUTCOMES:

CLASSROOM PRACTICES & ATTITUDES TOWARD MATHEMATICS

Measure: Classroom Practices, including:• Students used objects or tools, such as rulers, protractors or

algebra tiles• The teacher asks students to explain how they got their answers.• When I didn't understand something my teacher tried to help by

asking questions• Students are asked to show more than one way of solving a math

problem• I worked on math problems during class time with other students

in my class• I was asked to write a few sentences about how I solved a math

problem• Students worked in small groups or with a partner• My teacher uses real-life examples to help us understand math• The teacher made sure that everyone understood before moving

on to another topic

Measure: Attitudes Toward Math Class, including:• I liked coming to math class• I paid attention in math class• I did my math homework• I felt that I could do almost all the work in

math if I didn’t give up.• I felt confused in math class.• I worked hard in math class.• I studied for math tests and quizzes. • I felt successful in math.• I felt confident that I could do the math work.

Estimated Effect of Classroom Practices & Attitudes on Student Outcomes

PRACTICES ATTITUDESEffect P-Value Effect P-Value

Intermediate Mathematics

0.11 .838 1.64 .000*

Algebra 0.10 .846 3.04 .000*1st Semester Math Mark

0.20 .062 1.22 .000*

2nd Semester Math Mark

0.31 .000* 1.42 .000*

Attitudes 0.33 .000* N/A N/A

Estimated treatment effect on Classroom Practices & Attitudes

PRACTICES Effect

P-value

Level 1 ES

Level 3 ES

Uncontrolled .23 .000* .26 1.10Student, teacher, school controls

.25 .000* .29 1.19

ATTITUDES Effect

P-value

Level 1 ES

Level 3 ES

Uncontrolled .07 .256 .07 .37 Student, teacher, school controls

.10 .085 .11 .53

Finally…• Fall to Winter Gains made in Intermediate

Math were significantly related to spring Algebra scores when modeled

• So, while TAM students may not have performed significantly higher in Spring Algebra, their 1st semester gains in basic math skills were related to improved Algebra performance

Conclusions• TAM students improved significantly more in

terms of their intermediate math skills.• TAM students performed equally to Stretch

students in Algebra, despite spending only half as much of the school year on the topics.

• TAM students experienced higher levels of intermediate outcomes which indirectly lead to better Algebra outcomes (intermediate math gains, and better Classroom Practices).