Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes for Research / University of Oregon Ben Clarke, PhD Pacific Institutes for Research Hank Fien, PhD University of Oregon Keith Smolkowski, PhD Oregon Research Institute Chris Doabler, PhD Pacific Institutes for Research David Chard, PhD Southern Methodist University IES Conference June 2010
Transcript
Early Learning in Mathematics (ELM)The Efficacy of a Kindergarten Curriculum
Implemented in Whole Classroom Settings
Scott K. Baker, PhDPacific Institutes for Research / University of Oregon
Ben Clarke, PhDPacific Institutes for Research
Hank Fien, PhDUniversity of Oregon
Keith Smolkowski, PhDOregon Research Institute
Chris Doabler, PhDPacific Institutes for Research
David Chard, PhDSouthern Methodist University
IES Conference June 2010
Acknowledgements
Institute of Education SciencesThe research reported here was supported by the Institute of
Education Sciences, U.S. Department of Education, through Goal 2 development grant, #R305K040081, and a Goal 3 efficacy grant, #R305A080114, to Pacific Institutes for Research. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
Additional Oregon Project StaffKathy Jungjohann / Karen Davis: Curriculum developmentMari Strand Cary / Rhonda Griffiths: Coordination and research Chris Doabler: Observation measurement and research
Early Learning in Mathematics (ELM)
• 4-year randomized efficacy control trial• Measuring the efficacy of a kindergarten
mathematics curriculum in Oregon and Texas.
• Research Design• Randomized Controlled Block Design• Classrooms within school matched on full / half
day and randomly assigned to treatment (ELM) or control conditions
3
Purpose of the 4-year project• Study 1: Efficacy trial of the whole group
curriculum (ELM) on kindergarten students’ mathematics achievement
• Study 2: Efficacy trial of small group ELM component – Roots – on the achievement of at-risk students
• Examine potential mediation variables, dose-response variables, and moderation factors
Structure of the Curriculum
• Daily Calendar Lessons / Activities– 15 minutes daily, whole class “circle” time– Monthly booklets with objectives and
application activities
• 120 Core Lessons divided into 4 quarters– 30 minutes whole class instruction– 15 minutes teacher directed written work– End of quarter assessment of progress
ELM Instructional Content• National Math Advisory Panel (2008) recommends
a focused, coherent progression of mathematics learning with emphasis on proficiency with key topics
• ELM focuses on key strands rather than a broad array of mathematical content– Numbers and Operations– Geometry– Measurement– Vocabulary (NCTM Process Standard, 2000)
NCTM Curriculum Focal Points for K (2006)
ELM Conceptual Framework
Development of Mathematical
Concepts / Models
Mathematics-relatedVocabulary and
Discourse
Mathematics-relatedVocabulary and
Discourse
Procedural Fluency
Research questions• What is the immediate impact of ELM taught in general
education kindergarten classrooms on mathematics achievement compared to standard district practice?
• Is impact moderated by student level of risk for mathematics difficulties?
• Does rate of teacher models or student practice opportunities mediate a condition effect?
• Is there evidence of an interaction between condition and student practice on student outcomes?
Study Sample• Assignment at the classroom level blocked on
school– Districts = 3– Schools = 24
• Intervention classrooms = 34 • Control classrooms = 30
• Students nested in classrooms– Whole class instruction– N = 1,349
Student Demographics
• 56.3% eligible for free or reduced lunch• 38.4% English Learners• 8.4% special education
• 49.5% White• 36.4% Latino• 4.8% Asian / Pacific Islander• 2.3% African American
Intervention Mediators Proximal Outcomes
DistalOutcomes
Hypothesized Model of ELM Impact
Descriptive data on implementation
• Classroom fidelity observations – treatment and control
• Classroom observations focusing on instructional interactions – treatment and control
• Teacher logs addressing content coverage – treatment and control classrooms
Implementation Fidelity
• General ratings (8 items)– Models skills/concepts appropriately and with
ease– Engages students in learning throughout the
lesson
• Uses ELM / completes all lesson activities (dichotomous)
• For each ELM activity (range 1-7 per lesson): Full (2) / Partial (1) / Not Taught (0)
Implementation Fidelity Data: ELM Lesson Activities
• 81 ELM (fidelity) observations during the year• Fall : mean = 1.71 (SD = .19)• Winter: mean = 1.65 (SD = .33)• Spring: mean = 1.62 (SD = .43)• Overall:mean = 1.65 (SD = .36) (83% of Full)• 2 of 81 lessons had a mean below Partial (1)
level of implementation
Student measures of impact
• Test of Early Mathematics Ability (TEMA)• Early Numeracy – CBM
– Oral Counting– Number Identification– Quantity Discrimination– Missing Number
Method and Analysis Framework
• Competing curricula– All students received instruction– Time balanced across conditions
• Sample– At risk (some or high risk)
• < 40th percentile• 66% of student sample
– No risk • ≥ 40th percentile• 34% of student sample
Nested Time × Condition Analysis• Outcome: net differences from pre to post• Nested students within classrooms
– Control for nonindependence (e.g., ICCs)– Controls for teacher effects
• Maximum likelihood (restricted)– Includes all cases with data at either T1 or T2– Reduces bias from missing data
• Moderation added Time × Risk × Condition interaction
• Effect sizes: Hedges’ g
Sample Means, SDs, and NsT No Risk Some Risk
ELM Control ELM Control
TEMA Raw T1 29.0 (6.8) 28.8 (7.3) 14.1 (6.0) 14.6 (6.2)
T2 190 174 341 312Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms.
• Test of Condition– t = 1.99– df = 61– p = .0509– ES = +0.14
• T1 differences were not statistically significant (t = 0.60)
TEMA Raw ScoresCondition by Risk Status
• Main Effects– Difference in gains: 1.32– t = 2.41, df = 61, p = .0190
• Condition by Risk Status– t = 2.47, df = 61, p = .0162– No Risk ≥ 40th %tile
• Difference in gains: 0.04• t = 0.51, df = 61, p = .9586
– Risk < 40th %tile• Difference in gains: 1.98• t = 3.29, df = 61, p = .0017
CBM Total ScoresCondition by Risk Status
• Main Effects– Difference in gains: 7.67– t = 1.99, df = 61, p = .0509
• Condition by Risk Status– t = 2.24, df = 61, p = .0289– No Risk ≥ 40th %tile
• Difference in gains: -0.27• t = -0.05, df = 61, p = .9570
– Risk < 40th %tile• Difference in gains: 10.81• t = 2.54, df = 61, p = .0138
Effect Sizes (Hedges’ g)
Measure Not At Risk At Risk
Tema Raw Score +0.006 +0.242**
EN-CBM Total +0.014 +0.215*
*p < .05; **p < .01
Summary• ELM classrooms outperformed controls
– TEMA raw and percentile scores– EN-CBM Total
• Students at risk– Improve on all measures more than no-risk students– Control students at risk catching up on no-risk students
• TEMA: 14.0 percentile gain on no-risk students• EN-CBM: 9.6 point gain on no-risk students
– ELM students at risk catching no-risk students faster• TEMA: 18.6 percentile gain on no-risk students• EN-CBM: 20.63 point gain on no-risk students
• No condition effects for students with no risk
Intervention Mediators Proximal Outcomes
DistalOutcomes
Preliminary Analysis of Association between Observation Data and Student Outcomes
Coding of Academic Teacher-Student Interactions (CATS) Observation
Instrument CATS uses a frequency count approach to measure
teacher-student instructional interactionsObservers code behavior occurrences in a continual, serial fashion.Utilizes a strict coding structure
CATS based on evidence of effective instruction in early literacy and beginning mathematics, and adapted from the STICO observation instrument (Smolkowski & Gunn, 2010)
The Role of Teacher Modeling and Student Practice in Student
Outcomes
Preliminary Mediation Analysis
• Does rate of teacher models or student practice opportunities mediate condition effect?
• Rates of (a) teacher models, (b) student group practice opportunity and (c) individual student practice opportunities entered as mediators to determine if they decreased condition effect– Condition effect was still significant– No evidence to support this mediation hypothesis
Secondary Analysis
• If student practice overall is not mediating impact, perhaps the value (quality) of practice differs by classroom and is related to condition
Interaction between Rate of Practice and Condition
• By condition do rates of practice opportunities show the same pattern of impact on student outcomes?
• Are treatment – control differences on student outcomes similar in classrooms with high rates of practice vs. low rates of practice?
Interaction between Rate of Practice and Condition
• Number of Classrooms by Treatment Condition and Median Rate of Individual and Group Practice Opportunities
• Practice Opportunity Quartiles in Rate per Minute
Control ELM Total
Above Median
6 26 32
Below Median
24 8 32
Total 30 34 64
Control ELM Total
Minimum 0.4 1.0 0.4
25th %ile 0.8 1.9 1.2
Median 1.3 2.3 1.9
75th %ile 1.7 3.0 2.4
Maximum 4.1 4.1 4.1
TEMA Scores by Rate of Practice
• Within ELM condition– High-practice classrooms outperform low-practice classrooms– Difference = 2.59, t = 2.58, df = 29, p = 0.0151
• Within control condition– No difference between high- and low-practice classrooms– Difference = 0.39, t = 0.31, df = 29, p = .7575
• Within classrooms with an above-median practice rate– ELM classrooms (might) outperform control classrooms– Difference = 2.33, t = 1.85, df = 29, p = .0747
• Within classrooms with a below-median practice rate– No difference between ELM and control classrooms– Difference = 0.14, t = 0.14, df = 29, p = .8935
EN-CBM Scores by Rate of Practice
• Within ELM condition– No difference between high- and low-practice classrooms– Difference = 9.83, t = 1.46, df = 29, p = 0.1538
• Within control condition– No difference between high- and low-practice classrooms– Difference = -7.59, t = -0.92, df = 29, p = .3635
• Within classrooms with an above-median practice rate– ELM classrooms outperform control classrooms– Difference = 19.37, t = 2.35, df = 29, p = .0257
• Within classrooms with a below-median practice rate– No difference between ELM and control classrooms– Difference = 1.95, t = 0.29, df = 29, p = .7731
Next steps
• Have just completed implementation of Study 1 in Dallas, Texas
• Have just completed implementation of Study 2 in Oregon
• Will implement Study 2 in Dallas, Texas in 2010-11
• Ongoing analysis to investigate impact of condition and mediation and moderation variables associated with impact
