PEARSON SUCCESS MAKER RESPONSE TO INTERVENTION STUDY · PEARSON SUCCESS MAKER RESPONSE TO...

SuccessMaker RTI 2012-13 Gatti Evaluation Inc. 10-1-13

PEARSON SUCCESS MAKER

RESPONSE TO INTERVENTION STUDY

FINAL REPORT

10-1-2013

This report and its contents, including all graphics, are proprietary information belonging to Gatti Evaluation Inc. No part of this report may be reproduced or used for any purpose without permission from Gatti Evaluation Inc.


Principal Investigator

Guido G. Gatti Gatti Evaluation Inc.

162 Fairfax Rd. Pittsburgh, PA 15221

(888) 300-5530 [email protected]

Primary Stakeholder

Funded By Pearson

For Information Please Contact:

Marcy Baughman Director of Academic and Efficacy Research

(724) 863-1621 [email protected]


TABLE OF CONTENTS

EXECUTIVE SUMMARY

I. INTRODUCTION __________________________________________________1-3

Instructional Technology Literature 1

Study Goals and Research Questions 2

II. METHODOLOGY __________________________________________________4-21

Student Outcome Measures 4

Teacher Measures 7

Site Recruitment and Selection 9

Non-SuccessMaker Intervention Instruction 11

SuccessMaker Program Implementation 11

Participants 18

Data Analysis Procedures 19

III. RESULTS _________________________________________________________22-42

Baseline Group Equivalence 22

Students’ Achievement Gains 27

Student Academic Attitudes 39

Teacher and Student SuccessMaker Opinions 39

IV. DISCUSSION _____________________________________________________ 43-44

A.1 Study Site Descriptions 45-63

SM-RTI 2012-13 RCT Sample Demographic Information

Table 1: SuccessMaker 2012-13 Training Dates 12

Table 2: Digits 2012-13 RCT Sample Demographic Information 15

Table 3: 3rd

Grade Baseline Study Group Scores 23

Table 4: 5th

Grade Baseline Study Group Scores 23

Table 5: 3rd

Grade Baseline Study Group Score Comparisons 24

Table 6: 5th

Grade Baseline Study Group Score Comparisons 24

Table 7: 3rd

Grade Baseline Study Group Scores by SuccessMaker Implementation 25

Level

Table 8: 5th

Grade Baseline Study Group Scores by SuccessMaker Implementation 25

Level

Table 9: 3rd

Grade Baseline Study Group Score Comparisons by SuccessMaker 26


Implementation level

Table 10: 5th

Grade Baseline Study Group Score Comparisons by SuccessMaker 27

Implementation level

Figure 1: 3rd

Grade Group Comparison Achievement Gains 30

Figure 2: 5th

Grade Group Comparison Achievement Gains 34


EXECUTIVE SUMMARY

Gatti Evaluation partnered with Pearson to evaluate the effectiveness of SuccessMaker as a RTI

program. The primary goal of this study was to conduct rigorous research to support the

assertion that the SuccessMaker program increases RTI students’ English language arts and

mathematics achievement over similar non-users that did not regularly use a computer-based

adaptive program. The second goal of the study was to collect information on teacher and

student attitudes toward specific features and aspects of the program.

The research team recruited diverse 3rd

and 5th

grade intervention students from eighteen urban,

suburban, and rural schools districts in six different states (i.e., AZ, CA, KS, MI, OR, and TX).

The study sample included 292 3rd

grade (i.e., SuccessMaker = 154, comparison = 138) and 490

5th

grade (i.e., SuccessMaker = 239, comparison = 251) at-risk students that would benefit from a

well-conceived and implemented reading and math intervention.

Specifically, students were included if they tested below the 30th

percentile in reading fluency

and pacing or either math computation or problem solving. The samples from both grade levels

are largely non-Caucasian and free/reduced lunch eligible. Also, the 5th

grade sample, selected

on math achievement is more female than male. Each qualifying student was randomly assigned

to one of two study groups (i.e., comparison group of non-users v. SuccessMaker users). The 3rd

grade treatment group used the reading version of the program while the 5th

grade SuccessMaker

students used math.

All study students, both SuccessMaker users and comparison group students, still received their

required non-SuccessMaker intervention instruction as mandated by the district. A minority of

the study students from both groups received regular intervention instruction. In addition to

intervention instruction, the students in each study group received similar regular classroom

reading and math instruction. No new reading or math content was introduced when

SuccessMaker students where using the program.

SuccessMaker interventionists were expected to conduct one-to-one sessions where they review

the progress monitoring graphs and incorrect program exercises with each student at least once a

week. Not all interventionist were able to conduct these sessions regularly with their

SuccessMaker students. Sixty percent received at least a single one-to-one session with an

average of six sessions per student.

Four schools had large SuccessMaker user to SuccessMaker interventionist ratios that did not

permit systematic individualized attention; these schools were considered Low Implementers.

Schools that conducted regular one-to-one program intervention sessions and/or had small ratios

(i.e., SMU/SMI < 14), and were able to deliver regular over-the-shoulder attention were

designated the High implementation group. It should be noted that those students receiving non-

SuccessMaker intervention instruction from both grades were from the High implementation

group.

Interventionists and students quickly became comfortable with the SuccessMaker program.

Students indicated they liked the program and all but a small minority found aspects they liked

about the program. A majority of the 3rd

and 5th

grade users liked the animation, graphics and


scaffolding offered by the program, and most found activities and stories from the program they

liked. Most importantly, the majority of the 3rd

and 5th

grade users felt the questions were of

medium difficulty; this of course, is right where the offerings from the program should be.

The overall interventionist response was very positive (i.e., 82% positive comments).

Interventionists appreciated the program’s interactive exercises, formative assessment and

motion, read-to-me audio, immediate feedback coupled with scaffolding and tutorials,

customized coursework capability and very much liked that the activities, in conjunction with the

reporting system, allowed for effective one-to-one intervention instruction. Interventionists did,

however, have issues with the automatic initial placement and the fluency component.

SuccessMaker students used the program in an educational laboratory setting for at least four

twenty to thirty minute sessions each week. SuccessMaker students were productive on the

program with the large majority (i.e., 75%) of students logging at least 28 and 24 hours at 3rd

and

5th

grade, respectively.

Two challenging assessment batteries were administered to students to measure gains in

important areas of achievement. These areas were total reading including comprehension,

vocabulary and word reading scales, as well as, fluency/pacing and reading academic attitude.

Math scales included total math with concepts and communication, operations and computation,

and process and applications (i.e., problem solving), as well as, timed computation and concept

and application (i.e., problem solving) scales and math academic attitude.

The achievement data indicates clearly that diverse, at-risk populations of students can be

successful the first school year they are exposed, and their interventionists implement the

program. SuccessMaker students demonstrated large statistically significant achievement gains

on all achievement outcome measures. This was especially true for students receiving proper

implementation in the traditionally difficult to affect areas of comprehension and problem

solving.

The 3rd

grade SuccessMaker students saw statistically significantly larger gains in their

vocabulary than their comparison group counterparts, while the 5th

grade SuccessMaker students

saw statistically significantly larger gains in computation and problem solving. Further, those 5th

grade SuccessMaker students coupling the program with regular individualized attention from

their interventionist saw statistically significantly larger gains in mathematics achievement

overall, as well as, on concepts, computation and problem solving scales.

Lastly, the study groups saw statistically equivalent gains in their academic attitude with the

exception of the 5th

grade Low implementation group which had statistically lower gains than

their comparison group counterparts.

This summary and its content are proprietary information belonging to Gatti Evaluation Inc.

- 1 -

I. INTRODUCTION

As elementary schools strive to respond to their reading and math intervention (RTI) students,

many are attempting to maximize their efforts by turning to instructional technology like the

SuccessMaker©1 program. Gatti Evaluation partnered with Pearson to evaluate the effectiveness

of SuccessMaker as a RTI program. Information gathered during this study will inform future

revisions of the program and provide evidence of program efficacy. Such evidence has become a

necessity as state adoption committees, independent watchdog groups (i.e., Best Evidence

Encyclopedia, What Works Clearinghouse), and the federal government (i.e., No Child Left

Behind Act2) require publishers to conduct rigorous efficacy research to support their educational

materials.

Pearson partnered with Gatti Evaluation to study the efficacy of the

SuccessMaker program in achieving positive educational attitudes and

achievement outcomes with RTI students.

This report provides methods and results from the RTI phase of the efficacy research conducted

during the 2012-13 school year on the SuccessMaker Reading and Mathematics program. This

report includes study methodology, nuanced program usage information, interventionist and

administrator attitudes, as well as student attitudinal and achievement gains. This study

evaluated the program with RTI students in eighteen school districts from six different states

(i.e., AZ, CA, KS, MI, OR, TX).

Instructional Technology Literature

SuccessMaker is an adaptive, computer-based learning program that offers an

instructional management system, placement and formative assessment,

individualized elementary and middle grades reading and mathematics

curriculum resources, and a student progress reporting system.

SuccessMaker is an adaptive, computer-based learning program that offers an instructional

management system, placement and formative assessment, individualized elementary and middle

grades reading and mathematics curriculum resources, and a reporting system to inform

administrators and teachers as to student progress.

It is widely believed that making formative assessment an integral part of instructional practice is

one of the best ways to improve student learning, this can be especially true for at-risk students.3

Further, creative technology interventions show promise as a means to reach RTI students.4

Instruction may be aided by technology in various ways, with the technology assuming the role

1 http://www.pearsondigital.com/ 2 http://www.ed.gov/nclb/landing.jhtml 3 National Council of Teaching of Mathematics (2000). Principles and Standards for School Mathematics. Reston, Va.: National Council of

Teachers of Mathematics. 4 Cheung, A., & Slavin, R. E. (2011, May). The effectiveness of educational technology for enhancing reading achievement: A meta-analysis. Baltimore, MD: Johns Hopkins University, Center for Research and Reform in Education.


- 2 -

of enhancing, amplifying, and organizing curricula.5 It is also well-documented that both the

scope of ways and effectiveness of technology in aiding instruction is increasing with each

passing decade.6 What remains unclear are the best ways to utilize technology to find significant

improvement in student achievement over non-technology methods that make use of the same

pedagogy.

Theoretically, well-designed interventions are expected to increase student achievement.

Although an intervention may be skillfully applied to create an educational environment that

significantly increases achievement, poorly designed and implemented interventions will provide

little or no benefit, and may even be detrimental. Poorly designed and implemented curricula

can confuse and frustrate students and teachers, proving to be a waste of money and valuable

learning time. For these reasons, state adoption committees and the federal government (i.e., No

Child Left Behind Act7) require publishers to conduct rigorous efficacy research to support their

educational materials.

Study Goals and Research Questions

The primary goal of this study was to conduct rigorous research to support the assertion that the

SuccessMaker program increases RTI students’ English language arts and mathematics

achievement, specifically vocabulary, comprehension, fluency, computational skills, and

process/application as well as academic attitudes. The achievement and attitudes of program

users was compared against non-users that did not regularly use a computer-based adaptive

program. All study students still received required intervention reading and math instruction as

mandated by the district.

Students testing as lower achieving in reading fluency and pacing or either math computation

and/or problem solving (i.e., below the 30th

percentile) were randomly assigned to study groups

from within each participating school. Further, this study tested the SuccessMaker program

during its first year of implementation, the most challenging year for any new program to impact

student achievement.

The second goal of the study was to collect information on teacher and student attitudes toward

specific features and aspects of the SuccessMaker program. Specifically, how do teachers and

students respond to the program, and how is the program being used?

The research questions for this study are outlined in the following four parts:

RQ1: Do RTI students using the SuccessMaker program demonstrate a significant improvement

in achievement over otherwise similar students in classrooms supplementing reading and math

intervention instruction without using an adaptive, computer-based program like SuccessMaker?

5 Heid, M. K. (1997). The technological revolution and the reform of school mathematics. American Journal of Education, 106(1), p5-61. 6 Jenks, M. S., & Springer, J. M. (2001). A view of the research on the efficacy of CAI. Electronic Journal for the Integration of Technology in

Education, 1(2). 7 http://www.ed.gov/nclb/landing.jhtml


- 3 -

RQ2: Do RTI students using the SuccessMaker program demonstrate more positive attitudes

toward reading/math and reading/math instruction when compared to their non-SuccessMaker

counterparts?

RQ3: How was the SuccessMaker program implemented?

RQ4: How did teachers and students react to the SuccessMaker program?


- 4 -

II. METHODOLOGY

The SuccessMaker RTI efficacy study employed a two-group, randomized

design. SuccessMaker students received the program for reading (i.e., 3rd

grade)

and math instruction (i.e., 5th

grade) during this initial school year of exposure

and implementation, while comparison students received intervention instruction

from those familiar materials and methods preferred by their intervention

teachers.

The program was evaluated via a two-group randomized, baseline to post observation

assessment, research design. Two grade levels participated, 3rd

and 5th

. The potential study

sample included those students receiving regular instruction with their classmates and testing as

lower achieving (i.e., below the 30th

percentile) on the AIMSweb R-CBM (i.e., median words

read correctly in one minute), or at 5th

grade, either the AIMSweb MCOMP (i.e., math

computation) or MCAP (i.e., concepts and applications). Selected students from each

participating school were randomly assigned to one of the two study conditions (i.e., comparison

or SuccessMaker) prior to the start of the study.

The 3rd

grade SuccessMaker group made regular use of the reading version of the program as

part of their reading intervention instruction while the 5th

graders used the math version with

their interventionists. Students from both study groups continued to receive similar basal and

intervention instruction from district adopted programs, and those materials and methods familiar

to students and preferred by their classroom teachers.

Gatti Evaluation provided participating schools all data collection materials, maintained constant

communication with study participants, and followed clear data collection procedures throughout

the study to ensure that both study and program implementation ran smoothly and effectively.

Appendix 1 has descriptions of each study school.

The following sections provide information on study procedures, including; student and teacher

level data collection, site recruitment and selection, the nature of classroom and intervention

instruction at the study sites, program training and implementation, demographic information for

study participants, and the statistical methodologies used to analyze outcomes.

Student Outcome Measures

A challenging assessment battery was group administered to students to measure

achievement and academic attitude growth during the school year.

At each school potential participating RTI students were tested with the AIMSweb benchmark

probes and those scoring below the 30th

percentile were selected to participate in the study.

Third grade students were tested on the AIMSweb R-CBM (i.e., median words read correctly in

one minute) and 5th

grade students were tested on the AIMSweb MCOMP (i.e., computation) and


- 5 -

MCAP (i.e., concepts and applications). The 5th

grade students were selected if they scored

below the 30th

percentile on either the MCOMP or MCAP scale.

When participating students were selected, the remaining assessment battery was administered to

selected students. This battery was comprised of the Group Reading/Mathematics Assessment

and Diagnostic Evaluation (GM/RADE) and an academic attitude survey was used to measure

gains in student achievement and attitude over the course of the school year. The assessment

battery was intended to challenge the students; attempting to adequately assess baseline

knowledge, while also providing room for growth as knowledge is acquired during the school

year. The GMADE, GRADE and academic attitude surveys were group-administered by the

classroom teachers at the beginning (i.e., corresponding to initial training) and end of school year

(i.e., within four weeks of end of school).

Group Mathematics/Reading Assessment and Diagnostic Evaluation The GMADE and GRADE are standardized, nationally norm-referenced achievement tests,

published by Pearson Assessments. They were constructed with all fifty states’ standards in

mind, covering a wide range of content topics and skills. They include nine levels that span

grades K-12, each with two parallel forms. Form A was administered at baseline and form B

was administered at the end of the school year. The GMADE and GRADE are not timed tests,

but generally take an estimated 105 and 60 minutes to administer, respectively. Schools returned

completed student tests to the research team for hand-scoring.

Overall and subtest scores were reported for both the GMADE (82 questions) and GRADE (107

questions). The subtest scores allowed the research team to evaluate the effectiveness of the

program on important dimensions of reading and mathematics. The GMADE subtests are

Concepts and Communication (28 questions), Operations and Computation (24 questions), and

Process and Applications (30 questions). These subtests address students’ knowledge of

mathematics representations and language, use of basic computational algorithms and operations,

and the ability to solve problems presented in written form, respectively.

The GRADE subtests include a Vocabulary section comprised of two subtests, Word Reading

(30 questions) and Vocabulary (30 questions), and the Comprehension section is also broken

down into Sentence Comprehension (19 questions) and Passage Comprehension (28 questions).

Listening Comprehension is not included in the total GRADE score and is not reported. Total

scores from the GRADE and GMADE have been found to have intraclass reliability in excess of

0.90 and in the neighborhood of 0.80 for subtest scores.

Academic Attitude Surveys and SuccessMaker Opinion Survey The math and reading academic attitude surveys were developed by the Gatti Evaluation

principal investigator. Students responded to sixteen self-report questions regarding general

attitude, confidence, motivation, and self-perceived aptitude. Student responses were coded as 1

for a positive response, 0 for a neutral response, and -1 for a negative response. This scoring

method anchors a completely neutral student at an overall score of zero with positive total scores

indicating an overall positive attitude.

Lastly, students using the SuccessMaker program were surveyed online as to their opinions on

several aspects of the program in the later part of the school year. Specifically, the survey asked

students for their general opinions of the program, as well as how they liked specific features of


- 6 -

the graphics and animation, question difficulty and scaffolding, and reading passages and

activities.

AIMSweb R-CBM, MCOMP & MCAP The AIMSweb Reading Curriculum-Based Measurement is a test of reading fluency and pacing

published by NCS Pearson. The outcome of interest on the AIMSweb is the number of words a

student can read correctly in one minute starting from the beginning of an appropriately leveled

pre-determined passage. Each student must read three different passages of which the middle

score is recorded.

The AIMSweb has two separate math scales; Math Computation (MCOMP) and Concepts and

Applications (MCAP). Both scales are strictly timed, lasting eight minutes for 39 items on the

MCOMP and 30 questions on the MCAP. The MCOMP is, as would be expected, a test of

computation and algorithmic skill. The MCAP consists of 30 word problems. All questions are

not scaled equally, the MCOMP scale scores range from 0 to a maximum of 76 points and the

MCAP scale scores range from 0 to a maximum of 51 points.

Reliability The estimated intraclass reliability from the study sample for the GRADE and GMADE total

scores tested highly reliable for this population. As would be expected, the subtests tested less

reliable, but reliable enough for summary statistics. Also, the baseline (BOY) scores tended to

be less reliable than the end-of-year (EOY) scores as the content of the test was initially difficult

for this sample of students. The estimated intraclass reliability for the academic attitude scores

tested as less reliable, but again reliable enough for reporting summary statistics. Lastly, the 3rd

grade AIMSweb R-CBM scores tested as highly reliable. It should be noted that accurate

reliability statistics are difficult to calculate for the AIMSweb MCAP and MCOMP because

these tests are both strictly timed, with no students completing them in the allotted time, and the

total score is a weighted function of those items answered correctly.

3rd

Grade GRADE Reliability1 BOY EOY

GRADE Total 0.92 0.93

Word Reading 0.81 0.82

Sentence Comprehension 0.82 0.80

Vocabulary 0.82 0.83

Passage Comprehension 0.69 0.79

AIMSweb R-CBM (WRC) 0.86 0.93

Reading Academic Attitude Survey 0.58 0.72

1. Sample estimated coefficient alpha intraclass reliability.


- 7 -

5th

Grade GMADE Reliability1 BOY EOY

GMADE Total 0.82 0.90

Concepts and Communication 0.71 0.77

Operations and Computation 0.69 0.81

Process and Applications 0.56 0.76

MCOMP N/A N/A

MCAP N/A N/A

Math Academic Attitude Survey 0.71 0.76

1. Sample estimated coefficient alpha intraclass reliability.

The research team collected achievement, attitudinal, as well as, observational

and self-report data making the study both quantitative and qualitative in nature.

Teacher Measures

In addition to the assessment battery, qualitative data collection methods were also employed.

The research team collected qualitative data through self-report interventionist logs and

classroom observations, as well as interviews and focus groups. The data was compiled and

content analyzed to examine interventionists’ attitudes, pedagogy and program implementation,

as well as to illuminate the various ways SuccessMaker interventionists and students interact

with the program. This data also increased the validity of the research findings by verifying

results through multiple data collection methods, by adding context to the achievement results

through reporting the perspectives of various study participants, and by collecting data

throughout the project period. Continuous monitoring of the study sites was of immense

importance, and interventionists were routinely asked to share their opinions and concerns

throughout the school year.

Weekly Intervention Logs All interventionists were required to complete weekly logs in which they described their

intervention instruction including the integration of SuccessMaker. Information from the weekly

logs was important for two reasons; to guarantee interventionists and students fully and regularly

utilized all key components of the program in an attempt to positively influence student

achievement, and to document the intervention instructional model utilized by each school and

received by each student, including sessions, minutes, content and methods. The information in

these logs was checked each week, and the project manager asked for clarification when

necessary.

Teacher Observations Each study school was observed three times during the school year. Site visits took place

between late October and early December, again between January and March, and lastly during

April. Classroom observations were conducted by representatives of the research team. Study


- 8 -

students were observed in both their classrooms, receiving math or reading instruction, and

receiving intervention instruction, including using the SuccessMaker program.

Observers took extensive note and completed observation instruments for each classroom lesson

and/or lab session observed. Portions of the observation forms included; a description of the

classroom environment, summary of the lesson taught, teacher interviews, student comments,

observed teaching strengths and weaknesses, pacing, and supplemental instruction information.

The observations also allowed researchers to observe general classroom/lab environment for the

SuccessMaker students and to verify the ability and willingness of SuccessMaker interventionists

to properly implement the program.

It should be noted that a handful of observations show just a snapshot of the classroom

environment, instructional competence, and implementation. Some teachers/interventionists

were required to change their normal class time due to scheduling conflicts, which occasionally

resulted in the observer having less than optimal time to spend in the classroom. The

observations are, however, worthwhile because they are the only opportunity the research team

has to directly observe the study teachers/interventionists in action and verify self-reported

information. It should also be noted that the SuccessMaker trainers had opportunities to observe

interventionists using the program during the follow-up training visits. Interventionists that

missed training sessions, or that were generally perceived as struggling with their

implementation were prioritized for observation and/or additional training visits so they could be

given support and assistance.

SuccessMaker Interventionist Focus Group Focus groups were executed by the research team to ascertain attitudes toward the SuccessMaker

program. The nature of a focus group, though more labor intensive, can be superior to simple

questionnaires in collecting detailed attitudinal information from participants. When properly

conducted, the focus group discussion gravitates to those topics most important to the

participants, and can provide more nuanced information.

The focus group results describe what teachers and students liked about the

SuccessMaker program, how the program could be improved, and how teachers

are using specific features of the program.

All SuccessMaker interventionists first received the focus group questions so they could digest

them and respond with their initial thoughts prior to their WebEx sessions. Representatives from

the research team facilitated each session. The sessions lasted approximately 60 minutes. Of the

24 SuccessMaker interventionists half attended one of four WebEx sessions and all returned their

responses to the initial questionnaire.

These sessions provided a forum for interventionists and administrators to respond to specific

questions about the program, as well as express their professional and personal opinions about

the program. Each session held the teachers’ comfort level as a high priority. The teachers were

encouraged to speak without hesitation or inhibition, and to be as honest and candid as possible.

Though the facilitator followed a structured interview format, the teachers were allowed to direct

the discussion and provide their reactions to, and comment on, any and all aspects of the

program. The focus group sessions provided extensive insight into teacher and student


- 9 -

experiences with, and attitudes about, the program. This information was supplemented with

opinions informally shared by students during the observations.

Each session was recorded and then transcribed verbatim, allowing the research team to compile

a large master file of participant responses. Following an exhaustive review of the teacher

responses, a two-dimensional coding system was developed to organize the responses.

Responses were categorized by Topic Area and Attitude. Topic area codes have a two digit

numeric format, with the first digit on the left indicating general topic category and the

remaining digits indicating a specific topic within each general category. The topic codes are

categorized by grade level and study site; then they are paired with either an ‘N’ to indicate

neutral, a ‘+’ to indicate positive, or a ‘–‘ to indicate a negative attitude toward an aspect of the

program, or to indicate the tone of the comment.

Site Recruitment and Selection

Prior to the 2012-13 school year, potential research schools were identified by Pearson sales

representatives and via email blasts sent to districts with specific demographics. Schools that

indicated interest were sent a study description that included responsibilities and incentives.

Possible research schools were further vetted through local sales representatives. If the school

indicated interest after reviewing the study description and being approved by the sales

representative, they were asked to complete a detailed questionnaire and an infrastructure

checklist.

The intent of the questionnaire was to ensure participants understood all the requirements and

benefits associated with participation. It was required that participating schools not currently

make regular use of an interactive digital intervention program in 3rd

grade reading or 5th

grade

math intervention instruction, abide by the random assignment, and ensure that SuccessMaker

interventionists fully implement the program with their students assigned to use it. The purpose

of the infrastructure checklist was to ensure that the program could be installed and successfully

run at each site.

When sites were deemed eligible for participation and approved by the Principal Investigator, the

school was invited to be a study participant. The Principal Investigator then completed the

research application process necessary for each site. Finally, both a district level administrator

(ex., curriculum director, superintendent) and a school level administrator (ex., principal) signed

a memorandum of understanding outlining the responsibilities of each stakeholder. No available

and eligible students of any socio-economic level or ethnic background, who opted to participate

in the study, were excluded from the study. The research team adhered to the informed consent

requirements of each participating school and/or district.

Ultimately, eighteen schools from eleven urban, suburban and rural school districts in six

different states (i.e., AZ, CA, KS, MI, OR, and TX) were recruited. Appendix 1 provides details

about the community, educational environment and demographic breakdown for each study site.

This information is crucial for determining how applicable results from this study may be to the

consumers of this report. Ethnic and socio-economic diversity in the study sample were two

criteria the evaluation team considered when recruiting study sites.


- 10 -

3rd

Grade SuccessMaker comparison

Intervention

Reading

Instruction

Hours

N = 35 (154)

Ave = 30 (7)

Min = 1

P50 = 22

Max = 72

Sessions

N = 35 (154)

Ave = 60 (5)

Min = 2

P50 = 44

Max = 108

Hours

N = 50 (138)

Ave = 38 (12)

Min = 2.5

P50 = 35

Max = 74

Sessions

N = 50 (138)

Ave = 64 (14)

Min = 43

P50 = 46

Max = 147

SuccessMaker

Usage

Hours

N = 154

Ave = 34

Min = 18

P25 = 28

P50 = 32

P75 = 40

Max = 59

1 on 1 Sessions

N = 94 (154)

Ave = 10 (6)

Min = 1

P50 = 9

Max = 26

N indicates the number of students in a grouping receiving intervention instruction or one-to-one intervention instruction using the program. The parentheses

indicate the number of students for the entire sample.

The sample average in parentheses is calculated including the entire sample of students.

5th


Intervention

Reading

Instruction

Hours

N = 14 (239)

Ave = 46 (2)

Min = 12

P50 = 41

Max = 66

Sessions

N = 14 (239)

Ave = 65 (2)

Min = 49

P50 = 69

Max =81

Hours

N = 32 (251)

Ave = 26 (3)

Min = 5

P50 = 20

Max = 64

Sessions

N = 32 (251)

Ave = 46 (5)

Min = 10

P50 = 40

Max = 83

SuccessMaker

Usage

Hours

N = 239

Ave = 31

Min = 10

P25 = 24

P50 = 28

P75 = 35

Max = 58

1 on 1 Sessions

N = 141 (239)

Ave = 8 (5)

Min = 1

P50 = 7

Max = 32

N indicates the number of students in a grouping receiving intervention instruction or one-to-one intervention instruction using the program. The parentheses

indicate the number of students for the entire sample.

The sample average in parentheses is calculated including the entire sample of students.


- 11 -

Non-SuccessMaker Intervention Instruction

Intervention Instruction

The research team requested that each study site detail all reading and math intervention services

(ex., pull-out/push-in, Title I, before/after school tutoring, etc.) that were made available

throughout the entire school year to the 3rd

and 5th

grade study participants. It should be noted

that all study students, both SuccessMaker users and comparison group students, still received

their required non-SuccessMaker intervention instruction as mandated by the district.

A minority of the 3rd

grade study students received intervention reading instruction, 23% (i.e.,

35/154) of the SuccessMaker group and 36% (i.e., 50/138) of the comparison group. An even

smaller portion of the 5th

grade study students received math intervention instruction, 6% (i.e.,

14/239) of the SuccessMaker group and 13% (i.e., 32/251) of the comparison group.

The 3rd

grade comparison group received more intervention instruction per student than the

SuccessMaker group, about three times as many sessions (i.e., SuccessMaker = 5 and

comparison = 14 sessions per student) and nearly twice as many hours (i.e., SuccessMaker = 7

and comparison = 12 hours per student). Those students in each study group, that did receive

intervention instruction, had a similar number of sessions and hours in those sessions.

The 5th

grade comparison group received more intervention sessions per student than the

SuccessMaker group (i.e., SuccessMaker = 2 and comparison = 5 sessions per student), however,

the time in intervention was similar (i.e., SuccessMaker = 2 and comparison = 3 hours per

student). Though fewer 5th

grade SuccessMaker students received intervention instruction than

their comparison group counterparts, those that did received both more sessions and hours.

Obviously, in addition to intervention instruction, the students in each study group received

similar regular classroom reading and math instruction. The 3rd

grade study students received an

average of 103 minutes of reading instruction each day (i.e., Min = 60, P25 = 90, P50 = 100, P75 =

120, Max = 160) while the 5th

grade study students received an average of 80 minutes of math

instruction each day (i.e., Min = 60, P25 = 65, P50 = 75, P75 = 90, Max = 120). No new reading or

math content was introduced when SuccessMaker students where using the program.

SuccessMaker Program Implementation

This section will attempt to answer research question three:

RQ3: How was the SuccessMaker program implemented?

First, the SuccessMaker training will be addressed followed by program usage and finally one-

to-one program intervention instruction.


- 12 -

SuccessMaker Teacher Training

To initiate the study, Gatti Evaluation conducted study orientations for all teachers at the start of

the school year. The study orientation formally introduced the teachers to the research team,

explained in detail the requirements and benefits of participation in the study, as well as,

addressed any immediate questions or concerns about the research. All teachers were required to

read and sign informed consent forms. The publisher ensured that sites had full access to the

program and that access was continual throughout the duration of the study. Pearson also

provided free product training and funding to cover the cost of substitute teachers during

training.

All teachers implementing the SuccessMaker program with participating study students (i.e.,

SuccessMaker Interventionist, SMI) were required to attend training sessions facilitated by an

educational consultant. Initial training took place on-site over the course of one full school day.

This training introduced administrators, classroom teachers, interventionists, and technicians to

the key components of the SuccessMaker program, including; student login, learning

environments, classroom management and reporting systems, as well as how to best implement

these in practice. Initial product training sessions typically began with a group presentation.

Then interventionists moved to computers where they were given the opportunity to use the

program as students would. Interventionists had the responsibility of training their students to

use the program. The initial training dates are presented in Table 1.

Teachers and interventionists received multiple training sessions by Pearson

educational consultants.

The follow-up training sessions typically lasted three to four hours and were provided to each

site to support consistent usage of the program, acquaint teachers and interventionists with the

reporting system, provide a more detailed understanding of the program, identify and correct

technical issues and address students’ special needs. Further, the follow-up training was used to

train interventionists on how to utilize the program features to conduct one-to-one intervention

instruction sessions (i.e., see SuccessMaker One-to-One Instruction, page 14). Interventionists

were expected to begin the one-to-one program intervention instruction after winter break.

Table 1 SuccessMaker 2012-13 RCT Training Dates

State District School School Start Initial Training Follow-up Final Training

AZ 1 1 08/13/12 10/04/12 12/10/12 04/09/13

AZ 1 2 08/13/12 10/04/12 12/12/12 04/10/13

AZ 2 1 08/13/12 10/26/12 12/06/12 04/10/13

CA 1 1 08/28/12 09/10/12 12/10/12 04/15/13

CA 1 2 08/28/12 09/10/12 12/10/12 04/15/13


- 13 -

CA 2 1 08/29/12 10/22/12 12/11/12 05/14/13

CA 3 1 08/20/12 08/28/12 11/06/12 04/16/13

KS 1 1 08/15/12 10/03/12 12/11/12 02/19/13

KS 1 2 08/20/12 10/03/12 12/11/12 02/19/13

MI 1 1 09/04/12 09/25/12 12/06/12

MI 2 1 09/04/12 10/12/12 12/18/12 03/05/13

MI 2 2 09/04/12 10/12/12 12/06/12 03/05/13

OR 1 1 09/04/12 10/13/12 11/29/12 03/13/13

OR 1 2 09/04/12 10/13/12 11/30/12 03/13/13

TX 1 1 08/27/12 08/29/12 11/26/12 04/10/13

TX 1 2 08/27/12 08/29/12 11/26/12 04/10/13

TX 1 3 08/27/12 08/29/12 11/26/12 04/10/13

TX 2 1 08/28/12 09/12/12 11/27/12 05/30/13

SuccessMaker Program Usage

Students randomly assigned to use SuccessMaker were expected to use the program for at least

four twenty to thirty minute sessions each week. Participating students were pulled out of their

classrooms to work on the program in an educational laboratory setting facilitated by their

interventionist(s). No new reading or math content was introduced when SuccessMaker students

were away using the program.

SuccessMaker students used the program with their interventionists four times a

week in twenty-thirty minute sessions.

The program users at both grade levels had similar median8 usage (i.e., 3

rd = 32 hours, 5

th = 28

hours). The large majority (i.e., 75%) of students used the program at least 28 and 24 hours at

3rd

and 5th

grade respectively. The lowest usage time for any 3rd

grade SuccessMaker student

was 10 hours with two other students between 19-20 hours on the program. More of the 5th

grade users had sub 20 hour usage. Of these nineteen, seven had more than 19 hours, and

thirteen had more than 18 hours. The 5th

grade program user with the least time on SuccessMaker

8 The sample median is defined as the 50th percentile or the score for those students in the very center of the distribution of scores.


- 14 -

had 10 hours of usage. Program usage varied due to transition time, technology failures,

behavioral problems, tardiness and/or absenteeism.

Initially SuccessMaker interventionists allowed students to complete their placement without

intervening. After initial placement was completed interventionists were to actively help students

progress through the program. The 3rd

graders’ median initial placement level was below grade

level at 2.75 (i.e., Min = 0.50, P25 = 2.25, P75 = 3.00, Max = 4.00) while the median 5th

grader

placed further below grade level at 4.13 (i.e., Min = 2.49, P25 = 3.75, P75 = 4.84, Max = 6.50).

SuccessMaker interventionists were required to check last/end-of-session reports after each

sitting with the program and regularly review cumulative reports (i.e., missed items/skills,

progress graph, minutes logged) with each student a minimum of once a week. The median user

made almost a grade level gain on the program, 3rd

graders 0.86 (i.e., Min = 0.31, P25 = 0.73, P75

= 1.18, Max = 1.76) and 5th

graders 0.91 (i.e., Min = 0.25, P25 = 0.67, P75 = 1.24, Max = 2.58).

The research team required that each site coordinator regularly (i.e., every two weeks of back-to-

back instruction) download cumulative reports and send them to the research team to monitor

and verify students’ progress and proper program usage. In rare cases, flagged students were

more rigorously monitored while using the program.

The median 3rd

grade program user was productive, attempting 18 reading exercises per 20

minutes (i.e., Min = 9, P25 = 16, P75 = 20, Max = 27) with 72% correct (i.e., Min = 51%, P25 =

66%, P75 = 76%, Max = 81%), and attempting 56 skills, mastering 39 (i.e., Min = 11/17, P25 =

29/44, P75 = 49/65, Max = 84/89). The median 5th

grade program user was also productive,

attempting 28 math exercises per 20 minutes (i.e., Min = 11, P25 = 23, P75 = 33, Max = 56) with

66% correct (i.e., Min = 45%, P25 = 61%, P75 = 69%, Max = 83%), and attempting 263 skills,

mastering 208 (i.e., Min = 53/59, P25 = 144/149, P75 = 263/273, Max = 603/617).

SuccessMaker One-to-One Instruction Additionally, interventionists were expected to conduct one-to-one sessions where they review


week. This instruction was meant to intervene when students did not understand the content,

presentation, or procedures for different exercises, were going off-task, or performing below

their expectations. These sessions become necessary to get beyond sticking points, since users

will tend to slow in their program gains after several hours on the program, as incorrect exercises

and failed skills accumulate.

Not all interventionists were able to conduct these sessions regularly with their SuccessMaker

students. Most that did conduct regular one-to-one sessions concentrated their efforts on those

students in most need of individualized attention. A similar portion of students however, at each

grade level, received at least a single one-to-one session (i.e., 3rd

= 61%, 5th

= 59%), as well as, a

similar average number of sessions per student (i.e., 3rd

= 6, 5th

= 5).

Four schools had large SuccessMaker user to SuccessMaker interventionists ratios (i.e.,

SMU:SMI 22:1, 23:1, 32:1, 35:1) that did not permit systematic individualized attention;

these schools were considered Low Implementers. Schools that conducted regular one-to-one

program intervention sessions and/or had small ratios (i.e., SMU/SMI < 14), and were able to

deliver regular over-the-shoulder attention were designated the High implementation group. A


- 15 -

small portion (i.e., 21/154 = 14%) of the 3rd

grade sample came from low implementing schools

while a larger portion (i.e., 84/239 = 35%) of the 5th

grade sample were in schools with no

individualized program instruction. It should be noted that those students receiving additional

intervention instruction outside of SuccessMaker from both grades were from the High

implementation group.

Table 2 SM-RTI 2012-13 RCT Sample Demographic Information

Group Grade Student

Count

Percent Not

English

Proficient

Percent

Free/Reduc

ed Lunch

Percent

Caucasian

Percent

Hispanic

Percent

African

American

Other

Ethnicity

Whole Sample

SM

CP 3

154

138

32%

31%

79%

82%

24%

25%

62%

57%

6%

7%

8%

12%

SM

CP 5

239

251

16%

18%

70%

69%

32%

34%

47%

50%

10%

6%

11%

11%

Arizona District 1 School 1

SM

CP 5

22

18

5%

0%

82%

94%

18%

6%

64%

78%

14%

11%

5%

6%


SM

CP 3

8

6

0%

0%

100%

100%

13%

0%

63%

100%

25%

0%

0%

0%

SM

CP 5

7

8

0%

0%

100%

100%

14%

13%

57%

63%

14%

13%

14%

13%


SM

CP 3

8

8

50%

25%

100%

100%

0%

0%

100%

88%

0%

0%

0%

13%

SM

CP 5

13

15

0%

0%

100%

100%

0%

0%

100%

100%

0%

0%

0%

0%

California District 1 School 1

SM

CP 3

18

17

83%

82%

100%

94%

0%

0%

94%

100%

0%

0%

6%

0%

SM

CP 5

10

9

50%

89%

100%

100%

0%

0%

100%

100%

0%

0%

0%

0%


SM

CP 3

12

11

75%

91%

100%

100%

0%

0%

0%

0%

0%

0%

100%

100%


- 16 -

SM

CP 5

20

20

40%

50%

95%

95%

0%

0%

0%

0%

0%

0%

100%

100%


SM

CP 3

9

8

56%

25%

89%

75%

0%

13%

100%

88%

0%

0%

0%

0%

SM

CP 5

14

15

14%

13%

43%

73%

0%

0%

100%

100%

0%

0%

0%

0%


SM

CP 3

9

8

33%

25%

78%

50%

0%

13%

100%

75%

0%

0%

0%

13%

SM

CP 5

15

15

13%

27%

67%

60%

7%

7%

80%

87%

7%

0%

7%

7%

Kansas District 1 School 1

SM

CP 3

2

3

0%

0%

0%

0%

100%

100%

0%

0%

0%

0%

0%

0%

SM

CP 5

4

5

0%

0%

25%

0%

75%

100%

25%

0%

0%

0%

0%

0%

Kansas District 1 School 2

SM

CP 3

5

5

0%

0%

20%

20%

60%

80%

40%

0%

0%

0%

0%

20%

SM

CP 5

8

8

0%

0%

25%

0%

88%

88%

13%

13%

0%

0%

0%

0%

Michigan District 1 School 1

SM

CP 3

11

7

0%

0%

0%

29%

91%

86%

0%

0%

9%

14%

0%

0%

SM

CP 5

15

14

0%

0%

13%

14%

100%

86%

0%

0%

0%

7%

0%

7%


SM

CP 5

27

40

4%

0%

48%

43%

82%

80%

4%

10%

15%

5%

0%

5%


SM

CP 3

12

14

0%

14%

58%

64%

83%

79%

0%

7%

17%

7%

0%

7%


- 17 -

Oregon District 1 School 1

SM

CP 3

6

8

67%

50%

100%

100%

33%

38%

67%

50%

0%

13%

0%

0%

SM

CP 5

20

19

25%

16%

80%

74%

55%

58%

40%

42%

0%

0%

5%

0%

Oregon District 1 School 2

SM

CP 3

9

8

11%

13%

44%

100%

89%

50%

11%

25%

0%

13%

0%

13%

SM

CP 5

19

20

0%

5%

47%

45%

63%

70%

16%

25%

11%

0%

11%

5%

Texas District 1 School 1

SM

CP 3

10

8

50%

38%

100%

100%

10%

13%

70%

75%

20%

13%

0%

0%

SM

CP 5

12

12

58%

50%

83%

100%

8%

8%

83%

75%

8%

17%

0%

0%


SM

CP 3

9

9

11%

11%

100%

100%

0%

0%

100%

78%

0%

22%

0%

0%

SM

CP 5

13

17

39%

53%

92%

100%

0%

0%

69%

82%

31%

18%

0%

0%


SM

CP 3

18

12

11%

17%

94%

92%

0%

0%

89%

75%

11%

25%

0%

0%

SM

CP 5

20

16

5%

13%

100%

94%

0%

0%

65%

81%

35%

19%

0%

0%


SM

CP 3

8

6

0%

0%

88%

100%

0%

0%

100%

100%

0%

0%

0%

0%


- 18 -

Participants

The final study sample consisted of 782 3rd

and 5th

grade students from eighteen

schools, in six states, located in different regions of the US.

The final sample is comprised of students from eighteen schools distributed across six different

states (i.e., AZ, CA, KS, MI, OR, and TX). The final study sample (i.e., students tested at

baseline, remained in assigned group, tested at end-of-year) consisted of 292 3rd

grade (i.e.,

SuccessMaker = 154, comparison = 138) and 490 5th

grade (i.e., SuccessMaker = 239,

comparison = 251) students.

Of the 321 3rd

grade students baseline tested, 7% percent withdrew from study participation (i.e.,

opted out of study, required to switch classrooms resulting in crossing study groups, moved or

otherwise withdrew from participating school) and 2% did not test at end-of-year due to testing

issues or chronic absenteeism. At 5th

grade, 541 students baseline tested, 8% percent withdrew

from study participation and 1% did not test at end-of-year due to testing issues or chronic

absenteeism.

The data in Table 2 provides the demographic breakdown of the final study sample. The study

schools demonstrated considerable variation in ethnicity, as well as percentage of students

eligible for free or reduced-priced lunch. As would be expected with the student level random

assignment, the study samples at each grade level are very similar in their demographic makeup.

The samples from both grade levels are largely non-Caucasian and free/reduced lunch eligible.

Also, the 5th

grade sample, selected on math achievement, is more female than male.

3rd


gender (male) 55% 51%

special education 9% 9%

not English proficient 32% 31%

free/reduced lunch 79% 82%

Hispanic 62% 57%

African American 6% 7%

5th


gender (male) 43% 45%

special education 13% 8%

not English proficient 16% 18%

free/reduced lunch 70% 69%


- 19 -

Hispanic 47% 50%

African American 10% 6%

Data Analysis Procedures

Statistical analyses were performed on students’ achievement outcome measures raw gain

scores9, as well as, math academic attitude survey raw gain scores. Gain scores are calculated by

subtracting the beginning-of-year raw score from the end-of-year raw score. The performance

for the comparison group was compared to the entire SuccessMaker group as well as groups

based upon two levels SuccessMaker program implementation (i.e., low, high). For a detailed

description of the implementation categories and their meaning see the section on SuccessMaker

One-to-One Instruction, page 14.

Comparisons were made between study groups (i.e., comparison vs. SuccessMaker) using group

mean gain differences. An ordinary least squares fixed effects model was employed to

statistically test group mean differences. While students were the unit of analysis, the schools

were the independent units. The hierarchical nature of the data (i.e., students nested within

classrooms, classrooms nested within schools, schools nested within districts) has the effect of

reducing the amount of independent information available in the sample, therefore decreasing the

precision of estimates and the power of hypothesis tests to find these estimates statistically

significant.10

A naïve covariance structure within a robust empirical standard error formulation

was used to calculate confidence intervals for estimated effects. This procedure results in

estimates that are unbiased and statistical hypothesis tests that are consistent11,12

despite the

nested nature of the data.

Twenty-three covariates were entered into the statistical models for analyses. These covariates

included student demographic information (ex., meal status, ethnicity), as well as lab/classroom

and school environment indicators (ex., enrollment, baseline achievement, minutes daily

classroom reading/math instruction). Adding variables known to impact achievement outcomes

to the statistical models may reduce the residual variation or error about the estimates, resulting

in more precisely estimated results and additional power to statistically detect significant

differences. Also, the additional variables may reduce the effect from nesting and help

normalize the residual distribution.

All statistical significance tests are two-tailed with a Type I error rate of 0.05. Statistically

significant estimates are ones in which the probability of sampling scores that result in a mean

difference different from zero when it is in fact null, is no better than 1 in 20 samples.

Significance implies that the samples are likely drawn from two separate populations or that the

groups are unlikely to be the same in the population. Coupled with the rigorous study design we

9 GMADE scale raw score = number of questions answered correctly, see the description under Mathematics Academic Attitude and

SucessMaker Opinion Surveys from the section Student Outcome Measures for an explanation of how the raw scores are calculated for the Math Academic Attitude Survey 10 Donnar, A. & Klar, N. (2000) Design and analysis of cluster randomization trials in health research. Arnold Publishers, London. 11 Liang, N. M. & and Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, pp. 13-22. 12 SAS’s Mixed procedure was used to analyze the data, see SAS Institute Inc. (2008) Online documentation 9.2. A linear model was defined

with all fixed effects, full degrees of freedom, using the sandwich estimator for all standard errors with districts set as the independent level of

nesting, and a naïve independent working covariance structure.


- 20 -

may then hold these statistically significant differences as evidence for one group outperforming

the other.

Standardized effect size estimates (i.e., effect size = group mean gain difference / comparison

group sample gain score standard deviation) along with a percentile rank based effect size

measure are computed for statistically significant differences.13

The latter effect size measure

indicates the percentile rank for the average SuccessMaker gain in relation to the comparison

group’s distribution. For example, if the treatment group outperformed the comparison group by

0.20 standard deviations the average score for the treatment group was larger than 58% of the

comparison group scores.

3rd

Grade Detectable ES Observed ES ES Differences

Average 0.19 (57%) 0.14 (56%) -0.04

Minimum 0.07 (53%) 0.00 (50%) -0.23

25th

percentile 0.13 (55%) 0.08 (53%) -0.13

50th

percentile 0.20 (58%) 0.13 (55%) -0.07

75th

percentile 0.21 (58%) 0.17 (57%) 0.06

Maximum 0.34 (63%) 0.41 (66%) 0.17

5th

Grade Detectable ES Observed ES ES Differences

Average 0.21 (58%) 0.28 (61%) 0.07

Minimum 0.13 (55%) 0.01 (50%) -0.13

25th

percentile 0.17 (57%) 0.14 (55%) -0.04

50th

percentile 0.19 (57%) 0.23 (59%) 0.03

75th

percentile 0.21 (58%) 0.35 (64%) 0.13

Maximum 0.53 (70%) 0.64 (74%) 0.44

The detectable effect size (ES) indicates the minimal standardized group gain difference necessary to find statistical significance (i.e., p-

value = 0.05 Type I error level).

The effect size in parentheses indicates the percentile rank for the average treatment group gain score in relation to the comparison group.

The absolute value effect size indicates the size of the observed group gain score differences irrespective of which group gained more.

The effect size difference = absolute value effect size – detectable effect size with values > 0 indicating a significant test result.

The statistical models were able to find small to moderate effect sizes statistically significant.

The group mean gain differences for 3rd

grade outcomes were not large. The average absolute

value for calculated effect sizes across all 3rd

grade outcomes and group comparisons was 0.14

standard deviations or a percentile rank of 56% (i.e., Min = <0.01, P25 = 0.08, P50 = 0.13, P75 =

0.17, Max = 0.41). The statistical models were able to find 37% (i.e., 13/35) of the 3rd

grade

comparisons statistically significant.

13 Hedges, L. V. & Olkin, I. (1985). Statistics methods for meta-analysis. Academic Press, NY.


- 21 -

The 5th

grade significance tests had more power to find statistical significance as a consequence

of larger group gain differences. The average absolute value for calculated effect sizes across all

5th

grade outcomes and group comparisons was 0.28 standard deviations or a percentile rank of

61% (i.e., Min = 0.01, P25 = 0.14, P50 = 0.23, P75 = 0.35, Max = 0.64). The 5th

grade statistical

models were able to find 63% (i.e., 22/35) of the comparisons statistically significant.

The careful review of efficacy studies for educational materials14

indicate that the average group

mean difference for studies with large samples (i.e., more than 250 students) is only 0.13

standard deviations or a percentile rank of 55%. The most recent Best Evidence Encyclopedia’s

review15

of the comparative effectiveness of supplemental computer-assisted instruction with

struggling early elementary readers was found to be an increase of 0.36 standard deviations over

standard procedures, or a percentile rank of 64%. Another review16

of the comparative

effectiveness of supplemental computer-assisted instruction on mathematics achievement found

an increase of 0.19 standard deviations over traditional methods, or a percentile rank of 58%.

14 Slavin, R. & Smith, D. (2009). The relationship between sample sizes and effect sizes in systematic reviews in education. Educational

Evaluation and Policy Analysis, 31(4) pp. 500-506. 15Cheung, A., Slavin, R.E. (2012, June). Effects of Educational Technology Applications on Reading Outcomes for Struggling Readers: A Best

Evidence Synthesis. Baltimore, MD: Johns Hopkins University, Center for Research and Reform in Education. The full report can be downloaded

at www.bestevidence.org/reading/tech/tech_strug_read.html 16 Cheung, A., Slavin, R.E. (2011, July). The Effectiveness of Educational Technology Applications for Enhancing Mathematics Achievement in

K-12 Classrooms: A Meta-Analysis. Baltimore, MD: Johns Hopkins University, Center for Research and Reform in Education. The full report

can be downloaded at www.bestevidence.org/math/tech/tech_math.htm


- 22 -

III. RESULTS

Report section III summarizes the results of data analyses, including statistical and qualitative

results, and group comparisons at baseline. The first subsection demonstrates the closeness of

the samples on the outcome measures at baseline. The second subsection addresses research

question one, showing baseline to end-of-year achievement gain for the SuccessMaker group and

comparing those gains to that of the comparison group for two levels of program

implementation. For a detailed description of the implementation categories and their meaning

see the section on SuccessMaker One-to-One Instruction, page 14. The third and fourth

subsections address both research questions two and three. That is, do SuccessMaker students

demonstrate more positive academic attitudes and how did interventionists and students react to

the program?

Baseline Group Equivalence

Tables 4 and 5 present the 3rd

and 5th

grade simple sample17

baseline SuccessMaker and

comparison group means for each measure of achievement and academic attitude. Tables 6 and

7 present both the simple sample and model adjusted18

baseline group mean differences for each

outcome measure. Tables 6 and 7 also show statistical significance test results and effect size

measures for the baseline group mean differences. Tables 8 through 11 present 3rd

and 5th

grade

baseline results broken out for the two SuccessMaker implementation levels.

The 3rd

grade assessment battery consisted of the GRADE with its four subtests, Word Reading

(28 questions), Vocabulary (35 questions), Sentence Comprehension (19 questions) and Passage

Comprehension (28 questions), as well as, the AIMSweb R-CBM (i.e., median words read

correctly in one minute) and the reading academic attitude survey (i.e., score range is -16 to

+16). The 5th

grade assessment battery consisted of the GMADE with its three subtests,

Concepts and Communication (28 questions), Operations and Computation (24 questions), and

Process and Applications (30 questions), as well as, the AIMSweb MCOMP (i.e., 39

computation questions, 76 points) and MCAP (i.e., 30 concepts and applications questions, 51

points), and the math academic attitude survey (i.e., score range is -16 to +16).

The 3rd

grade study groups were statistically equivalent at baseline on all outcome measures

while the 5th

grade study groups were statistically equivalent at baseline on all outcome measures

except the MCAP. The 3rd

grade study groups only differed by -0.19 to -0.01 standard

deviations on all outcome measures. The 5th

grade study groups differed by less than a single

point on all outcome measures or -0.05 to 0.14 standard deviations, and though significant, the

groups differed on the MCAP by only a third of a point and 0.15 standard deviations.

The 3rd

grade High implementing sites’ students were statistically equivalent at baseline on all

outcome measures (i.e., 0.06 to 0.23 standard deviations). The 3rd

grade Low implementing

students were statistically outperformed at baseline on the GRADE overall, the Sentence

17 Sample group mean differences are estimated allowing student demographics to vary as they were sampled and randomly assigned. 18 Adjusted baseline group mean differences are estimated holding student and classroom demographic variables constant across groups.


- 23 -

Comprehension and Vocabulary subtests, as well as, the AIMSweb R-CBM (i.e., Words Read

Correctly). These differences were moderate to large at -0.30 to -0.48 standard deviations.

At 5th

grade the SuccessMaker students from both high and low implementing sites were

statistically equivalent to the comparison group on all outcome measures except two. That is, the

High students were statistically outperformed on the MCAP (i.e., -0.25 standard deviations) and

the Low group was statistically outperformed on the Process and Applications subtest.

Table 3 3rd Grade Baseline Study Group Scores

Measure Sample Size

(SM/CP)1

SM Mean (SD)2 CP Mean (SD)

GRADE Total 154/138 61.69 (14.724) 60.42 (14.248)

Word Reading 154/138 25.51 (3.500) 25.09 (4.096)

Sentence Comprehension 154/138 11.36 (4.080) 11.33 (3.815)

Vocabulary 154/138 13.79 (5.618) 13.74 (5.375)

Passage Comprehension 154/138 11.03 (4.435) 10.27 (4.063)

Reading Survey 154/138 5.55 (4.452) 4.78 (4.036)

Words Read Correctly 154/138 46.25 (13.325) 44.65 (13.850)

1. SM/CP indicates the SM group size / comparison group size

2. Mean indicates the group sample mean value and (SD) indicates the group sample standard deviation

Table 4 5th Grade Baseline Study Group Scores

Measure Sample Size

(SM/CP)1


GMADE Total 239/251 37.28 (9.117) 37.68 (9.902)

Concepts and Communications 239/251 16.29 (4.253) 16.38 (4.480)

Operations and Computation 239/251 11.34 (3.807) 11.14 (4.047)

MCOMP 239/251 11.70 (7.618) 12.10 (7.665)

Process and Applications 239/251 9.65 (3.331) 10.16 (3.702)

MCAP 239/251 4.58 (1.792) 4.90 (2.062)

Math Survey 239/251 2.45 (4.571) 2.49 (4.915)




- 24 -

Table 5 3rd Grade Baseline Study Group Score Comparisons

Measure Sample Size

SM/CP1

Sample

Difference2

Sample

p-value

Sample

Effect Size3

Adjusted

Difference4

Adjusted

p-value

Adjusted

Effect Size

GRADE Total 138/154 -1.2745 0.4437 -0.09 -1.0944 0.5093 -0.08

Word Reading 138/154 -0.4260 0.4823 -0.10 -0.4419 0.4745 -0.11

Sentence Comprehension 138/154 -0.0311 0.9459 -0.01 -0.0238 0.9583 -0.01

Vocabulary 138/154 -0.0531 0.9194 -0.01 0.1032 0.8457 0.02

Passage Comprehension 138/154 -0.7644 0.0787 -0.19 -0.7320 0.0784 -0.18

Reading Survey 138/154 -0.7628 0.0859 -0.19 -0.7744 0.0905 -0.19

Words Read Correctly 138/154 -1.5982 0.4056 -0.12 -1.6145 0.4022 -0.12


2. Sample group mean differences are estimated allowing student demographics to vary as they were sampled and randomly assigned.

3. Effect Size = estimated sample or adjusted group difference / comparison sample standard deviation

4. Adjusted baseline group mean differences are estimated holding student demographic variables constant across groups

Table 6 5th Grade Baseline Study Group Score Comparisons

Measure Sample Size

SM/CP1

Sample

Difference2

Sample

p-value

Sample

Effect Size3

Adjusted

Difference4

Adjusted

p-value

Adjusted

Effect Size

GMADE Total 239/251 0.3928 0.7330 0.04 0.1477 0.9029 0.01

Concepts and Communications 239/251 0.0856 0.8536 0.02 -0.0402 0.9344 -0.01

Operations and Computation 239/251 -0.1955 0.6683 -0.05 -0.2810 0.5607 -0.07

MCOMP 239/251 0.4009 0.4754 0.05 0.2249 0.7051 0.03

Process and Applications 239/251 0.5027 0.1612 0.14 0.4688 0.2119 0.13

MCAP 239/251 0.3188 0.0395 0.15 0.2782 0.0662 0.13

Math Survey 239/251 0.0382 0.9253 0.01 0.0225 0.9572 0.00


2. Sample group mean differences are estimated allowing student and demographics to vary as they were sampled and randomly assigned.


4. Adjusted baseline group mean differences are estimated holding student demographic variables constant across groups


- 25 -

Table 7 3rd Grade Baseline Study Group Scores by SuccessMaker

Implementation Level

Measure Implementati

on Level

Sample Size

(SM/CP)1


GRADE Total High 133/138 62.68 (14.530) 60.42 (14.248)

GRADE Total Low 21/138 55.43 (14.740) 60.42 (14.248)

Word Reading High 133/138 25.74 (3.249) 25.09 (4.096)

Word Reading Low 21/138 24.10 (4.647) 25.09 (4.096)

Sentence Comprehension High 133/138 11.54 (4.020) 11.33 (3.815)

Sentence Comprehension Low 21/138 10.19 (4.366) 11.33 (3.815)

Vocabulary High 133/138 14.21 (5.527) 13.74 (5.375)

Vocabulary Low 21/138 11.14 (5.597) 13.74 (5.375)

Passage Comprehension High 133/138 11.20 (4.552) 10.27 (4.063)

Passage Comprehension Low 21/138 10.00 (3.521) 10.27 (4.063)

Reading Survey High 133/138 5.56 (4.437) 4.78 (4.036)

Reading Survey Low 21/138 5.48 (4.654) 4.78 (4.036)

Words Read Correctly High 133/138 47.39 (12.326) 44.65 (13.850)

Words Read Correctly Low 21/138 39.00 (16.535) 44.65 (13.850)



Table 8 5th Grade Baseline Study Group Scores by SuccessMaker


Measure Implementation

Level

Sample Size

(SM/CP)1


GMADE Total High 155/251 37.57 (9.422) 37.68 (9.902)

GMADE Total Low 84/251 36.76 (8.557) 37.68 (9.902)

Concepts and Communications High 155/251 16.28 (4.370) 16.38 (4.480)

Concepts and Communications Low 84/251 16.31 (4.054) 16.38 (4.480)

Operations and Computation High 155/251 11.41 (3.753) 11.14 (4.047)

Operations and Computation Low 84/251 11.20 (3.923) 11.14 (4.047)

MCOMP High 155/251 10.99 (6.619) 12.10 (7.665)

MCOMP Low 84/251 13.00 (9.082) 12.10 (7.665)

Process and Applications High 155/251 9.87 (3.420) 10.16 (3.702)


- 26 -

Process and Applications Low 84/251 9.25 (3.139) 10.16 (3.702)

MCAP High 155/251 4.39 (1.800) 4.90 (2.062)

MCAP Low 84/251 4.94 (1.731) 4.90 (2.062)

Math Survey High 155/251 2.65 (4.648) 2.49 (4.915)

Math Survey Low 84/251 2.08 (4.429) 2.49 (4.915)



Table 9 3rd Grade Baseline Study Group Score Comparisons by SuccessMaker Implementation

Level


Level

Sample

Size

SM/CP1

Sample

Difference2

Sample

p-value

Sample

Effect

Size3

Adjusted

Difference4

Adjusted

p-value

Adjusted

Effect

Size

GRADE Total High 133/138 2.2639 0.2038 0.16 1.4118 0.4552 0.10

GRADE Total Low 21/138 -4.9917 0.0012 -0.35 -0.7451 0.6024 -0.05

Word Reading High 133/138 0.6499 0.2497 0.16 0.7106 0.2433 0.17

Word Reading Low 21/138 -0.9917 0.0632 -0.24 -1.1157 0.0071 -0.27

Sentence Comprehension High 133/138 0.2153 0.6725 0.06 0.0294 0.9561 0.01

Sentence Comprehension Low 21/138 -1.1356 0.0003 -0.30 -0.0087 0.9798 0.00

Vocabulary High 133/138 0.4714 0.4050 0.09 -0.1357 0.8238 -0.03

Vocabulary Low 21/138 -2.5963 <.0001 -0.48 0.0847 0.8904 0.02

Passage Comprehension High 133/138 0.9274 0.0553 0.23 0.8075 0.0882 0.20

Passage Comprehension Low 21/138 -0.2681 0.4390 -0.07 0.2946 0.5695 0.07

Reading Survey High 133/138 0.7738 0.0807 0.19 0.7531 0.1110 0.19

Reading Survey Low 21/138 0.6936 0.3456 0.17 0.8975 0.2842 0.22

Words Read Correctly High 133/138 2.7424 0.1399 0.20 2.6604 0.1633 0.19

Words Read Correctly Low 21/138 -5.6486 0.0008 -0.41 -4.4475 0.0098 -0.32




4. Adjusted baseline group mean differences are estimated holding student demographic variables constant across groups.


- 27 -

Table 10 5th Grade Baseline Study Group Score Comparisons by SuccessMaker



Level

Sample

Size

SM/CP1

Sample

Difference2

Sample

p-

value

Sample

Effect

Size3

Adjusted

Difference4

Adjusted

p-value

Adjusted

Effect

Size

GMADE Total High 155/251 -0.1095 0.9097 -0.01 0.4649 0.6614 0.05

GMADE Total Low 84/251 -0.9154 0.5985 -0.09 -1.2675 0.4636 -0.13

Concepts and Communications High 155/251 -0.0946 0.8172 -0.02 0.2463 0.5779 0.05

Concepts and Communications Low 84/251 -0.0690 0.9113 -0.02 -0.3367 0.5819 -0.08

Operations and Computation High 155/251 0.2695 0.5739 0.07 0.3821 0.4597 0.09

Operations and Computation Low 84/251 0.0590 0.9397 0.01 0.0963 0.9015 0.02

MCOMP High 155/251 -1.1061 0.1467 -0.14 -0.7666 0.3580 -0.10

MCOMP Low 84/251 0.9004 0.5837 0.12 0.7655 0.6673 0.10

Process and applications High 155/251 -0.2844 0.3981 -0.08 -0.1635 0.6547 -0.04

Process and Applications Low 84/251 -0.9054 0.0305 -0.24 -1.0271 0.0160 -0.28

MCAP High 155/251 -0.5133 0.0066 -0.25 -0.4359 0.0212 -0.21

MCAP Low 84/251 0.0401 0.8570 0.02 0.0100 0.9633 0.00

Math Survey High 155/251 0.1616 0.6719 0.03 0.3123 0.4714 0.06

Math Survey Low 84/251 -0.4067 0.4091 -0.08 -0.6346 0.2622 -0.13




4. Adjusted baseline group mean differences are estimated holding student demographic variables constant across groups.

Students’ Achievement Gains

SuccessMaker Students’ Achievement Gains In this section SuccessMaker students’ achievement gains are presented for each achievement

measure at each grade level, as well as broken out by level of program implementation (i.e.,

High, Low). For a detailed description of the implementation categories and their meaning see

the section on SuccessMaker One-to-One Instruction, page 14.

The effects for all outcome measures at both grades are all large and statistically significant,

ranging from 0.51 to 2.00 standard deviations. When broken out for the two SuccessMaker

implementation levels, the High implementation group saw large statistically significant

achievement gains (i.e., 0.58 to 1.97 standard deviations). The Low implementation group also

saw large statistically significant achievement gains (i.e., 0.43 to 2.16 standard deviations)


- 28 -

except in two cases, Passage Comprehension and Process and Applications. These achievement

areas have traditionally been difficult for software to significantly affect achievement.

Intervention students demonstrated large statistically significant achievement

gains in the first school year implementing the program. This was especially

true for students receiving proper implementation in the traditionally difficult

areas of Passage Comprehension and Process and Applications

3rd

Grade Scale Sample Size Gain (SD) GMADE Effect Size

GRADE Total 154 11.30 (8.74) 1.29

Word Reading 154 1.58 (2.65) 0.60

Sentence Comprehension 154 2.97 (2.66) 1.12

Vocabulary 154 3.70 (4.00) 0.93

Passage Comprehension 154 3.04 (4.85) 0.63

Words Read Correctly 154 31.79 (15.91) 2.00

Sample Size indicates the number of SM group students.

Gain = sample mean baseline to post-test gain score, SD = sample gain score (number questions correct) standard deviation

Effect Size = sample mean baseline to post-test gain score / sample gain score standard deviation

5th

Grade Scale Sample Size Gain (SD) GMADE Effect Size

GMADE Total 239 8.23 (7.91) 1.04

Concepts and Communication 239 2.44 (3.92) 0.62

Operations and Computation 239 3.80 (3.69) 1.03

MCOMP 231 18.53 (13.76) 1.35

Process and Applications 239 2.00 (3.88) 0.51

MCAP 231 3.26 (4.13) 0.79




3rd

Grade Scale Implementation Sample Size Gain (SD) Effect Size

GRADE Total High 133 11.65 (8.74) 1.33

GRADE Total Low 21 9.05 (8.74) 1.03

Word Reading High 133 1.53 (2.65) 0.58

Word Reading Low 21 1.95 (2.65) 0.74

Sentence Comprehension High 133 3.13 (2.66) 1.18


- 29 -

Sentence Comprehension Low 21 2.00 (2.66) 0.75

Vocabulary High 133 3.65 (4.00) 0.91

Vocabulary Low 21 4.05 (4.00) 1.01

Passage Comprehension High 133 3.35 (4.85) 0.69

Passage Comprehension Low 21 1.05 (4.85) 0.22

Words Read Correctly High 133 31.37 (15.91) 1.97

Words Read Correctly Low 21 34.43 (15.91) 2.16

Implementation level (i.e., High, Low) refers to how the program was implemented at each school.




5th

Grade Scale Implementation Sample Size Gain (SD) Effect Size

GMADE Total High 155 9.69 (7.91) 1.22

GMADE Total Low 84 5.55 (7.91) 0.70

Concepts and Communication High 155 2.84 (3.92) 0.72

Concepts and Communication Low 84 1.70 (3.92) 0.43

Operations and Computation High 155 4.29 (3.69) 1.16

Operations and Computation Low 84 2.89 (3.69) 0.78

MCOMP High 152 18.14 (13.76) 1.32

MCOMP Low 79 19.28 (13.76) 1.40

Process and Applications High 155 2.56 (3.88) 0.66

Process and Applications Low 21 0.95 (3.88) 0.25

MCAP High 152 2.89 (4.13) 0.70

MCAP Low 79 3.96 (4.13) 0.96





Group Comparisons of Achievement Gains This section will attempt to answer research question one:

RQ1: Do RTI students using the SuccessMaker program demonstrate a significant improvement

in achievement over otherwise similar students in classrooms supplementing reading and math

intervention instruction without using an adaptive, computer-based program like SuccessMaker?

In this section SuccessMaker students’ achievement gains are compared to their peers not

regularly using an adaptive computer-based intervention program. Achievement gains are

compared for each outcome measure as well as broken out by level of program implementation

(i.e., High, Low). For a detailed description of the implementation categories and their meaning

see the section on SuccessMaker One-to-One Instruction, page 14.


- 30 -

Figures 1 and 2 present the group differences for all achievement outcome measures. Figure 1

shows the comparative gains for the 3rd

grade reading outcomes (i.e., GRADE, GRADE subtests,

AIMSweb R-CBM Words Read Correctly) with the 5th

grade math outcomes in Figure 2 (i.e.,

GMADE, GMADE subtests, AIMSweb MCAP and MCOMP). The bars in these graphs indicate

the magnitude of the group difference as the position, in percentile rank, of the SuccessMaker

group mean gain score in the distribution of comparison group gain scores. By definition, 50%

indicates there is no difference in the groups and that the distribution of scores for each group

perfectly overlaps. Further, the absence of a bar indicates the groups were not statistically

significantly different (i.e., NS).

Figure 1. Grade 3 Group Comparison Achievement Gains

NS

44%

NS

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

GRADE Total

SuccessMaker All SuccessMaker Low SuccessMaker High

SM = CP

CP > SM

SM > CP


- 31 -

NS

58%

NS

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Words Read Correctly


SM = CP

CP > SM

SM > CP

NS

40%

NS

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Sentence Comprehension


SM = CP

CP > SM

SM > CP


- 32 -

NS

34%

NS

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Passage Comprehension


SM = CP

CP > SM

SM > CP

NS NS NS

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Word Reading


SM = CP

CP > SM

SM > CP


- 33 -

57%

60%

57%

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Vocabulary


SM = CP

CP > SM

SM > CP

When comparing the complete 3

rd grade study groups after the first school year of program

implementation, there is one statistically discernible effect. That is, 3rd grade SuccessMaker

students outperformed the comparison group on the Vocabulary subtest by 0.18 standard

deviations. After dissecting the 3rd

grade SuccessMaker sample into High and Low

implementing groups, both groups of SuccessMaker students outperformed the comparison

group on the Vocabulary subtest by 0.17 and 0.25 standard deviations, respectively. The High

SuccessMaker group was statistically equivalent to the comparison group on the remaining 3rd

grade outcomes. The Low SuccessMaker group, however, was outperformed on the GRADE

and its comprehension subtests. The Low group did also consistently read more words correctly

than the comparison group.

Comparatively, the 5th

grade SuccessMaker students outperformed the comparison group on the

AIMSweb Computation (MCOMP) and Applications (MCAP) by 0.59 and 0.44 standard

deviations, respectively. The complete groups scored statistically equivalent on the GMADE

and its subtests. When splitting the 5th

grade SuccessMaker sample into High and Low

implementing groups, both groups of SuccessMaker students still outperformed the comparison

group on the AIMSweb Computation and Applications scales.

The High SuccessMaker group statistically outperformed the comparison group on the GMADE

and two of its subtests, Concepts and Communications (i.e., 0.23 standard deviations), as well as,

Operations and Computation (i.e., 0.27 standard deviations). The High SuccessMaker group was

statistically equivalent to the comparison group on the Process and Applications subtest.

Conversely, the Low SuccessMaker group was outperformed on the GMADE and its Process and

Applications subtest by -0.23 and -0.26 standard deviations, respectively. The Low

SuccessMaker group was statistically equivalent on the other subtest.


- 34 -

The 3rd

grade students using SuccessMaker saw statistically significantly larger

gains in their vocabulary than their comparison group counterparts in the initial

school year with the program.

In the initial school year with the program the 5th


saw statistically significantly larger gains than their comparison group

classmates on the AIMSweb MCOMP and MCAP scales.

Those 5th

grade SuccessMaker students coupling the program with regular

individualized attention from their interventionist saw statistically significantly

larger gains than their comparison group classmates in mathematics

achievement overall, as well as, Concepts, Computation, and Problem Solving.

Figure 2. 5th

Grade Group Comparison Achievement Gains

NS

41%

63%

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

GMADE Total


SM = CP

CP > SM

SM > CP


- 35 -

NS NS

59%

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Concepts & Communication


SM = CP

CP > SM

SM > CP

NS NS

61%

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Operations & Computation


SM = CP

CP > SM

SM > CP


- 36 -

NS

40%

NS

30%

40%

50%

60%

70%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

Process & Applications


SM = CP

CP > SM

SM > CP

72%

74%

71%

30%

40%

50%

60%

70%

80%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

MCOMP


SM = CP

CP > SM

SM > CP


- 37 -

67%

74%

63%

30%

40%

50%

60%

70%

80%

Su

cce

ssM

aker G

ro

up P

erc

en

tile

Ran

k a

s

Co

mp

are

d to

No

n-U

sers

MCAP


SM = CP

CP > SM

SM > CP

3rd Grade Scale Implementation Sample Size GMADE Effect Size

GRADE Total SM-CP 154/138 ***

GRADE Total High-CP 133/138 ***

GRADE Total Low-CP 21/138 -0.14 (44%)

Word Reading SM-CP 154/138 ***

Word Reading High-CP 133/138 ***

Word Reading Low-CP 21/138 ***

Sentence Comprehension SM-CP 154/138 ***

Sentence Comprehension High-CP 133/138 ***

Sentence Comprehension Low-CP 21/138 -0.27 (40%)

Vocabulary SM-CP 154/138 0.18 (57%)

Vocabulary High-CP 133/138 0.17 (57%)

Vocabulary Low-CP 21/138 0.25 (60%)

Passage Comprehension SM-CP 154/138 ***

Passage Comprehension High-CP 133/138 ***

Passage Comprehension Low-CP 21/138 -0.41 (34%)

Words Read Correctly SM-CP 154/138 ***

TR NS

High NS


- 38 -

Words Read Correctly High-CP 133/138 ***

Words Read Correctly Low-CP 21/138 0.21 (58%)


Sample size indicates the number of SuccessMaker (SM) and comparison group students.

effect size = group gain score difference / comparison sample gain score standard deviation

The value in the parentheses indicates the percentile rank for the SM group mean gain score in relation to the comparison group

(i.e., 50% indicates SM and comparison gain score distributions overlap).

The average effect size for studies with large samples (i.e., more than 250 students) has been recently estimated at 0.13 standard

deviations.

*** Indicates estimated adjusted group difference is not statistically significant at p = 0.05 Type I error rate.

5th

Grade Scale Implementation Sample Size GMADE Effect Size

GMADE Total SM-CP 239/251 ***

GMADE Total High-CP 155/251 0.33 (63%)

GMADE Total Low-CP 84/251 -0.23 (41%)

Concepts and Communication SM-CP 239/251 ***

Concepts and Communication High-CP 155/251 0.23 (59%)

Concepts and Communication Low-CP 84/251 ***

Operations and Computation SM-CP 239/251 ***

Operations and Computation High-CP 155/251 0.27 (61%)

Operations and Computation Low-CP 84/251 ***

MCOMP SM-CP 231/247 0.59 (72%)

MCOMP High-CP 152/247 0.56 (71%)

MCOMP Low-CP 79/247 0.63 (74%)

Process and Applications SM-CP 239/251 ***

Process and Applications High-CP 155/251 ***

Process and Applications Low-CP 84/251 -0.26 (40%)

MCAP SM-CP 231/247 0.44 (67%)

MCAP High-CP 152/247 0.33 (63%)

MCAP Low-CP 79/247 0.64 (74%)




The value in the parentheses indicates the percentile rank for the SM group mean gain score in relation to the comparison group

(i.e., 50% indicates SM and comparison gain score distributions overlap).

The average effect size for studies with large samples (i.e., more than 250 students) has been recently estimated at 0.13 standard deviations.



- 39 -

Student Academic Attitudes

The study groups saw statistically equivalent gains in their academic attitude

with the exception of the 5th

grade Low implementation group, which had

statistically lower gains than their comparison group counterparts.

This section will attempt to answer research question two:

RQ2: Do RTI students using the SuccessMaker program demonstrate more positive attitudes

toward reading/math and reading/math instruction when compared to their non-SuccessMaker

counterparts?

There was no statistical difference between the two 3rd

grade study groups in reading academic

attitude across all students, or when the SuccessMaker group was split out by program

implementation. At 3rd

grade, there was no statistical difference between the two study groups in

math academic attitude across all students or the High implementing SuccessMaker group. The

3rd

grade comparison students did demonstrate statistically higher academic attitudes than the

Low implementing SuccessMaker group.

Grade Implementation Sample Size Effect Size

3 SM-CP 154/138 ***

3 High-CP 133/138 ***

3 Low-CP 21/138 ***

5 SM-CP 239/251 ***

5 High-CP 155/251 ***

5 Low-CP 84/251 -0.16 (43%)




The value in the parentheses indicates the percentile rank for the SM group mean gain score in relation to

the comparison group (i.e., 50% indicates SM and comparison gain score distributions overlap).

The average effect size for studies with large samples (i.e., more than 250 students) has been recently estimated at 0.13 standard deviations.


Teacher and Student SuccessMaker Opinions

This section addresses research question five:

RQ5: How did teachers and students react to the SuccessMaker program?


- 40 -

The first sub-section summarizes the results for SuccessMaker student opinion surveys. The

second sub-section summarizes comments collected from SuccessMaker interventionists’ focus

groups.

Student SuccessMaker Attitudes When SuccessMaker students were surveyed as to their opinions on several aspects of the

program (i.e., 437 responses), a majority (i.e., 51%) responded they definitely liked the program.

A large majority of the 3rd

graders responded they definitely liked the program while a much

smaller portion of the 5th

graders regularly using the math portion said they definitely liked the

program. Still, however, less than 16% of the 5th

grade users were adamant they did not like the

program. A large majority of the 3rd

grade users also responded they definitely liked the

animation and graphics, and most found stories in the program that they liked.

A majority of the 5th

grade users also liked the animation and most found activities from the

program they liked. Both the 3rd

and 5th

grade samples agreed they definitely liked the

scaffolding offered by the program. Lastly, the majority of the 3rd

grade users felt the stories

(i.e., 50%) and the questions (i.e., 60%) were of medium difficulty and a large majority of the 5th

graders (i.e., 75%) felt the questions were of medium difficulty, this of course, is right were the

offerings from the program should be.

Fifty-one percent of students surveyed indicated they liked the program and all

but a small minority (i.e., 11%) found aspects they liked about the program.

Grade Question Yes, Definitely Sometimes No, Not Really

3rd

Like SM. 67.6% (117) 27.2% (47) 5.2% (9)

Like activities and games in SM. 78.0% (135) 19.7% (34) 2.3% (4)

Like stories in SM Reading. 46.8% (81) 39.3% (68) 13.9% (24)

Like pictures in SM Reading. 71.7% (124) 21.4% (37) 6.9% (12)

Like videos in SM Reading. 65.9% (114) 27.2% (47) 6.9% (12)

Like how SM helps get to correct answer. 77.5% (134) 17.3% (30) 5.2% (9)

Like characters play and dance. 76.9% (133) 19.1% (33) 4.0% (7)

Able to log onto SM quickly. 53.2% (92) 36.4% (63) 10.4% (18)

5th

Like SM. 40.9% (108) 43.6% (115) 15.5% (41)

Like activities and games in SM. 49.2% (130) 40.2% (106) 10.6% (28)

Like how SM helps get to correct answer. 57.6% (152) 30.3% (80) 12.1% (32)

Like characters play and dance. 56.8% (150) 28.0% (74) 15.2% (40)

Able to log onto SM quickly. 52.7% (139) 39.4% (104) 8.0% (21)

Parentheses indicate number of student responses.


- 41 -

SuccessMaker Interventionist Attitudes Opinions about the SuccessMaker program were systematically collected from teachers on end-

of-year questionnaires and further developed during WebEx focus group sessions. All

SuccessMaker interventionists first received the focus group questions so they could digest them

and respond with their initial thoughts prior to their WebEx session. Representatives from the

research team facilitated each session. The sessions lasted approximately 60 minutes. Of the 24

SuccessMaker interventionists half attended one of four WebEx sessions and all returned their

responses to the initial questionnaire.

Each session was recorded and then transcribed verbatim, allowing the research team to compile

a large master file of participant responses. Following an exhaustive review of the teacher

responses, a two-dimensional coding system was developed to organize the responses.

Responses were categorized by Topic Area and Attitude. Topic area codes have a two digit

numeric format, with the first digit on the left indicating general topic category and the

remaining digits indicating a specific topic within each general category. The topic codes are

paired with either an ‘N’ to indicate neutral, a ‘+’ to indicate positive, or a ‘–‘ to indicate a

negative attitude toward an aspect of the program, or to indicate the tone of the comment.

The response to SuccessMaker was overwhelmingly positive, with 82% of all

responses coded as positive in nature.

The response to the program was overwhelmingly positive with 82% of the 1,061 recorded

comments coded as positive in nature. Interventionists felt that the program was a welcomed and

successful addition to their RTI efforts due to several features including its interactivity,

glossary, read-to-me audio, immediate feedback coupled with scaffolding and tutorials, and

customized coursework capability. Further, interventionists felt that the adaptive motion through

the content worked well (i.e., 78% positive, 14/18) and ultimately the program was an overall

good educational investment (i.e., 89% positive, 32/36).

The program’s reporting feature was also well-received with 92% of the 346 comments positive

in nature. Interventionists expressed an appreciation for how the program efficiently and

concisely demonstrates student progress in a way that is useful to interventionists and teachers.

Interventionists also very much liked that the activities, in conjunction with the reporting system,

allowed for effective one-to-one intervention instruction.

Interventionists believe that their students enjoy using the program. Interventionists were very

positive about their students’ interactions with the program. Of the 145 recorded comments

about the student response to SuccessMaker, 81% were positive in nature. It was felt that the

program successfully engaged and motivated students (i.e., 80% positive, 60 comments).

Additionally, it was indicated that the content of the SuccessMaker reading and math programs is

generally aligned with current curricula and just as appropriate for those students below grade

level as it is for others (i.e., 79%, 227 comments).

Interventionists did have some concerns with the fluency portion of the program and the initial

placement. Although comments regarding the initial placement were still overall positive (i.e.,

54%) it was felt that the placement was not accurate for all the students. Though placement

cannot be expected to be perfect, placement issues are easily dealt with by simply manually


- 42 -

moving students up or down. The fluency feature was not viewed positively (i.e., 35%, 43

comments). In addition to technical problems (i.e., sound levels, access to playback, not

recording), some students were initially embarrassed to read aloud into the microphone and

teachers noticed a decrease in interest among the students over time in the fluency portion of the

program.


- 43 -

IV. DISCUSSION

The research team recruited diverse 3rd

and 5th

grade intervention students from eighteen urban,

suburban, and rural schools districts in six different states (i.e., AZ, CA, KS, MI, OR, and TX).

The study sample included 292 3rd

grade (i.e., SuccessMaker = 154, comparison = 138) and 490

5th

grade (i.e., SuccessMaker = 239, comparison = 251) at-risk students that would benefit from a

well-conceived and implemented reading and math intervention.

Specifically, students were included if they tested below the 30th

percentile in reading fluency

and pacing or either math computation or problem solving. The samples from both grade levels

are largely non-Caucasian and free/reduced lunch eligible. Also, the 5th

grade sample, selected

on math achievement is more female than male. Each qualifying student was randomly assigned

to one of two study groups (i.e., comparison group of non-users v. SuccessMaker users). The 3rd

grade treatment group used the reading version of the program while the 5th

grade SuccessMaker

students used math.

All study students, both SuccessMaker users and comparison group students, still received their

required non-SuccessMaker intervention instruction as mandated by the district. A minority of

the study students from both groups received regular intervention instruction. In addition to

intervention instruction, the students in each study group received similar regular classroom

reading and math instruction. No new reading or math content was introduced when

SuccessMaker students where using the program.

SuccessMaker interventionists were expected to conduct one-to-one sessions where they review


week. Not all interventionist were able to conduct these sessions regularly with their

SuccessMaker students. Sixty percent received at least a single one-to-one session with an

average of six sessions per student.

Four schools had large SuccessMaker user to SuccessMaker interventionist ratios that did not

permit systematic individualized attention; these schools were considered Low Implementers.

Schools that conducted regular one-to-one program intervention sessions and/or had small ratios

(i.e., SMU/SMI < 14), and were able to deliver regular over-the-shoulder attention were

designated the High implementation group. It should be noted that those students receiving non-

SuccessMaker intervention instruction from both grades were from the High implementation

group.

Interventionists and students quickly became comfortable with the SuccessMaker program.

Interventionists found the program to be a valuable tool and a good use of time. Students

indicated they liked the program and all but a small minority found aspects they liked about the

program. The majority of students felt the questions were of medium difficulty; this of course, is

right where the offerings from the program should be.

SuccessMaker students used the program in an educational laboratory setting for at least four

twenty to thirty minute sessions each week. SuccessMaker students were productive on the

program with the large majority (i.e., 75%) of students logging at least 28 and 24 hours at 3rd

and

5th

grade, respectively.


- 44 -

Two challenging assessment batteries were administered to students to measure gains in

important areas of achievement. These areas were total reading including comprehension,

vocabulary and word reading scales, as well as, fluency/pacing and reading academic attitude.

Math scales included total math with concepts and communication, operations and computation,

and process and applications (i.e., problem solving), as well as, timed computation and concept

and application (i.e., problem solving) scales and math academic attitude.

The achievement data indicates clearly that diverse, at-risk populations of students can be

successful the first school year they are exposed, and their interventionists implement the

program. SuccessMaker students demonstrated large statistically significant achievement gains

on all achievement outcome measures. This was especially true for students receiving proper

implementation in the traditionally difficult to affect areas of comprehension and problem

solving.

The 3rd

grade SuccessMaker students saw statistically significantly larger gains in their

vocabulary than their comparison group counterparts, while the 5th


saw statistically significantly larger gains in computation and problem solving. Further, those 5th

grade SuccessMaker students coupling the program with regular individualized attention from

their interventionist saw statistically significantly larger gains in mathematics achievement

overall, as well as, on concepts, computation and problem solving scales.

Lastly, the study groups saw statistically equivalent gains in their academic attitude with the

exception of the 5th

grade Low implementation group which had statistically lower gains than

their comparison group counterparts.


- 45 -

A.1 Study Site Descriptions

This appendix summarizes the educational environment for each study site as well as a

demographic breakdown. This information is crucial for determining how applicable results

from this study may be to the consumers of this report.

Arizona District 1 School 1 This school resides in a midsize city and students are expected to follow a dress code. This

school and district is committed to partnering with their community, parents and families to

prepare students for higher education and for becoming productive citizens. In the 2009-10

school year, the district served a community of approximately 34,000. The median household

income was approximately $47,000 indicating a middle-class community.

This is a medium to large size school serving over nearly 750 students in grades four through

eight. The 5th

grade is made up of approximately 180 students. This school is primarily

Hispanic, which represents 72% of the school’s population. African American students represent

14% and Caucasian students represent 9% of the student population. Asian/Pacific Islander,

American Indian/Alaskan, and multi-racial students make up the remaining 5% of the student

population. This school falls into the high range for participation in the nation’s free or reduced-

price lunch program with 92% of students eligible to receive free or reduced-price lunch.

Approximately 25% of the students are designated as not English proficient.

The percentage of 5th

grade students testing at standard in mathematics in the 2011-12 school

year was 47%, 16% lower than the statewide results. In reading, the percentage of 5th

grade

students testing at standard in the 2011-12 school year was 21% lower, at 57%, than the

statewide results. The student/teacher ratio is approximately 20 to 1. The district adopted a

widely published elementary basal mathematics curriculum with a copyright date of 2010. All

teachers had strict adherence to the district adopted program. Daily math blocks range from 120-

125 minutes.

One interventionist participated in the SuccessMaker study with one 5th

grade intervention class.

The interventionist was trained approximately two months after school began, on October 4th

.

This teacher also received additional training sessions in December 2012 and April 2013. Only

SuccessMaker students were in the computer lab, and they did not receive one-to-one instruction.

The SuccessMaker students started using the program on a daily basis for 30 minute sessions on

November 5, 2012. The students’ last week using the program was the week of April 29, 2013.

The median 5th

grade student used the math program approximately 23 hours, attempting

approximately 25 exercises every twenty minutes with a success rate of 66%. The median 5th

grader mastered 98% of the presented skills, with a total of 158 math skills mastered. The

baseline testing was administered the third week of November 2012. The end-of-year testing

was given during the second week of May 2013.

Arizona District 1 School 2 This school resides in a midsize city. Students are expected to follow a dress code. This school

is committed to helping every child reach their full potential by providing children with an

environment that will help develop them academically and personally. In the 2009-10 school


- 46 -

year, the district served a community of 64,000. The median household income was

approximately $37,000 indicating a lower middle-class community.

This is a medium to large size school serving just over 700 students in grades pre-kindergarten

through eight. The 3rd

and 5th

grade levels have approximately 70 students in each grade. This

school is primarily Hispanic, which represents 66% of the school’s population. Caucasian

students represent 16% and African American students represent 11% of the student population.

Asian/Pacific Islander, American Indian/Alaskan, and multi-racial students make up the

remaining 7% of the student population. With 89% of the students eligible to receive free or

reduced-price lunch, this school falls into the high range for participation in the nation’s free or

reduced-price lunch program. Approximately 26% of the students are designated as not English

proficient.

The percentage of 3rd


year was 50%, 19% lower than the statewide results. In 5th

grade the percentage of students

testing at standard in mathematics during this school year was 7% lower than the statewide

results at 56%. In reading, the percentage of 3rd

grade students testing at standard for the 2011-

12 school year was 13% lower, at 62%, than the statewide results. The percent of 5th

grade

students testing at standard in reading was 69%, 9% lower than the statewide results. The

student/teacher ratio is approximately 16 to 1.

While teachers followed district standards and topics, their curriculum for 3rd

grade reading

consisted of adhering to the Daily 5 Framework while pulling from a variety of basal resources.

Conversely, for 5th

grade, the district adopted a widely published elementary basal mathematics

curriculum with a copyright date of 2010. All teachers had strict adherence to the district

adopted program. Daily English Language Arts blocks ranged from 75-125 minutes, while daily

math blocks were 80 minutes in length.

One interventionist, with the help of a parent volunteer, participated in the SuccessMaker study

with one 3rd

and one 5th

grade intervention class. The interventionist was trained approximately

two months after school began, on October 4th

. This teacher also received additional training

sessions in December 2012 and April 2013.

Both grade levels had only SuccessMaker students in the lab for program use. The interventionist

provided both 3rd

and 5th

grade students with one-to-one instruction on the program on a regular

basis. The SuccessMaker students in 3rd

grade utilized the program four days per week for 20

minute sessions starting on October 29, 2012. The SuccessMaker students in 5th

grade utilized

the program four days per week for 20 minute sessions starting on December 10, 2012. The

students’ last week using the program was the week of April 29, 2013.

The median 3rd

grade student used the reading program approximately 29 hours, attempting

roughly 19 exercises every twenty minutes with a success rate of 66%. The median 5th

grade

student used the math program approximately 25 hours, attempting approximately 29 exercises

every twenty minutes with a success rate of 66%. The median 3rd

grader mastered 62% of the

presented skills, with a total of 28 reading skills mastered. The median 5th

grader mastered 96%

of the presented skills, with a total of 225 math skills mastered. The baseline testing for 3rd

grade

was administered the third week of November 2012 and the 5th

grade was administered the first

week of December 2012. The end-of-year testing was given during the third week of May 2013.


- 47 -

Arizona District 2 School 1 This school is located in a large city. This research site prides itself in providing a creative and

innovative learning environment that encourages students to think critically and reach their

highest potential. In the 2009-10 school year, this school served a community of over 62,000.

The median household income was approximately $34,000 indicating a middle-class community.

This is a medium to large size school serving close to 700 students in kindergarten through grade

eight. The 3rd

and 5th

grades are made up of approximately 75 students each. This school is

primarily Hispanic, which represents 94% of the school’s population. Caucasian students make

up 2%, while African American students make up 3%. The remaining 1% of the population is

comprised of American Indian and Asian/Pacific Islander students. One hundred percent of the

students at this school are eligible for free or reduced-price lunch, which places this school into

the high range for participation in the nation’s free or reduced-price lunch program.

Approximately 47% of the students are designated as not English proficient.



year was 43%, which is 26% lower than the statewide results of 69%. In 5th

grade, the

percentage of students testing at standard in mathematics during this school year was 43%, 20%

lower than the statewide results. In reading, the percentage of 3rd

grade students testing at

standard for the 2011-12 school year was 17% lower, at 58%, than the statewide results. The

percent of 5th

grade students testing at standard in reading was 52%, 26% lower than the

statewide results. The student/teacher ratio was approximately 17 to 1.

The district adopted a widely published elementary basal reading curriculum for 3rd

grade with a

copyright date of 2006, and a math curriculum for 5th

grade with a copyright date of 2010. Third

grade teachers adhered primarily to the adopted curriculum, but also utilized outside readers and

worksheets. Fifth grade teachers had strict adherence to the district adopted program. Daily

English Language Arts blocks ranged from 160-170 minutes, while daily math blocks were 90

minutes in length.

One interventionist participated in the SuccessMaker study with one 3rd

and 5th

grade

intervention class. The interventionist was trained on October 26, 2012, which was

approximately two months after school began. There were also two follow-up trainings, one in

December 2012 and another in April 2013. Both grade levels had only SuccessMaker students

in the lab for program use. The interventionist provided both 3rd

and 5th

grade students with one-

to-one instruction on the program on a regular basis. The SuccessMaker students in both 3rd

and

5th

grade utilized the program simultaneously four days each week for 30 minutes each day for a

total of 120 minutes each week. Third grade started the program on October 29th

, while 5th

grade

began one month later on November 26th

. The students’ last week using the program was the

week of May 13, 2013.

The median 3rd


nearly 18 exercises every twenty minutes with a success rate of 67%. The median 5th

grade

student used the math program approximately 27 hours, attempting roughly 23 exercises every

twenty minutes with a success rate of 66%. The median 3rd

grader mastered approximately 66%

of the presented skills, with a total of approximately 35 reading skills mastered. The median 5th

grader mastered 98% of the presented skills, with a total of 138 math skills mastered.


- 48 -

The baseline testing for 3rd

grade was administered between the second and third week of

November 2012 and the 5th

grade was administered the fourth week of November 2012. The

end-of-year testing was given during the fourth week of April 2013.

California District 1 School 1 This school resides in a large suburb. This is an economically disadvantaged site where most

students come from low income families. In addition, teachers are challenged with large classes

and not enough staff. In the 2009-10 school year, this district served a community of 33,000.

The median household income was approximately $47,000 indicating a middle-class community.

This is a medium size school serving close to 550 students in grades kindergarten through five.

The 3rd

grade is made up of approximately 100 students, while 5th

grade is made up of

approximately 50 students. This school is primarily Hispanic, which represents 94% of the

school’s population. Asian/Pacific Islander students make up 5% of the student population,

while the remaining 1% is made up of Caucasian students. Ninety-three percent of the students

at this school are eligible for free or reduced-lunch, which places this school into the high range

for participation in the nation’s free or reduced-price lunch program. Approximately 64% of the

students are designated as not English proficient.

In mathematics, 52% of 3rd

grade students tested at standard in the 2011-12 school year, as

compared to 69% testing at standard statewide. The percentage of students in 5th

grade testing at

standard in mathematics during this school year was 35%, 30% lower than the statewide results.

In reading, the percentage of 3rd

grade students testing at standard for the 2011-12 school year

was 24%, which is 24% lower than the statewide results. The percent of 5th

grade students

testing at standard in reading was 22% lower than the statewide results of 63%. The



grade, with a

copyright date of 2003, and a math curriculum for 5th

grade, with a copyright date of 2009. Both

3rd

and 5th

grade teachers had strict adherence to the district adopted program. Daily English

Language Arts blocks were 135 minutes, while daily math blocks were 70 minutes in length.

One interventionist participated in the SuccessMaker study with one 3rd

and 5th

grade

intervention class. The SuccessMaker initial training for this site occurred on September 10,

2012, which was approximately two weeks after school began. Follow-up trainings occurred in

December 2012 and April 2013.

Both grade levels had only SuccessMaker students in the lab for program use. The interventionist

provided both 3rd

and 5th

grade students with occasional one-to-one instruction on the program.

The SuccessMaker students in both 3rd

and 5th

grade utilized the program five days each week

for 30 minutes each day. Third grade started the program on October 29th

, while 5th

grade began

one week later on November 5th

. The students’ last week using the program was the week of

May 20, 2013.

The median 3rd



grade

student used the math program approximately 55 hours, attempting 33 exercises every twenty


- 49 -

minutes with a success rate of 63%. The median 3rd

grader mastered 62% of the presented skills,

with a total of 45 reading skills mastered. The median 5th

grader mastered 93% of the presented

skills, with a total of approximately 343 math skills mastered.

The baseline testing was administered the fourth week of November 2012. The end-of-year

testing was given during the last two week of May 2013.

California District 1 School 2 This school resides in a large suburb. This site values strong leadership, as evidenced by a high

level of involvement and support by the principal. Students at this school come from lower

socio-economic backgrounds and tend to struggle academically. In the 2009-10 school year, the

district served a community of just over 43,000. The median household income was

approximately $54,000 indicating a middle-class community.

This is a medium size school serving a little fewer than 600 students in grades kindergarten

through six. The 3rd

grade is made up of approximately 90 students, while 5th

grade is made up of

approximately 75 students. This school is primarily Hispanic, which represents 88% of the

school’s population. Asian/Pacific Islanders comprise 11% of the school’s population, with

Caucasian representing less than 1%. Ninety-five percent of this school’s students are eligible to

receive free or reduced-price lunch, which places the school in the high range for participation in

the nation’s free or reduced-price lunch program. Approximately 66% of the students are

designated as not English proficient.



year was 61%, which is 8% lower than the statewide average of 69%. The percentage of students

in 5th

grade testing at standard in mathematics during this school year was 62%, 3% lower than

the statewide results. In reading, the percentage of 3rd

grade students testing at standard for the

2011-12 school year was 22%, which is 26% lower than the statewide results. The percent of 5th

grade students testing at standard in reading was 39%, which is 24% lower than the statewide

results of 63%. The student/teacher ratio is approximately 21 to 1.

The district adopted a 3rd

grade reading curriculum (2003©) and a 5th

grade elementary basal

mathematics curriculum (2009©) from widely known publishers. Teachers at both grade levels

had strict adherence to the district adopted program. Daily English Language Arts blocks were

100 minutes, while daily math blocks were 65 minutes in length.

Initially two interventionists participated in the SuccessMaker study overseeing one 3rd

and one

5th

grade intervention class. However, during the 2nd

semester one interventionist was removed

due to health related concerns. The open position was covered by a substitute teacher and the

principal for the remainder of the year. The interventionists were trained approximately two

weeks after school began, on September 10, 2012. These interventionists also received

additional training sessions in December 2012 and April 2013.

Both grade levels had only SuccessMaker students in the lab for program use. No one-to-one

instruction was provided for either grade level. The SuccessMaker students in both 3rd

and 5th

grade utilized the program simultaneously five days per week for 20 minute sessions starting on

October 3rd

, 2012 for 3rd

grade and October 8, 2012 for 5th

grade. The students’ last week using

the program was the week of May 20, 2013.


- 50 -

The median 3rd



grade student used the math program approximately 36 hours, attempting roughly 33 exercises



presented skills, with a total of nearly 40 reading skills mastered. The median 5th

grader mastered

93% of the presented skills, with a total of 249 math skills mastered.

The baseline testing was administered the last week of October 2012. The end-of-year testing

was given during the fourth week of May 2013.

California District 2 School 1 This school is located in a large suburb. This site, along with the district, is technology driven

and future oriented. In fact, this school was recently recognized as an Apple Distinguished

Program as a result of their 1 to 1 laptop program. In addition, the district was recognized by

Greatschools.org as one of the 10 greatest improving school districts in their respective state. In

the 2009-10 school year, the district served a community of approximately 12,000. The median

household income was approximately $69,000 indicating an upper middle-class community.

This is a small to medium size school serving approximately 450 students in grades kindergarten

through five. This school is primarily Hispanic, which represents 93% of the school’s

population. Caucasian students make up 4% and students with multiple ethnicities make up 3%

of the student population, while Asian/Pacific Islander and African American students comprise

less than 1%. Fifty-nine percent of this school’s students are eligible to receive free or reduced-

price lunch, which places the school in the medium range for participation in the nation’s free or

reduced-price lunch program.



year was 76%, 7% higher than the statewide results. In 5th

grade, the percentage of students

testing at standard in mathematics was 49%, which is 16% lower than the statewide average of

65%. In reading, the percentage of 3rd

grade students testing at standard for the 2011-12 school

year was 10% lower, at 38%, than the statewide results. The percent of 5th

grade students testing

at standard in reading was 54%, 9% lower than the statewide results. The student/teacher ratio is

approximately 19 to 1.

The district adopted a commonly used elementary basal reading curriculum published in 2003

for 3rd

grade. Fifth grade utilized a math curriculum published in 2002, also from a widely used

publisher. Third grade has several mixed grade classrooms, which precludes the teachers from

adhering strictly to the basal curriculum; however, the 5th

grade teachers tend to follow their

district adopted program closely. English Language Arts blocks were 100 minutes daily, while

daily math blocks were 65 minutes in length.

The 3rd

and 5th

grade students received intervention from SuccessMaker simultaneously four

days each week for 20 minutes sessions. The interventionist’s initial training occurred on

October 22, 2012, approximately three weeks after school started, with follow-up trainings in

December 2012 and May 2013.


- 51 -

Both grade levels had only SuccessMaker students in the lab for program use; however, the lab

was actually part of a large multi-purpose room in which several different classes were being

held simultaneously. While each class was separate, there was other activity in the room. The

interventionist provided both 3rd

and 5th

grade students with one-to-one instruction on the

program on a sporadic basis. Third grade began using SuccessMaker on October 25th

, while 5th

grade began on November 7, 2012. The students’ last week using the program was the week of

May 27, 2013.

The median 3rd

grade student used the reading program 31 hours, attempting approximately 16

exercises every twenty minutes with a success rate of 65%. The median 5th

grade student used

the math program approximately 31 hours, attempting nearly 31 exercises every twenty minutes

with a success rate of 66%. The median 3rd

grader mastered 62% of the presented skills, with a

total of 29 reading skills mastered. The median 5th


with a total of 234 math skills mastered.


grade was administered the last week of November 2012 and the 5th

grade was administered the third week of November. The end-of-year testing was given during

the first week of June 2013.

California District 3 School 1 This school resides in a large suburb. Teachers and administration at this site make a strong

effort to recognize students who achieve. This is illustrated by their Accelerated Reader program,

which awards charms to students making gains in their reading and honors those that reach

significant goals in reading. They also have a Character Education Program that, on a monthly

basis, introduces students to different positive character traits, educates them about this trait, and

then recognizes one student from each class for exhibiting this character. In the 2009-10 school

year, the district served a community of approximately 45,000. The median household income

was approximately $63,000 indicating an upper middle-class community.

This is a small to medium size school serving close to 475 students in grades kindergarten

through five. This school is primarily Hispanic, which represents 77% of the school’s population.

Asian/Pacific Islander students make up 14% of the student population, while Caucasian students

make up 6%, African Americans make up 3% and students with multiple ethnicities make up less

than 1%. 67% of this school’s students are eligible to receive free or reduced-price lunch, which

places the school in the medium range for participation in the nation’s free or reduced-price

lunch program. Approximately 22% of the students are designated as not English proficient.





testing at standard in mathematics was 80%, which is 15% higher than the statewide average of



year was equal to the statewide results of 48%. The percent of 5th


standard in reading was 72%, 9% higher than the statewide results. The student/teacher ratio is

approximately 20 to 1.

The district adopted a widely published elementary basal reading curriculum (2009©) for 3rd

grade, and a math curriculum (2009©) for 5th

grade. Both 3rd

and 5th

grade teachers had strict


- 52 -

adherence to the district adopted program. English Language Arts blocks were 115 minutes

daily, while daily math blocks ranged from 80-85 minutes in length.

There was one 3rd

and one 5th

grade intervention class receiving SuccessMaker at this site. The

principal was trained as the interventionist for both classes. The initial training occurred on

August 28th

, one week after school started, with follow-up trainings in November 2012 and April

2013. Both grade levels had SuccessMaker and non-SuccessMaker students in the lab during

intervention time. The third grade lab held 34 students, nine of which were SuccessMaker and

the 5th

grade lab held 35 students, 15 of which were SuccessMaker. One-to-one instruction was

provided occasionally. The SuccessMaker students in both 3rd

and 5th

grade utilized the program

four days each week for 30 minutes sessions. Third grade began using SuccessMaker on October

3rd

, while 5th

grade began on October 15, 2012. The students’ last week using the program was

the week of May 6, 2013.

The median 3rd



grade





grader mastered 93%

of the presented skills, with a total of 104 math skills mastered.


grade was administered the third week of October 2012 and the 5th

grade was administered between the second and third week of October 2012. The end-of-year

testing was given during the fourth week of May 2013.

Kansas District 1 School 1 This school is located in a large suburb and students are expected to follow a dress code. This

site is a high performing Catholic school that belongs to the Archdiocese. In the 2009-10 school

year, the district served a community of approximately 9,000. The median household income

was approximately $101,000 indicating an upper-class community.

This a medium to large size school serving just over 650 students in grades pre-kindergarten

through eight. The 3rd

and 5th

grade levels are made up of approximately 70 students each. This

school is primarily Caucasian, which represents 91% of the school’s population. Hispanic

students make up 5%, Asian/Pacific Islander students make up 2%, and students with multiple

ethnic backgrounds make up 1% of the population. None of this school’s students are eligible to

receive free or reduced-price lunch, which places the school in the low range for participation in

the nation’s free or reduced-price lunch program. The student/teacher ratio is approximately 17

to 1.


grade and

math curriculum for 5th

grade; both with copyright dates of 2011. Teachers at both grade levels

followed the basal curriculum with some supplementation. Daily English Language Arts blocks

were 70 minutes, while daily math blocks ranged from 45-60 minutes in length.

Two interventionists, one for each 3rd

and 5th

grade, participated in the SuccessMaker study. The

initial training occurred on October 3, 2012, a little more than one month after school started,

with follow-up trainings occurring in December 2012 and February 2013.


- 53 -

Both grade levels had only SuccessMaker students in the lab for program use. The

interventionists provided both 3rd

and 5th


program on an as-needed basis. Students were identified for one-to-one instruction based on their

level of need. The 3rd

grade SuccessMaker students used the program 75 minutes each week in

three 25 minute sessions. Fifth grade utilized SuccessMaker four 20 minute weekly sessions.

Third and 5th

grade began using the program on October 30th

and 31st respectively. The students’

last week using the program was the week of May 27, 2013.

The median 3rd



grade




presented skills, with a total of roughly 42 reading skills mastered. The median 5th

grader

mastered 99% of the presented skills, with a total of approximately 155 math skills mastered.

The baseline testing was administered the last week of October 2012. The end-of-year testing

was given to 3rd

grade during the last week of April 2013 and to the 5th

grade the third week of

April 2013.

Kansas District 1 School 2 This school resides in a large suburb and students are expected to follow a dress code. This site is

a high performing Catholic school that belongs to the Archdiocese. In the 2009-10 school year,

the district served a community of a little fewer than 20,000. The median household income was


This is a medium size school serving close to 550 students in grades kindergarten through eight.

The 3rd

grade level is made up of approximately 60 students and the 5th

grade level is made up of

approximately 70 students. This school is primarily Caucasian, which represents 78% of the

school’s population. Hispanic students make up 11%, African American students make up 2%,

and students with multiple ethnic backgrounds make up 9% of the population. None of this

school’s students are eligible to receive free or reduced-price lunch, which places the school in

the low range for participation in the nation’s free or reduced-price lunch program. The



for 3rd


publisher. Neither 3rd

nor 5th

grade follow the basal program closely. Third grade teachers pull

from a variety of basal programs and other materials, but they do follow the Daily 5 Framework.

Teachers in 5th

grade have a list of the skills necessary for state testing, and they follow the

district adopted curriculum as needed to cover these skills. English Language Arts blocks range

from 105-120 minutes daily, while daily math blocks range from 60-90 minutes in length.

There is one interventionist at this site that oversees the 3rd

and 5th

grade use of SuccessMaker.

Their initial training occurred on October 3, 2012, a little over one month after school started,

with follow-up trainings in December 2012, February and May 2013. Both grade levels had only

SuccessMaker students in the lab for program use. The interventionist sporadically provided both

3rd

and 5th

grade students with one-to-one instruction on the program. The SuccessMaker


- 54 -

students in both 3rd

and 5th

grade utilized the program four days each week for 30 minutes

sessions. Third grade began using SuccessMaker on October 15th

, while 5th

grade began on

October 30, 2012. The students’ last week using the program was the week of May 27, 2013.

The median 3rd



grade

student used the math program approximately 27 hours, attempting 26 exercises every twenty

minutes with a success rate of 72%. The median 3rd


with a total of 53 reading skills mastered. The median 5th

grader mastered 97% of the presented

skills, with a total of approximately 187 math skills mastered.

The baseline testing was administered to the 3rd

grade the first week of October 2012 and to the

5th

grade the fourth week of October 2012. The end-of-year testing was given during the first

week of May 2013.

Michigan District 1 School 1 This school resides in a rural setting. It prides itself in helping students to develop as leaders. In

fact, they’ve been recognized as a Leader in Me School, which means that they teach students

about essential skills of leadership and how to implement those skills. This teaching is based on

the book The 7 Habits of Highly Effective People written by Stephen Covey. In the 2009-10

school year, the district served a community of just over 6,000. The median household income

was approximately $77,000 indicating a upper middle-class community.

This is a medium size school serving approximately 600 students in grades kindergarten through

five. The 3rd

and 5th

grade levels are made up of approximately 100 students each. This school

is primarily Caucasian, which represents 93% of the school population. Hispanic students and

multiple ethnicity students each comprise 2% of the school’s population, with American

Indian/Alaskan Native, African American and Asian making up the remaining 3%. This school

falls into the low range for participation in the nation’s free or reduced-price lunch program with

28% of students eligible to receive free or reduced-price lunch.




grade the percentage of students

testing at standard in mathematics during this school year was 18% higher than the statewide

results at 64%. In reading, the percentage of 3rd

grade students testing at standard for the 2012-

13 school year was 15% lower, at 81%, than the statewide results. The percent of 5th

grade

students testing at standard in reading was 74%, 4% higher than the statewide results. The


This site does not have a district adopted basal curriculum for reading in 3rd

grade. Alternatively,

the teachers pull from a variety of basal programs and materials for English Language Arts, but

follow a balanced literacy, rotation based approach. The district adopted a widely published 5th

grade elementary basal mathematics curriculum with a copyright date of 2007; however, teachers

supplement 40-60% of the time. Daily English Language Arts blocks range from 45-60 minutes,

while daily math blocks were 60 minutes in length.

Two interventionists were trained on September 25, 2012, three weeks after school started. This

site also had a follow-up training in December 2012 and then contacted the trainer on an as


- 55 -

needed basis for further support. Both grade levels had only SuccessMaker students in the lab

for program use. The interventionist provided both 3rd

and 5th

grade students with one-to-one

instruction on the program on a regular basis. The SuccessMaker students in 3rd

grade used the

program three days each week for 25 minute sessions. Fifth grade students attended four 25

minute weekly sessions. Both grade levels started using SuccessMaker on October 22, 2012. The

students’ last week using the program was the week of May 13, 2013.

The median 3rd







grader mastered 98%


The baseline testing was administered to the 3rd

grade the third week of October 2012 and to the

5th

grade the first week of October 2012. The end-of-year testing was given to the 3rd

grade

during the fourth week of May 2013 and to the 5th

grade the last week of May 2013.

Michigan District 2 School 1 This school resides in a large suburban area. This school is making an important shift in

education to focus on learning as opposed to putting an emphasis on teaching. The teachers have

identified areas where students have the greatest deficits and have decided to make it a school-

wide goal to focus on learning in these areas to support student development. In the 2009-10

school year, the district served a community of approximately 47,000. The median household

income was approximately $50,000 indicating a middle-class community.

This is a medium to large size school serving close to 700 students in grades five and six. The 5th

grade level is made up of approximately 330 students. This school is primarily Caucasian, which

represents 89% of the school population. African American, Hispanic, Asian and multiple

ethnicity students make up the remaining 11% of the student population. This school falls into

the low range for participation in the nation’s free or reduced-price lunch program with 33% of

students eligible to receive free or reduced-price lunch.

The percentage of 5th


year was 58%, 12% higher than the statewide results. In reading, the percentage of 5th

grade

students testing at standard for the 2012-13 school year was 5% higher, at 75%, than the

statewide results. The student/teacher ratio is approximately 18 to 1. The district adopted a

widely published elementary basal mathematics curriculum with a copyright date of 2007. All

teachers had strict adherence to the district adopted program. Daily math blocks range from 60-

90 minutes.

Two interventionists participated in the SuccessMaker study. These interventionists were trained

approximately one week after school began, on September 12, 2012. These teachers also

received additional training sessions in December 2012 and March 2013. There were three

intervention labs each day and each lab had both SuccessMaker and non-SuccessMaker students.

The ratio of SuccessMaker students to total students in each lab was 7 to 21, 14 to 22 and 6 to

25. One-to-one instruction was not provided to the SuccessMaker students. Starting November


- 56 -

19, 2012 there were three daily SuccessMaker sessions at this site, with each session lasting 25

minutes. The students’ last week using the program was the week of May 27, 2013.

The median 5th

grade student used the math program approximately 28 hours, attempting 27

exercises every twenty minutes with a success rate of 63%. The median 5th

grader mastered 94%

of the presented skills, with a total of 160 math skills mastered. The baseline testing was

administered the third week of October 2012. The end-of-year testing was given during the last

week of May 2013.

Michigan District 2 School 2 This school resides in a large suburban area. This site is committed to continually improving

their school, staff and learning environment for students. They value professional development

and using newly acquired information to enhance learning. In the 2009-10 school year, the

district served a community of approximately 47,000. The median household income was


This is a small school serving just over 350 students in grades kindergarten through four. The 3rd

grade level is made up of approximately 75 students. This school is primarily Caucasian, which

represents 88% of the school population. African American, Hispanic, Asian and multiple

ethnicity students make up the remaining 12% of the student population. This school falls into

the low range for participation in the nation’s free or reduced-price lunch program with 34% of

students eligible to receive free or reduced-price lunch.



year was 48%, which is 7% higher than the statewide average of 41%. The percentage of 3rd

grade students testing at standard in reading in the 2012-13 school year was 73%, which is 7%

higher than the statewide average of 66%. The student/teacher ratio is approximately 18 to 1.

The district adopted a widely published elementary basal language arts curriculum with a

copyright date of 2009. All teachers had strict adherence to the district adopted program. Daily

reading blocks were 90 minutes in length. One interventionist participated in the SuccessMaker

study from this site. The interventionist was trained approximately one week after school began,

on September 12, 2012. This teacher also received additional training sessions in December

2012 and March 2013.

Only SuccessMaker students were in the intervention lab. The interventionist provided students

with one-to-one instruction on the program on a regular basis. Starting November 5, 2012 this

site held four sessions per week at 25 minutes each session. The students’ last week using the

program was the week of May 27, 2013. The median 3rd

grade student used the reading program

approximately 26 hours, attempting roughly 21 exercises every twenty minutes with a success

rate of 71%. The median 3rd

grader mastered 73% of the presented skills, with a total of 37

reading skills mastered.

The baseline testing was administered during the third week of October 2012. The end-of-year

testing was given during the last week of May 2013.


- 57 -

Oregon District 1 School 1 This school is located approximately 25 miles outside of a large metropolitan area. This site

promotes collaboration within the school, and also with parents to promote a supportive learning

environment for the students. In the 2009-10 school year, the district served a community of

approximately 30,000. The median household income was is approximately $65,500 indicating

a upper middle-class community.

This is a small school serving just over 350 students in grades kindergarten through five. The 3rd

and 5th

grade levels are made up of approximately 60 students each. This school is comprised of

two primary ethnic groups, Caucasian and Hispanic, which represent a total of 50% and 45% of

the school population respectively. Asian, African American, American Indian/Alaskan, and

multiple ethnicity students make up the remaining 5% of the student population. This school

falls into the high range for participation in the nation’s free or reduced-price lunch program with

77% of students eligible to receive free or reduced-price lunch.



year was 88%, which is 9% higher than the statewide average of 79%. The percentage of 5th

grade students testing at standard in mathematics in the 2009-10 school year was 93%, which is

14% higher than the statewide average of 79%. The percentage of 3rd


standard in reading in the 2009-10 school year was 90%, which is 7% higher than the statewide

average of 83%. The percentage of 5th

grade students testing at a standard in reading in the 2009-

10 school year was 87%, which is 10% higher than the statewide average of 77%. The

student/teacher ratio is approximately16 to 1.


for 3rd


publisher. Teachers at both grade levels had strict adherence to the district adopted curriculum.

English Language Arts blocks range from 75-245 minutes daily, while daily math blocks were

120 minutes in length.

One interventionist oversees both 3rd

and 5th

grade implementation of SuccessMaker. At

approximately the mid-year point, the site changed interventionists. The initial training occurred

on September 13, 2012, a little over one week after school started, with follow-up trainings in

November 2012 and March 2013. Both grade levels had only SuccessMaker students in the lab

for program use. No one-to-one instruction was provided. The SuccessMaker students in both 3rd

and 5th

grade utilized the program five days each week for 15 minutes sessions. Third grade

began using SuccessMaker on October 8th

, while 5th

grade began on October 15, 2012. The


The median 3rd

grade student used the reading math program approximately 22 hours, attempting


grade





grader mastered 96%


The baseline testing was administered the fourth week of October 2012. The end-of-year testing



- 58 -

Oregon District 1 School 2 This school is located approximately 25 miles outside of a large metropolitan area. This site is

small, but the teachers and administration view their size as a strength to provide an

individualized educational experience for each student. Building strong relationships with

families, community and students promotes a collective responsibility for student success. In the

2009-10 school year, the district served a community of approximately 30,000. The median

household income was is approximately $65,500 indicating an upper middle-class community.

This is a small school serving just over 400 students in grades kindergarten through five. The 3rd

and 5th

grade levels are made up of approximately 70 students each. This school is comprised of

two primary ethnic groups, Caucasian and Hispanic, which represent a total of 69% and 22% of

the school population respectively. Asian, African American, and multiple ethnicity students

make up the remaining 9% of the student population. This school falls into the mid-range for

participation in the nation’s free or reduced-price lunch program with 50% of students eligible to

receive free or reduced-price lunch.

In mathematics, 66% of 3rd

grade students tested at standard in the 2009-10 school year, as

compared to 79% testing at standard statewide. The percentage of students in 5th

grade testing at

standard in mathematics during this school year was 60%, 19% lower than the statewide results.

In reading, the percentage of 3rd

grade students testing at standard for the 2009-10 school year

was 71%, which is 12% lower than the statewide results. The percent of 5th

grade students

testing at standard in reading was 15% lower than the statewide results of 77%. The


The district adopted a widely published elementary basal reading curriculum (2008©) for 3rd

grade, and math curriculum (2012©) for 5th

grade. Both 3rd

and 5th

grade teachers had strict

adherence to of the district adopted program. English Language Arts blocks were 75 minutes

daily, while daily math blocks ranged from 60-75 minutes in length.

There was one interventionist at this site and she received the initial training on September 13,

2012, a little over one week after school started. Follow-up trainings occurred in November 2012

and March 2013. Both grade levels had only SuccessMaker students in the lab for program use.

The interventionist provided both 3rd

and 5th


program on a regular basis. Both grade levels used SuccessMaker four days each week with 3rd

grade having 30 minute sessions and 5th

grade utilizing the program in 25 minute sessions. Third

grade began using SuccessMaker on October 11th

, while 5th

grade began on October 25, 2012.

The students’ last week using the program was the week of May 20, 2013.

The median 3rd







grader mastered 96%


The baseline testing was administered during the third and fourth weeks of October 2012. The

end-of-year testing was given during the last week of May 2013.


- 59 -

Texas District 1 School 1 This school resides in a rural area. This school places a high emphasis on the individual learner.

They are committed to meeting individual student learning needs by offering a variety of

instructional programs. As a result of this type of commitment this site has been recognized for

their its strong educational programs and quality education. In the 2009-10 school year, the

district served a community of approximately 2,000. The median household income was is


This school is small, serving approximately a little more than 400 students in grades pre-

kindergarten through five. The 3rd

and 5th

grade levels are made up of approximately 60 students

each. Hispanic students make up 69% of the school population. Caucasian students make up the

next largest proportion at 21%. African American students make up the other 10%. This school

falls into the high range for participation in the nation’s free or reduced-price lunch program with

75% of students eligible to receive free or reduced-price lunch. Approximately 19% of the

students are designated as limited English proficient.



year was 78%, which is 9% lower than the statewide results of 87%. In 5th

grade, the percentage

of students testing at standard in mathematics during this school year was 88%, 7% higher than

the statewide results. In reading, the percentage of 3rd

grade students testing at standard for the

2010-11 school year was 8% lower, at 81%, than the statewide results. The percent of 5th

grade

students testing at standard in reading was 87%, 5% higher than the statewide results. The



for 3rd


publisher. Neither 3rd

nor 5th

grade follow the basal program closely. Third grade teachers pull

from a variety of programs using a balanced literacy approach. Teachers in 5th

utilize their basal

curriculum approximately 70% of the time. English Language Arts blocks were 90 minutes in

length and the 5th

grade math blocks were 70 minutes daily.

Two interventionists, one for each grade level, from this site participated in the study. The initial

training occurred on August 29, 2012, two days after school started, with follow-up trainings in

November 2012 and April 2013.

Only SuccessMaker students were present in the lab for program use. The interventionists

provided both 3rd

and 5th

grade students with one-to-one instruction regularly; however, this

instruction did not start until April 2013. The SuccessMaker students in both 3rd

and 5th

grade

utilized the program four days a week, simultaneously for 20 minutes sessions. Third grade

began using SuccessMaker on October 16, 2012 while 5th

grade began on October 22, 2012. The


The median 3rd



grade





- 60 -

presented skills, with a total of approximately 44 reading skills mastered. The median 5th

grader

mastered 96% of the presented skills, with a total of approximately 165 math skills mastered.

The baseline testing was administered the first week of November 2012. The end-of-year testing


Texas District 1 School 2 This school is located in a large suburban area. This school offers a diverse population of

students. In an effort to be sensitive to their multi-cultural students, the school plans special

activities around events such as black History Month, Cinco de Mayo and other occasions that

promote diversity. In the 2009-10 school year, the district served a community of approximately

13,000. The median household income was approximately $97,000 indicating an upper-class

community.

This is a small school that serves a little fewer than 400 students in grades pre-kindergarten

through five. The 3rd


grade level

is made up of approximately 65 students. This school is primarily Hispanic, which represents

88% of the school population. African American students make up 10% of the population, while

Caucasian and American Indian/Alaskan students make up the remaining 2%. This school falls

into the high range for participation in the nation’s free or reduced-price lunch program with

93% of students eligible to receive free or reduced-price lunch. Approximately 33% of the

students are designated as limited English proficient.



year was 92%, which is 5% higher than the statewide average of 87%. The percentage of

students in 5th

grade testing at standard in mathematics during this school year was 96%, 15%

higher than the statewide results. In reading, the percentage of 3rd


standard for the 2010-11 school year was 83%, which is 6% lower than the statewide results.

The percent of 5th

grade students testing at standard in reading was 94%, which is 12% higher

than the statewide results of 82%. The student/teacher ratio is approximately 13 to 1.


for 3rd


publisher. Third grade primarily uses other materials as opposed to the basal text. While the 5th

grade teachers only used their basal curriculum 30-50% of the time. English Language Arts

blocks were 110 minutes daily, while daily math blocks were 90 minutes in length.

Two interventionists, one assigned to each grade level, oversee SuccessMaker implementation.

The initial training occurred on August 29, 2012, two days after school started, with follow-up

trainings in November 2012 and April 2013. Only SuccessMaker students were present in the

lab for program use. The interventionists provided both 3rd

and 5th

grade students with one-to-

one instruction regularly; however, this instruction did not start until April 2013. The

SuccessMaker students in 3rd

and 5th

grade utilized the program simultaneously, daily for 30

minute sessions. Both grade levels began using SuccessMaker on October 18, 2012. The



- 61 -

The median 3rd



grade





grader mastered 96%


The baseline testing was administered between the last two weeks of October 2012. The end-of-

year testing was given during the last two weeks of May 2013.

Texas District 1 School 3 This school resides in a large suburban area. This site believes that success begins with the

student. Students are recognized and rewarded for their various successes. In addition, the school

makes a strong effort to foster strong leadership skills, positive citizenship, and good social skills

through a variety of programs developed to specifically address these characteristics. In the

2009-10 school year, the district served a community of approximately 11,800. The median

household income was approximately $81,000 indicating an upper-class community.

This is a medium size school, serving just over 600 students in grades pre-kindergarten through

five. The 3rd


grade level is made

up of approximately 70 students. This school is primarily Hispanic, which represents 80% of the

school population. African American students make up 18% of the population, while Caucasian

American Indian/Alaskan, and students with multiple ethnicities make up the remaining 2%.

This school falls into the high range for participation in the nation’s free or reduced-price lunch

program with 94% of students eligible to receive free or reduced-price lunch. Approximately

33% of the students are designated as limited English proficient.



year was 77%, 10% lower than the statewide results. In 5th


testing at standard in mathematics was 94%, which is 13% higher than the statewide average of



year was 82%, which is 7% lower than the statewide results. The percent of 5th

grade students

testing at standard in reading was 94%, 12% higher than the statewide results. The



for 3rd


publisher. Teachers in 3rd

grade follow the district skills/standards list in order to be ready for

regular benchmark testing. Fifth grade teachers incorporate small groups into their lessons, but

also utilize the basal math curriculum. English Language Arts blocks range from 60-120 minutes

in length and the 5th

grade math blocks were 90 minutes daily.

One interventionist for 3rd

grade and two interventionists for 5th

grade participated in the study.

The initial training occurred on August 29, 2012, two days after school started, with follow-up

trainings in November 2012 and April 2013. Both grade levels had only SuccessMaker students

in the lab for program use. The interventionists provided 3rd

grade with regular one-to-one

instruction; however, 5th

grade did not start this instruction until April 2013. The SuccessMaker

students in both 3rd

and 5th

grade utilized the program daily for 20 minute sessions in separate


- 62 -

computer labs. Third and 5th

grade began using SuccessMaker on October 1, 2012. The


The median 3rd



grade student used the math program approximately 48 hours, attempting approximately 27

exercises every twenty minutes with a success rate of 61%. The median 3rd

grader mastered 81%

of the presented skills, with a total of 51 reading skills mastered. The median 5th

grader mastered

89% of the presented skills, with a total of 250 math skills mastered.

The baseline testing was administered the second week of October 2012. The end-of-year

testing was given during the fourth week of May 2013.

Texas District 2 School 1 This school is located in a large suburban area. This site welcomes an involved community and

encourages parental involvement and participation in the school. In the 2009-10 school year, the

district served a community of approximately 56,000. The median household income was

approximately $36,000 indicating a lower middle-class community.

This a medium to large size school serving just over 650 students in grades pre-kindergarten

through four. The 3rd

grade level is made up of approximately 90 students. This school is

primarily Hispanic, which represents 92% of the school’s population. Caucasian students make

up 6%, Asian/Pacific Islander and African American students make up the remaining 2% of the

population. Approximately 57% of the students are designated as limited English proficient.

Information regarding student eligibility for free or reduced-price lunch was not available for this

site.



year was 79%, which is 8% lower than the statewide average of 87%. The percentage of 3rd

grade students testing at standard in reading in the 2010-11 school year was 92%, which is 3%

higher than the statewide average of 89%. The student/teacher ratio is approximately 17 to 1.

The district adopted a widely published elementary basal language arts curriculum with a

copyright date of 2011. The 3rd

grade teachers at this site generally do not follow the basal

curriculum, but instead use it more as a supplement. They prefer to take a balanced literacy

approach and utilize leveled reading groups. Daily reading blocks range from 45-60 minutes in

length.

One interventionist participated in the SuccessMaker study from this site. The interventionist

was trained on September 12, 2012, approximately two weeks after school began. Follow-up

trainings occurred in November 2012 and May 2013. Only SuccessMaker students were in the

lab for program use. No one-to-one instruction was provided to the students. They started using

SuccessMaker on October 1, 2012. This usage plan for this site varied from week to week;

however, each session was set for 30 minutes and they used SuccessMaker at least three times

each week. The students’ last week using the program was the week of May 6, 2013.

The median 3rd

grade student used the reading program approximately 26 hours, attempting 15

exercises every twenty minutes with a success rate of 75%. The median 3rd

grader mastered 65%


- 63 -

of the presented skills, with a total of 24 reading skills mastered. The baseline testing was

administered the third week of October 2012. The end-of-year testing was given during the third

week of May 2013.

Date post:	23-Jun-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

PEARSON SUCCESS MAKER RESPONSE TO INTERVENTION STUDY · PEARSON SUCCESS MAKER RESPONSE TO...

Documents