An Examination of the Promise of the NumberShire Level 1 Gaming Intervention for Improving Student Mathematics Outcomes Hank Fien, Christian T. Doabler, Nancy J. Nelson, Derek B. Kosty, Ben Clarke & Scott K. Baker Journal of Research on Educational Effectiveness, 9:4, 635-661, DOI: 10.1080/19345747.2015.1119229 Published online: 06 Jan 2016. Funding: Development of a Game-based Integrated Learning and Assessment System to Target Whole Number Concepts (Project NumberShire), Award Number R324A120071 Source: Institute of Education Sciences. National Center for Special Education Research. And This research was supported in part by Project NumberShire 1, Grant No. EDIES11C0026, a subcon- tract with ThoughtCycle through the U.S. Department of Education, Institute of Education Sciences, Small Business Innovation Research Program. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.
An Examination of the Promise of theNumberShire Level 1 Gaming Intervention forImproving Student Mathematics Outcomes
Hank Fien, Christian T. Doabler, Nancy J. Nelson, Derek B. Kosty, Ben Clarke& Scott K. Baker
To cite this article: Hank Fien, Christian T. Doabler, Nancy J. Nelson, Derek B. Kosty, Ben Clarke& Scott K. Baker (2016) An Examination of the Promise of the NumberShire Level 1 GamingIntervention for Improving Student Mathematics Outcomes, Journal of Research on EducationalEffectiveness, 9:4, 635-661, DOI: 10.1080/19345747.2015.1119229
To link to this article: http://dx.doi.org/10.1080/19345747.2015.1119229
Accepted author version posted online: 06Jan 2016.Published online: 06 Jan 2016.
An Examination of the Promise of the NumberShire Level 1Gaming Intervention for Improving Student MathematicsOutcomes
Hank Fiena, Christian T. Doablera, Nancy J. Nelsona, Derek B. Kostyb, Ben Clarkea, andScott K. Bakera
ABSTRACTThe purpose of this study was to test the promise of the NumberShireLevel 1 Gaming Intervention (NS1) to accelerate math learning for first-grade students with or at risk for math difficulties. The NS1 interventionwas developed through the Institute of Education Sciences, SmallBusiness Innovation Research Program (Gause, Fien, Baker, & Clarke,2011) as a digitally based technology tool to allow educators tointervene early and strategically with students struggling to learnmathematics. This study used a randomized controlled trial design totest the promise of the NS1 intervention. In total, 250 first-gradestudents were randomly assigned within classrooms to the treatmentcondition or a control condition. Results indicate significant effectsfavoring the treatment group on proximal measures of whole-numberconcepts and skills. Intervention effects were not statistically significantfor distal outcome measures. Treatment effects were not moderated byspecial education or English learner status; however, the condition byinitial skill level interaction approached significance. Additionally, therewas no relationship between dosage variables and students’ response tothe intervention. Limitations and future directions for research arediscussed.
KEYWORDSinterventiongamingmath
National assessments indicate that the majority of U.S. students are struggling to acquireknowledge of mathematical concepts and skills necessary to become proficient in mathemat-ics. Results of the 2013 National Assessment for Educational Progress (NAEP) indicate thatonly 42% of fourth-grade students score at or above Proficient in mathematics, and 17% areperforming below Basic, suggesting that many students struggle to meet grade-level expecta-tions in mathematics (National Center for Education Statistics [NCES], 2014). Difficulties inmathematics are even more pronounced for students with disabilities. An alarming 45% ofall fourth-grade students identified with a disability scored below Basic on the 2013 NAEPin mathematics (NCES, 2014). Likewise, a staggering 41% of English language learners(ELLs) scored below Basic on the 2013 NAEP.
CONTACT Hank Fien [email protected] University of Oregon, Center on Teaching and Learning, 1600 Millrace,Suite 206, 1211 University of Oregon, Eugene, OR 97403-1211, USA.aUniversity of Oregon, Eugene, Oregon, USAbOregon Research Institute, Eugene, Oregon, USA
Evidence also suggests that students who exhibit mathematics difficulties (MD) early intheir formal schooling are likely to continue to struggle throughout the elementary years (Bod-ovski & Farkas, 2007; Morgan, Farkas, & Wu, 2009). Several recent longitudinal studies havedemonstrated that mathematics achievement trajectories are relatively stable and are estab-lished as early as kindergarten and first grade (Bodovski & Farkas, 2007; Morgan et al., 2009;Morgan, Farkas, & Wu, 2011). In an analysis of the Early Childhood Longitudinal Study-Kin-dergarten (ECLS-K) Cohort data set, Morgan et al. (2009) found that students who displayedmathematics difficulties in both fall and spring of kindergarten subsequently had the lowestgrowth trajectories through the end of fifth grade. Whereas mathematics achievement gaps areevident for important subgroups of students early in their schooling, evidence suggests thatthese gaps are widening over time. For example, the mathematics achievement gap representedas average scores between ELLs and non-ELLs is 25 points in fourth-grade (219 for ELLs and244 for non-ELLs). By eighth grade the mathematics achievement gap nearly doubles to a 41-point gap (246 for ELLs and 287 for non-ELLs; NCES, 2014).
Arguably, early intervention is a viable approach to prevent these gaps in mathematicslearning and should be orchestrated before these gaps become apparent (Clarke, Baker,et al., 2015; Morgan et al., 2009, 2011), using instructional approaches designed to supportstudents at risk for MD. Converging evidence supports the use of explicit instruction as aviable approach for preventing early mathematics difficulties (Gersten, Beckmann, et al.,2009, National Mathematics Advisory Panel [NMAP], 2008). Researchers and policymakersrecommend that such explicit instruction approaches occur in an early detection, preven-tion-oriented support system, such as models of Response to Intervention (RtI; Gersten,Beckmann, et al., 2009; National Association of State Directors of Special Education[NASDSE], 2006). Typically RtI models promote high-quality instruction and universalscreening for all students, with layers of increasingly intensive “tiers” of support for studentsas the level of student’s instructional need increases. Student progress is monitored fre-quently to determine whether intervention efforts are intensive enough to adequately sup-port students to reach important academic outcomes.
Unfortunately, a major limitation of current instructional practice in mathematics is thesignificant lack of evidence-based, teacher-directed, explicit instruction for students with orat risk for MD. Although the evidence behind providing students with or at risk for MDwith daily, explicit, teacher-directed mathematics instruction is quite strong (Gersten, Chard,et al., 2009), a recent population-based, longitudinal study revealed a limited use of this evi-dence-based practice in first-grade classrooms (Morgan, Farkas, & Maczuga, 2015). Duringa month of mathematics instruction, which averaged five days per week and 54 minutes perday, there was an average of only 11.8 instances of teacher-directed instruction, a surpris-ingly meager dose, considering the instructional practice has strong empirical support toimprove the mathematics achievement of struggling learners (Gersten, Beckmann, et al.,2009). Furthermore, Morgan and colleagues (2015) found no relation between the numberof students with MD in a classroom and the use of teacher-directed instruction. In otherwords, teachers, in general, are not differentiating instruction based on the student composi-tion of their particular classroom.
Against this backdrop, we developed the Numbershire Level 1 Gaming Intervention(NS1) as a school-based, digital learning tool designed to accelerate mathematics learningfor students struggling to develop an understanding of whole-number concepts and skills.NS1 was conceptualized to serve as a supplemental intervention that can be employed within
636 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
an RtI service delivery model (e.g., used in Tier 2 as a supplement to core mathematicsinstruction: Gersten, Beckmann, et al., 2009). We designed NS1 as a tool for teachers toincrease the availability of explicit and systematic mathematics instruction for students withor at risk for MD. To increase the mathematics achievement of students with or at risk forMD, we integrated the science of explicit instruction and a focus on critical mathematicscontent in the early grades (i.e., whole-number concepts) with emerging game-based learn-ing design principles (Doabler & Fien, 2013; Klopfer, Osterweil, & Salen, 2009) that have thepotential to engage students in learning opportunities.
Promise of Education Technology and Current Research
There is ostensibly great promise for leveraging both education technology and learninggames to improve learning outcomes. Indeed, there is no shortage of theoretical, conceptual,and practical reasons put forth in the literature for the wide-scale adoption of educationtechnology in U.S. classrooms (Atkins et al., 2010). Although there is a major push to inte-grate technology products with regular classroom practice (Atkins et al., 2010), an importantmissing element is a solid evidentiary basis for wide-scale dissemination of existing products(Dynarski et al., 2007). Moreover, there are relatively few rigorous programs of researchto develop and study technology tools and interventions in the area of early mathematics(Doabler, Fien, Nelson-Walker, & Baker, 2012; Young et al., 2012). For example, the WhatWorks Clearinghouse (WWC) has reviewed nearly 100 elementary mathematics programsto date, a quarter of which are technology programs. Of the technology programs reviewed,only a few have research studies that meet WWC screening criteria to determine the level ofevidence available and the degree of program effectiveness. Less than 10% of technology pro-grams reviewed have any research that can be used to evaluate program efficacy. Of thatsmall subset of reviewable programs, only two (Odyssey Math and Dreambox Learning)demonstrated positive or potentially positive effects on mathematics achievement for stu-dents in elementary school. This has bearing for schools and teachers looking for supple-mentary mathematics interventions that effectively differentiate instruction according to thestandards they are tasked to address.
In addition, results of large-scale evaluations of education technology tools have beenmixed, at best. For example, Dynarski et al. (2007) conducted a large-scale cluster-random-ized controlled trial, randomly assigning 428 teachers to one of 16 reading or mathematicstechnology programs or control classrooms that did not have access to either set of technol-ogy programs. The researchers were interested in testing the average effect of teachers havingaccess to education technology tools, not the effect of any particular technology program.Overall, compared to control-group classrooms, test scores were not significantly higher inclassrooms using the selected reading and mathematics software programs.
In a follow-up to the Dynarski et al. (2007) study, Campuzano, Dynarski, Agodini, andRall (2009) sampled 176 classrooms from the original sample of 428 classrooms to furtherstudy the effects of 10 technology programs on student learning. Of the 10 programs studied,the researchers found a small, significant effect for only one program in one grade level: LeapFrog for fourth-grade reading (Hedges’s gD .09). There were no significant effects on mathe-matics learning for any of the mathematics programs (Campuzano et al., 2009). Together,the Dynarski et al. (2007) and Campuzano et al. (2009) results suggest that, despite thecapacity of technology to create dynamic, individualized learning opportunities for students,
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 637
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
the potential benefits of technology are not being realized in classrooms, especially thoseteaching mathematics.
Theoretical and Empirical Support for the NS1 Intervention Components
We believe a primary reason for the general lack of positive outcomes for education technol-ogy programs to improve student learning is a major disconnect between cutting-edge tech-nology and empirically validated instructional design features. For example, mathematicscontent and instructional design features that have been demonstrated as efficacious inprint-based curricula for students with or at risk for MD could be carefully integrated witheducation technology as one approach to realize the potential for education technology tools(Baker, Gersten, & Lee, 2002; Clarke, Baker, Chard, Smolkowski, & Fien, 2008; Clarke,Baker, & Fien, 2009; Dede, 2009; National Council of Teachers of Mathematics [NCTM],2006; NMAP, 2008). In this context, we hypothesize that the careful integration of research-based instructional design principles, early mathematics content, and gaming technologycould improve student mathematics achievement. Therefore, NS1 incorporates three designcomponents: (a) evidence-based, explicit instructional design and delivery features (Baker,Fien, & Baker, 2010; Coyne, Kame’enui, & Carnine, 2011; Doabler et al. 2012); (b) criticalearly mathematics content focused on key whole-number concepts and skills identified inthe Common Core State Standards for Mathematical Practice (CCSS-M; National GovernorsAssociation Center for Best Practices & Council of Chief State School Officers, 2010; Gers-ten, Beckmann, et al, 2009); and (c) a highly engaging gaming platform (Dede, 2009). Eachof these components is described in detail below.
Evidence-Based Explicit Instructional Design and Delivery Features
A converging body of empirical evidence suggests that explicit, systematic mathematicsinstruction significantly enhances the mathematics achievement of students with or at riskfor difficulties and disabilities and provides them with meaningful access to critical mathe-matics content (Baker et al., 2002; Gersten, Beckmann, et al., 2009; Gersten, Chard, et al.,2009; Kroesbergen & Van Luit, 2003; NMAP, 2008). For example, Gersten, Chard et al.(2009) conducted a meta-analysis and reviewed 41 studies that focused exclusively on stu-dents with learning disabilities and only included intervention studies that employed ran-domized controlled trials or strong quasi-experimental research designs. The authors foundthat explicit instruction had the largest impact, g D 1.22, 95% CI [0.78, 1.67], among sevendimensions of math instruction.
Likewise, Kroesbergen and Van Luit (2003) conducted a meta-analysis that includedinterventions that targeted a broader range of students with special needs, including at-riskstudents, students with learning disabilities, and low-achieving students. In a review thatincluded 58 mathematics intervention studies that employed within-subject (e.g., single-casedesigns) and between-subject research designs (e.g., RCTs and quasi-experiments), theauthors found that teacher-led, direct instruction was one of the most effective means forsupporting mathematics learning for kindergarten and elementary-aged students, particu-larly for learning foundational math concepts and skills.
Baker et al. (2002) conducted a research syntheses using meta-analytic techniques includinga set of 15 intervention studies that targeted low-achieving, school-aged students. The authors
638 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
only included studies that used experimental or quasi-experimental group research designs.Results demonstrated an aggregate weighted effect size of d D .58, 95% CI [0.40, .77] formathematics interventions that used an explicit instructional approach. More recently, theauthors of the IES Practice Guide, Assisting Students Struggling with Mathematics: Response toIntervention for Elementary and Middle Schools (Gersten, Beckmann, et al., 2009), concludedthat “explicit and systematic instruction” in both Tier 2 and Tier 3 contexts had a strong levelof evidence. This evidence was based on findings from six randomized controlled trials(RCTs) that demonstrated various components of explicit instruction were present in interven-tions that were shown to be efficacious for students with and at risk for MD. In summary, anumber of meta-analyses, research syntheses, and RCTs have demonstrated positive effects forexplicit instruction to effectively teach students across grade levels and across varying samplesof students (i.e., low achieving, at risk, students with learning disabilities).
NS1 was carefully developed using explicit instructional design and delivery principles(Coyne et al., 2011) drawn from these meta-analyses and research syntheses focused on lowachievers and students with MD (Baker et al., 2002; Gersten, Chard, et al., 2009). The goal ofexplicit instruction is to present and communicate new information in a manner that isunambiguous and easy to understand (Archer & Hughes, 2011). Explicit instruction makesextensive use of teacher modeling and overt think-alouds to illustrate key concepts and thesuccessful use of skills and strategies. Such approaches also make the otherwise obscure cog-nitive processes used by proficient learners during skill acquisition apparent to at-risk learn-ers (Gersten, Chard, et al., 2009). The explicit instructional framework of NS1 centers on thefollowing three interrelated design principles.
Utilize Instructional Scaffolding
Instructional scaffolding is a strategy used to provide support to students at critical pointsduring instruction through the systematic introduction and integration of key concepts,skills, and strategies (Chard & Jungjohann, 2006). For example, instruction will initiallybegin with simpler instructional examples and progress to more complex examples overtime as students demonstrate conceptual understanding. As students progress in theirunderstanding of concepts, supports are progressively and systematically faded to promotelearner independence (Coyne et al., 2011). In NS1, for example, instruction on new andcomplex mathematics content begins with virtual representations (e.g., place-value models)and transitions to more abstract representations (i.e., numbers) as students demonstrateproficiency.
Provide Opportunities for Guided Practice, Feedback, and Review
Often, students have difficulty performing procedural skills because they have not yet mas-tered prerequisite concepts in the instructional sequence (Hudson & Miller, 2006) and havenot been taught to a high criterion level of performance (Coyne et al., 2011). Therefore, tobecome proficient in the application of newly taught skills and strategies, students need mul-tiple opportunities to practice with immediate, highly specific, academic feedback (Clarke,Doabler, Nelson, & Shanley, 2015; Doabler et al., 2015; Gersten, Beckmann, et al., 2009).Research on mathematics instruction indicates that frequent, well-designed, guided practiceopportunities help students develop conceptual knowledge and attain automaticity with
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 639
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
critical skills and procedures (Kilpatrick, Swafford, & Findell, 2001). In NS1, guided andindependent practice opportunities are also deliberately sequenced to gauge initial learningand ongoing retention of new and previously learned concepts and skills (Coyne et al.,2011). Practice opportunities, as operationalized in NS1, consist of a student responding to amathematics-related question or task. For example, in an activity focused on using symbols(i.e., <, >, D ) to compare two-digit numbers, NS1 would record a practice opportunity forevery attempt made by a student to determine the relative magnitude of a targeted number.
Provide Differentiated Support to Learners
The benefits of providing differentiated and intensive support to students who struggle withearly mathematics concepts has strong empirical support (Clarke, Doabler, et al., 2015; Doa-bler & Fien, 2013; Gersten, Beckmann, et al, 2009). NS1 employs a Differentiated LearningPathway (DLP) to individualize and intensify instruction for students struggling to mastermathematics content. The DLP precisely calibrates instruction matched to students’ skill lev-els based on metrics of their mathematics performance, operationalized by latency and accu-racy scores in game activities. The DLP has the capacity to make within- and between-activity adjustments based on student performance data. These adjustments reroute studentsto optimal activities for improving procedural skills and building conceptual knowledge. Forexample, if the player reaches a performance criterion of 90% or greater on an independentpractice activity, she continues on the default gameplay pathway, following the standardscope and sequence. If the player scores 75%–89% accuracy, she is routed to the additionalpractice pathway, which provides the player with additional practice on a previously mas-tered, prerequisite skill designed to build knowledge required to access the target skill in thedefault pathway. If the player scores less than 75% accuracy, she is routed to the additionalinstruction and guided practice pathway, a more intensive, individualized experience thatprovides reteaching and practice with the target skill.
Critical Early Mathematics Content Focused on Whole-Number Concepts
There is consensus among experts that mathematics interventions in the early grades shouldfocus intensely on building understanding of whole-number concepts (Gersten, Chard,et al., 2009). Therefore, NS1 was strategically designed to build number sense and facilitateproficiency in three whole-number concept domains specified by the CCSS-M (NGA Centerfor Best Practices & CCSSO, 2010): Counting and Cardinality, Number and Operations inBase Ten, and Operations and Algebraic Thinking.
Counting and Cardinality
Counting and cardinality refers to students’ knowledge of the relationship between numbersand quantities and thus lays the foundation for building students’ number sense (NGA Cen-ter for Best Practices & CCSSO, 2010). As a broader construct, number sense encompasses achild’s fluidity with and flexibility in using and manipulating numbers, ability to performmental mathematics, and capability to make quantitative comparisons without difficulty(Berch, 1998; Gersten & Chard, 1999). Number sense in combination with numeration skills(e.g., number identification, one-to-one correspondence, counting, understanding of the
640 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
number line) sets the stage for students to solve foundational arithmetic problems. The earlylessons of NS1 target topics from the Counting and Cardinality domain to prime students’background knowledge and better ensure their success with more advanced topics addressedin the later lessons. For example, in the initial lessons of NS1, students learn how to “counton” from numbers other than 1 to gain proficiency with efficient and sophisticated countingstrategies for solving computational problems, such as number combinations (Gersten, Jor-dan, & Flojo, 2005; Fuchs et al., 2010).
Number and Operations in Base Ten
Key to understanding whole numbers is recognizing the convention and structure of the baseten system (Cawley, Parmar, Foley, Salmon, & Roy, 2001; NGA Center for Best Practices &CCSSO, 2010; Van de Walle, 2001). To develop knowledge of place value, NS1 teaches con-cepts such as ten-to-one relationships, the position of digits in two-digit numbers to deter-mine their value, and the groupings of ones and tens to compose and decompose two-digitnumbers. Mastery of these place-value concepts and skills represents a critical bridge torelated mathematics topics. For instance, NS1 explicitly teaches students how to composeand decompose two-digit numbers (e.g., 16 is made up of 1 ten and 6 ones) so that they canacquire the conceptual groundwork for solving multidigit addition and subtractionproblems.
Operations and Algebraic Thinking
The third domain emphasized in NS1 focuses on Operations and Algebraic Thinking. Toextend students’ understanding of whole numbers, NS1 promotes fluency of number combi-nations within 20, understanding of the equal sign and properties of operations, and solvingword problems. Because students with or at risk for MD typically struggle to solve mathe-matics word problems (Bryant & Bryant, 2008), a primary focus of NS1 is to promote arobust understanding of word problem solving by teaching the structural features underlyingdifferent problem types. For example, students are taught how to solve word problems thatrequire one- and two-step solutions and involve unknowns (e.g., addends) in all positions.
Gaming Platform
Motivation and engagement techniques for fostering the development of mathematics profi-ciency are particularly important for young students with and at risk for MD because theyhave often experienced a long line of “failure and frustration with math” (Gersten, Beck-mann, et al., 2009, p. 44). Expert panels, therefore, recommend that Tier 2 and 3 mathemat-ics interventions include motivational strategies such as (a) reinforcing or praising studentsfor their effort and engagement in mathematics lessons and (b) rewarding student accom-plishment (Gersten, Beckmann, et al., 2009; Woodward et al., 2012). The NS1 gaming plat-form was designed to increase students’ engagement and motivation to learn mathematicsby situating mathematics learning experiences within a rich and engaging narrative arc. Forexample, NS1 sessions are set in a Renaissance-style, fairy tale-inspired village called Num-berShire. In the game, players assume the role of a young member of the village and engagein brief mathematics activities focused on building proficiency with whole numbers.
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 641
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
Gameplay allows players to interact with key NS1 characters and receive effort- and perfor-mance-based rewards as they succeed in solving mathematics problems, such as individual-izing the attributes and attire of their gameplay character (i.e., avatar).
Purpose of the Study
Our NS1 research and development work has been enveloped in a robust line of incrementalresearch activities (Gause et al., 2011). In the early years of our Small Business InnovativeResearch (SBIR) project, a primary aim was to develop prototypes of the intervention usingan iterative design framework (Clements, 2007; Doabler et al., 2015). Once the lessons werefully developed and compiled into a complete intervention, we then assessed the feasibilityand usability of NS1 through a series of implementation studies (Nelson et al., 2014). Vari-ous data sources, such as professional and preferential feedback from teachers and students,were collected during these implementation studies and used to guide major revisions of theintervention. Our next objective, and the focus of the current study, was to test the promiseof NS1 under rigorous experimental conditions. Due to the need to conduct a final round ofintervention revisions at the end of this pilot study and our SBIR funding, we were limitedto testing only a portion of the NS1 intervention (i.e., 8 weeks of the full 12-week interven-tion). Therefore, the primary goal of this manuscript is to report the promise of an abbrevi-ated version of NS1 to improve student proximal and distal mathematics outcomes.
Research Questions and Hypotheses
This study was guided by three research questions:1. What are the statistical and practical effects of NS1 on student mathematics outcomes?
We hypothesized that students in the NS1 treatment condition would outperform theirpeers in the control group on the proximal mathematics measure. Additionally, weanticipated that mean trends would favor the treatment group on the distal mathemat-ics measures, but those differences would not reach statistical significance. The ratio-nale for this hypothesis was based on the abbreviated dose of the intervention (8 weeksinstead of the full 12 weeks) and the relatively low to modest statistical power for thetargeted distal mathematics measures.
2. Is treatment effect moderated by prior achievement or English language learners (ELLs)status? NS1 was developed as a supplemental Tier 2 intervention; therefore, wehypothesized that it would benefit the entire treatment sample but be particularly ben-eficial for students at the upper end of the at-risk sample. Because the NS1 interventioncarefully controls for mathematics language and vocabulary, we anticipated that ELLswould benefit commensurately with their non-ELL peers. In other words, we expectedno moderation effect for ELL status on the treatment effect.
3. Given variability in treatment dosage, is there a relationship between treatment expo-sure and response to the intervention, or in other words, is there evidence of doseresponse? We expected a positive relationship between treatment exposure (i.e., mea-sured as number of student practice opportunities in the NS1 game) and studentresponsiveness to the NS1 intervention.
642 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
Method
Design
This study used a randomized controlled trial (RCT) design to test the promise of the NS1intervention. In total, 250 first-grade students were randomly assigned within classrooms tothe treatment condition (n D 125) or a control condition (n D 125). Students assigned tothe treatment condition received the NS1 intervention in addition to the core mathematicsinstruction provided in their first-grade classroom. For students randomly assigned to thecontrol condition, schools provided core mathematics instruction and a variety of supple-mental mathematics interventions. Student mathematics achievement data was collected atpretest and posttest and approximately every two weeks during intervention gameplay. Datafrom direct observations and project surveys were also analyzed.
Participants and Procedures
SchoolsIn the fall of 2013, the principal investigators invited nine schools from two school districtsin different regions of Oregon to participate in the study. All of the first-grade classrooms(n D 26) from the nine schools expressed interest in participating in the study. Eleven of theclassrooms were set in five Title I schools, located in a suburban school district (District A)in the second largest city in the state (Eugene, Oregon). In District A, 57% of studentsreceived free or reduced-price lunch, 19.6% received special education services, 3.1% wereconsidered ELLs, and 24.1% identified as ethnic minorities. The remaining 15 classroomswere set in four schools in a suburban school district (District B) located in the Portlandmetropolitan area. Of the four schools in District B, three received Title 1 funding. In Dis-trict B, 35.5% of students received free or reduced-price lunch, 13.2% received special educa-tion services, 14.8% were considered ELLs, and 39.3% identified as ethnic minorities.
TeachersTwenty-six certified teachers delivered core mathematics instruction in the participatingfirst-grade classrooms. Teachers, who were predominantly female and White, reported anaverage of 16.1 years of experience teaching mathematics and 9.5 years providing interven-tions for students at risk for MD. Teachers had formal training in education, with all teach-ers reporting coursework in mathematics education and nearly half reporting othergraduate-level coursework in mathematics. Teachers indicated that they had a range of expe-rience using technology in the classroom, including computers and laptops. Of the 26 teach-ers, only two reported using technology-based mathematics interventions in their currentclassrooms.
InterventionistsNS1 was facilitated by 10 interventionists, of which nine were district-employed instruc-tional assistants, and one was a regular parent volunteer at a participating school. Participat-ing interventionists were predominantly female, and just over half were White, while theremaining interventionists were Hispanic or Latino. They reported an average of 6.9 years ofexperience working in schools and 3.3 years providing interventions for students at risk forMD. The majority of interventionists did not have formal training in mathematics or
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 643
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
education. Interventionists indicated that they had varied experiences using technology, withnearly all reporting that they used computers to support instruction. One-third of interven-tionists reported using technology-based mathematics interventions prior to participating inthe study.
StudentsEligibility for the NS1 intervention used a three-stage process. First, 632 consented first-grade students enrolled in the 26 participating classrooms were screened in fall 2013. Stu-dents were screened using the fall benchmark of the easyCBM-Common Core State Stand-ards (easyCBM-CCSS; Alonzo, Tindal, Ulmer, & Glasgow, 2006) assessment. In eachclassroom, the 10 students with the lowest scores on the easyCBM-CCSS were identified asNS1-eligible and then matched in pairs according to their scores. Each participating class-room had five pairs of NS1-eligible students. For example, the two lowest performers on theeasyCBM-CCSS formed the first pair, while the students who were rank-ordered in the ninthand tenth positions formed the fifth pair. A total of 250 students were determined eligible forthe intervention. According to easyCBM CCSS national norms, approximately 60% of thesample of the eligible students (n D 151) scored at or below the 25th percentile, and 97% ofthe eligible students (n D 243) scored at or below the 50th percentile. The remaining sevenstudents scored between the 50th and 75th percentile.
Next, all NS1-eligible students were pretested on a battery of pretest measures, includingthe fall benchmark of the easyCBM-National Council of Teachers of Mathematics(easyCBM-NCTM; Alonzo et al., 2006) assessment, Group-Administered Missing Number(GA-MN), and Group-Administered Quantity Discrimination (GA-QD) measures (Doableret al., 2015), and a researcher-developed mastery assessment of the NS1 intervention. Fol-lowing pretesting, pairs of NS1-eligible students within each classroom were randomlyassigned to one of two conditions: treatment (i.e., NS1 intervention) or a control condition(i.e., “business as usual” mathematics interventions). One classroom was unable to complywith random assignment and was dropped from the study. Consequently, randomizationresulted in 125 students in each condition.
Table 1 displays student demographic information by condition, reported by school dis-tricts at the beginning of the study. Across both conditions, participating students were pre-dominately White, and the average age of students was 6.5 years. As displayed in Table 1,treatment and control conditions were similar in demographic characteristics.
Mathematics Instruction
Core Mathematics InstructionAll treatment and control students continued to receive district-approved core mathematicsinstruction during the eight-week study. Teachers reported that core mathematics instructionwas delivered, on average, one hour per day, five days per week. Across the participating first-grade classrooms, the types of instructional materials varied. Of the 26 classrooms, 16 used acommercially available mathematics program as a primary mode of instruction. The mostpopular curricula used were Everyday Mathematics (n D 12) and Saxon Mathematics (n D 2).Six of the classrooms used instructional materials developed by their respective school dis-tricts. Teachers reported that core mathematics instruction focused primarily on concepts andskills from the CCSS-M domains of Operations and Algebraic Thinking and Numbers and
644 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
Operations in Base Ten. The primary instructional formats used for core mathematics instruc-tion included teacher-led instruction, peer and group work, and independent student work.
NumberShire Level 1 InterventionNS1 is a game-based, Tier 2 mathematics intervention designed to support first-grade stu-dents with or at risk for MD in developing proficiency with whole-number concepts andskills. Three domains of whole numbers represented in the CCSS-M (NGA Center for BestPractices & CCSSO, 2010) are targeted in the intervention: (a) Counting and Cardinality, (b)Number and Operations in Base Ten, and (c) Operations and Algebraic Thinking. Specifi-cally, NS1 provides explicit instruction in rational counting, decomposition of numbers,sophisticated counting strategies, properties of operations, number combinations, multidigitaddition and subtraction, and word problem solving.
NS1 consists of 48 sessions, themed into 12 weeks of instruction (four sessions per week).Each session is designed to provide 15 minutes of instruction. In total, the intervention offersstudents 12 hours of individualized instructional gameplay.
To promote mathematics proficiency among at-risk learners, NS1 utilizes an explicitinstructional framework. The intervention offers explicit modeling to demonstrate exactlywhat students are expected to learn, scaffolded instruction to guide students through thelearning process, and independent practice to facilitate learner independence. Each sessionis organized into four mathematics activities: Warm-Up, Teaching Event, Assessment Event,and Wrap-Up. The purpose of the Warm-Up is to allow students to practice a previouslymastered concept or skill. The Teaching Event introduces new whole-number concepts andskills that are central to the session’s learning objective. For this part of the session, NS1characters provide vivid demonstrations of the targeted concept or skill and offer clearexplanations of how students are expected to interact with the activity. Students then prac-tice the skill with support from NS1 characters, before practicing the skill independently.The third and fourth activities, Assessment Event and Wrap-Up, provide extended practice
Table 1. Treatment and control condition comparison.
Project conditions Treatment Control
Distinguishing featuresSupplemental Intervention NumberShire Level 1 Business-as-Usual (BAU)Setting Computer lab Push-in and pull-out group settingsManagement Facilitated by instructional assistants Taught by teachers and instructional
assistantsDelivery Gaming platform In-personMaterials Technology-based Print-based with incidental use of
technologyFormat Individual Individual and small groupContent CCSS-M, whole number concepts CCSS-M, variety of skillsFocus Explicit instruction Other methodsMinutes 15 per session 15–30 per sessionFrequency 4 times per week 4 times per weekSupport Initial facilitation training, plus ongoing
support as neededTypical district supports for initial training
Shared featuresTreatment and Control Core math instruction delivered by classroom
teacherConsent, random assignment, incentivesAll project assessments at all time points
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 645
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
to review whole-number concepts and skills. All four activities include a variety of virtualmathematical representations (e.g., number lines, base-10 blocks), frequent practice oppor-tunities, and high-quality academic feedback to facilitate students’ procedural fluency andbuild conceptual understanding of whole-number concepts.
NS1 Session Delivery. The intent of this study was to deliver the NS1 intervention four daysper week for eight weeks. Students received NS1 in addition to the core mathematics instruc-tion provided in their respective first-grade classrooms. Trained interventionists were paid bythe project to facilitate the NS1 intervention in school computer labs. Based on the availabilityof interventionists and space in the computer labs, the number of treatment students whoreceived NS1 at one time varied by school, with intervention groups ranging between 5 and 25students. On average, treatment students received 15 minutes of NS1 instruction per session.
At the start of each NS1 session, interventionists had students use a project-assigned pass-word to sign on to the intervention. Once logged on, students selected their NS1 avatar andthen completed the session’s Warm-Up, which lasted an average of two minutes per session.The Warm-Up activities, which take place in the avatar’s village house, have a primary focus onbuilding students’ fluency with basic number combinations. During the number combinationactivities, students received academic feedback about their accuracy with the targeted problems.
After the Warm-Up, students encountered a 5–7 minute Teaching Event. To enact thisactivity, the student’s avatar selected an object or building within the village of Tally-Ho.The Teaching Event entailed an NS1 character introducing students to a new concept or skillby offering step-by-step demonstrations and detailed explanations. For example, in the sev-enth session, Thatcher Tom initially demonstrates how the value of digits in teen numbersdepends on their place in the target number (16 is made up of 1 ten and 6 ones). To keepstudents engaged in these teaching moments, students were tasked with assisting NS1 char-acters in setting up the demonstrations (e.g., by identifying the underlying problem struc-tures in Teaching Events focused on word problem solving).
Following the Teaching Event, students spent approximately 3–5 minutes in the Assess-ment Event and 2 minutes in the Wrap-Up activity. Combined, these final activities offeredstudents additional practice to build deep understanding of whole-number concepts andskills. The Assessment Event reviewed the concept or skill introduced in the previous ses-sion. Primary aims of the Wrap-Up activities included strategic counting and number writ-ing. At the conclusion of each session, the student’s avatar was allowed to select a virtualreward and use it enhance the aesthetics of their personalized Tally-Ho village. Figure 1 pro-vides screenshots of the Tally-Ho Village and characters from the game as well math modelsused within the game.
NS1 Training. Prior to the start of the study, all interventionists received four hours of pro-fessional development comprised of a two-hour training presentation delivered by researchstaff and two one-hour, site-based meetings with the project team. Professional developmentfocused on preparing interventionists to (a) efficiently facilitate intervention groups, (b)troubleshoot and solve technical problems with the computers (e.g., resetting a student’scomputer during a session), and (c) monitor student progression during gameplay. It isimportant to note that interventionists did not provide any instructional assistance duringthe gameplay sessions. Interventionists were shown how to help students with the sign-onprocess at the start of intervention sessions and actively monitor students during gameplay.
646 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
Project staff also familiarized interventionists with the structure of NS1, describing the math-ematical content contained within the intervention and the rationale behind the use of anexplicit instructional framework and gaming technology to teach mathematics.
Fidelity of Implementation. Project staff directly observed each NS1 intervention grouponce during the eight-week study using the Technology Observation Tool (TOT; Nelson &Doabler, 2013). The TOT is a researcher-developed, standardized protocol designed to assessfidelity of implementation of the NS1 intervention. Project staff observed and rated each ofthe intervention session sites (e.g., computer labs) on six items of implementation fidelity:(a) use of effective procedures at start of gameplay, (b) students use headphones duringgameplay, (c) student engagement, (d) active monitoring and classroom management, (e)troubleshooting of technological issues, (f) use of effective procedures at end of gameplay.All items were rated on a 4-point scale (1 D not present, 4 D highly present) and were aver-aged to compute an overall implementation fidelity score. The average fidelity ratings forinterventionists’ use of effective procedures at the start of the session, active monitoring dur-ing student gameplay, and use of effective procedures at the conclusion of the session were3.5 (SDD 1.1), 3.6 (SDD 0.7), and 3.2 (SDD 1.2), respectively. Interventionists also receivedan average rating of 2.6 (SD D 1.1) for troubleshooting technology issues during sessions.Observers rated students’ engagement during gameplay and use of headphones, a criticalcomponent of NS1, as 3.7 (SD D 0.7) and 3.2 (SD D 0.8), respectively. The average overall
Figure 1. Screenshots from the NumberShire Level 1 online interactive mathematics intervention game.
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 647
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
fidelity score was 3.3 (SD D 0.8), indicating moderate overall fidelity with substantial vari-ability between NS1 groups.
Metrics gathered during NS1 gameplay served as an additional measure of fidelity ofimplementation, including number of sessions completed, number of items completed, andlatency and accuracy in responding. Between pretest and posttest treatment students, onaverage, completed 18.6 game sessions or 4.7 weeks of gameplay (SD D 8.1 sessions, range D2 to 33 sessions) and repeated 12.8 game sessions (SD D 5.9, range D 2 to 24). During game-play treatment students completed an average of 499.8 practice opportunities (SD D 269,range D 41 to 1,156) and completed 69% of the practice opportunities correctly (SD D 12%,range D 37% to 90%). Project staff used gameplay metrics to track student progress throughgame sessions on a weekly basis and corresponded regularly with interventionists to providesupport when needed (e.g., when a student’s gameplay progression deviated from the stan-dard schedule of four sessions per week).
Control ConditionStudents who were randomly assigned to the control condition received “business-as-usual”Tier 2 mathematics intervention supports in addition to core instruction for the duration ofthe eight-week study. Reported mathematics interventions for control students included sev-eral commercially available mathematics programs, a district-developed core program,teacher-developed materials, and other intervention resources. Everyday Mathematics wasreported as the supplemental intervention program implemented in six classrooms. Touch-Math and SRA Explorations and Applications were each used as supplemental interventionprograms in two participating classrooms. Seven of the classrooms reported using a district-developed core program as their Tier 2 intervention. Teachers from the remaining class-rooms reported using a host of intervention resources, including mathematics facts work-sheets and a program that focused on calendar concepts to provide supplementalintervention to students in the control group.
Collectively, interventions for control students emphasized instruction in the CCSS-M,addressing the domains of Operations and Algebraic Thinking, Counting and Cardinality,and Number and Operations in Base Ten. Twelve of the teachers reported that the controlgroup interventions prioritized teaching students the names of numbers and the appropriatecount sequence. Other commonly taught skills were counting to tell the number of objects,comparing numbers, and understanding concepts of addition and subtraction. Workingwith numbers 11–19 to gain foundations for place value was the least prioritized topicreported for the control condition. All control-group interventions were teacher-led, butmany involved peer or independent work as part of the intervention. Teachers reported thatcontrol students in three classrooms received additional support with technology-basedinterventions, including the software program IXL and games available through a leadingpublishing company (SRA). Students in the control condition received an average of24 minutes of supplemental mathematics intervention per day, four times per week (seeTable 1 for distinguishing features of the NS1 treatment and control conditions).
Student Measures
Trained project staff administered a battery of student mathematics assessments during thecourse of the study. Researcher-developed measures were used to assess mathematics
648 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
learning proximal to the NS1 intervention, while other established measures of proceduralfluency and conceptual understanding were administered to assess general mathematics per-formance. Researcher-developed mastery tests were administered approximately every twoweeks to treatment and control students according to treatment students’ progress in theintervention. Student mathematics achievement data were collected at pretest and posttest.
easyCBM MathematicsEasyCBM Mathematics (Alonzo et al., 2006) is a standardized, individualized, computer-administered assessment for students in kindergarten through eighth grade. There are cur-rently two versions available commercially—an early version of the assessment developed toassess the National Council of Teachers of Mathematics Focal Points (easyCBM-NCTM),and a subsequent version of the assessment developed to measure performance in the CCSS-M (easyCBM-CCSS). EasyCBM-CCSS was administered at the beginning of the study as ascreening assessment to determine eligibility for NS1 because the majority of participatingschools were already using this assessment for benchmarking classroom-wide. Once NS1-eli-gible students were identified, the easyCBM-NCTM was administered as a pretest and post-test measure of general mathematics performance.
easyCBM-CCSSThe CCSS version (Wray, Alonzo, & Tindal, 2014) of the assessment at each grade levelincludes 48 items developed to assess the CCSS-M. Measures are untimed, but the estimatedadministration time is 18–30 minutes. Results from K–8 studies of the technical adequacy ofthe easyCBM-CCSS indicate measures have strong internal consistency (a D .90) and split-half reliability (.80 first half, .86 second half: Wray et al., 2014).
easyCBM-NCTMEach grade level of the NCTM version of the assessment includes 15 items to assess each ofthree NCTM focal points at each grade level. In first grade, the focal points are Numbersand Operations; Numbers, Operations, and Algebra; and Geometry. Measures are untimed,but the estimated administration time is 18–30 minutes. The internal consistency of themathematics measures in Grade 1 is strong (a range D .78–.89) and the concurrent validitycorrelation with the Terra Nova in first grade is .73 (Anderson et al., 2010).
Group-Administered Missing Number (GA-MN) and Group-Administered QuantityDiscrimination (GA-QD)The Group-Administered Missing Number (GA-MN) and Group-Administered Quantity Dis-crimination (GA-QD) measures (Doabler et al., 2015) were administered at pretest and posttest.GA-MN and GA-QD represent modified versions of the Early Numeracy–Curriculum BasedMeasurement subtests: Missing Number and Quantity Discrimination (Clarke & Shinn, 2004).The GA-MN and GA-QD are one-minute, fluency-based measures that assess strategic count-ing and magnitude comparison, respectively. Unlike traditional mathematics CBMs, whichrequire verbal student responses, GA-MN and GA-QD have students record their responsesthrough a written format. The GA-MN requires students to write in the missing number amonga string of three numbers (0–10), with the first, middle, or last number of the string missing(e.g., 5_7). The GA-QD requires students to circle the number in a pair (numbers 0–10) withthe higher value. Test–retest reliability for GA-MN and GA-QD is reported, respectively, at .85
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 649
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
(p < .001) and .87 (p D .003). Predictive validity coefficients for GA-MN and GA-QD rangefrom .45 to .67 with a first-grade mathematics measure (Doabler et al., 2015).
ProFusion-Revised (ProFusion-R)The ProFusion-R is a proximal measure of student mathematics learning that was developedand utilized in one of our previous intervention development projects (Clarke et al., 2014;Doabler et al., 2015). A combination of group and individually administered items assess stu-dent understanding of concepts in place value, decomposing and composing numbers, wordproblem solving, and single and multi-digit addition and subtraction, among other skills. Pre-dictive validity correlations between ProFusion-R and GA-MN and GA-QD measures (Doa-bler et al., 2015) range from .49 to .61. Concurrent validity with the Stanford AchievementTest, 10th Edition Mathematics subtest is .68 (Clarke, Baker, et al., 2015). For the nontimedProFusion-R items, the standardized item alpha D .93; for the timed ProFusion items, theaverage item-total correlation was .51. The ProFusion-R was administered at pretest and post-test to assess procedural fluency and conceptual understanding in mathematics aligned withCCSS-M standards and domains taught during the NS1 intervention. At pretest, studentswere characterized as having low initial skills if they scored between 0 and 33 on the Profu-sion-R measure and having high initial skills if they scored equal to or higher than 34 on themeasure.
NumberShire Level 1 Mastery Tests (NMT)Mastery tests were developed by the research team to assess student learning every twoweeks during the NS1 intervention. Items included in each NMT were constructed to mea-sure only the standards taught and practiced in the two-week period preceding administra-tion. Approximately three items per standard were included in each NMT. NMTs wereadministered every two weeks of the study, up through week six.
Surveys
A survey was also administered during the study to gather data about participant experiencesand document practices employed in core and supplemental mathematics instruction deliv-ered to students in the treatment and control groups.
Demographics and PerceptionsAt the end of the study, teachers and interventionists completed a demographics survey todocument background information, previous teaching experience, and perceptions of NS1.The survey asks interventionists to self-report training and experience teaching mathematicsand using technology (e.g., level of education, mathematics education certifications, fre-quency of use of technology interventions in the classroom).
Instructional PracticesAt the end of the study, teachers were also asked to describe core and supplemental mathe-matics instruction provided to all NS1-eligible students. The survey asks teachers to describethe programs used, their content focus, the type of instructional strategies used duringinstruction, and the frequency and duration of instruction.
650 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
Statistical Analysis
Univariate effects of intervention condition on posttest outcome measures were examinedusing between-subjects analysis of covariance (ANCOVA), adjusting for pretest scores.Intervention effects on the two-, four-, and six-week interim mastery tests were evaluatedusing ANCOVAs, adjusting for pretest ProFusion-R total score as a covariate. Next, wetested for differential effects of condition by Special Education (SPED) status, ELL status, ini-tial skill level, and student engagement. For tests of differential effects of condition, weextended the primary ANCOVA models to include the main effects and cross product ofcondition and the proposed moderator. Pearson’s r correlation coefficients were used toexplore associations between number of sessions completed and change in outcomes frompretest to posttest among students assigned to the treatment condition. All analyses wereconducted with SPSS 21, and alpha was set to p < .05, two-tailed, for all tests.
Hedges’s g was reported as a metric of intervention effect size (What Works Clearing-house, 2008; effects of .25 and above are considered “substantively important”). Hedges’s gwas computed as the difference between the covariate adjusted means of the two groups atposttest divided by the posttest pooled standard deviation of the outcome. For our primaryoutcome analyses, we also reported partial h2, the proportion of variance explained by condi-tion, to facilitate interpretation and future meta-analyses.
Despite common misconceptions that multilevel modeling is required for education inter-vention research, the student-level analyses reported here are appropriate given that studentswere the unit of randomization and that the intervention was delivered at the student level. Arecent IES guide to nested randomized controlled trials (Lohr, Schochet, & Sanders, 2014)acknowledged the confusion regarding when one must account for clustering effects. Consis-tent with Bloom, Bos, and Lee (1999), Roberts and Roberts (2005), Rubin (1974), and others,Lohr et al. indicate that individual randomization “essentially cancels out the pre-existing clus-tering effect from the original schools, just as it cancels out pre-existing effects from unobservedconnections between the students such as belonging to the same church, softball team, or playgroup” (p. 34). Similarly, Raudenbush and Sadoff (2008), describe how individual randomiza-tion removes the effects of clusters on the average treatment effect. Because higher levels ofnesting have no effect on the average effect estimator or its standard error for this study design(e.g., Bloom et al., 1999; Raudenbush & Sadoff, 2008), and consequently no effect on the TypeI or Type II error rates (Murray, 1998), we have not included these levels in our analysis.
Results
Baseline Equivalence and Attrition
The expectation of baseline equivalence due to random assignment of groups was examined. Thetreatment and control groups were compared on demographic characteristics and outcomemeas-ures collected at pretest. Contingency table analyses and t tests were conducted on categorical andcontinuous measures, respectively. The groups did not significantly differ on any demographiccharacteristics (see Table 2 for demographic descriptive information). Compared to control stu-dents, treatment students performed significantly better on pretest GA-QD (M D 21.1, SD D7.5 vs.M D 19.0, SD D 7.6; t[235] D 2.18, p D .030, d D 0.28) and the ProFusion-R (M D 39.2,SDD 17.0 vs.MD 34.6, SDD 16.7; t[246]D 2.15, pD .032, dD 0.27). To control for nonequiva-lence at baseline, thesemeasures were included as additional covariates in all outcome analyses.
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 651
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
The extent to which attrition threatened the internal and external validity of this study wasevaluated using contingency table analyses and analysis of variance. Participants who com-pleted a posttest assessment were compared to those who did not with respect to demographiccharacteristics and pretest outcome measures. We also conducted analyses to test whether out-come variables were differentially affected across conditions by attrition. These latter analysesexamined the effects of condition, attrition status, and their interaction on pretest outcomes.Examination of attrition between pretest and posttest revealed that eight (6.4%) of the treat-ment participants did not complete a posttest assessment compared to four (3.2%) of the con-trol participants. Attrition rates did not significantly differ by condition or demographiccharacteristics. Compared to students who completed a pretest and posttest assessment, stu-dents who did not complete a posttest assessment performed significantly worse on the ProFu-sion-R assessment at pretest (M D 23.6, SD D 7.2 vs. M D 37.4, SD D 17.0; t[246] D 2.55, pD .011). We found no statistically significant interactions between attrition and condition pre-dicting baseline outcomes, suggesting that attrition was not systematic.
Intervention Effects
Table 3 provides means and standard deviations for each outcome by assessment time andcondition, along with results of the outcome analyses. Statistically significant effects of treat-ment over control were obtained on the ProFusion-R (p < .001, partial h2 D .063, Hedges’sg D 0.30) and the two-week NMT (p D .025, partial h2 D .022, Hedges’s g D 0.22). Interven-tion effects were not statistically significant for other study outcomes.
We extended the primary ANCOVA models to test for differential effects of condition onprimary outcomes (i.e., ProFusion-R, GA-MN, and GA-QD, and easyCBM-NCTM) bySPED status, ELL status, and initial skill level based on the pretest ProFusion-R assessmentscores. We found no statistically significant interactions with condition (ps > .224), indicat-ing no differential response to the intervention as a function of baseline studentcharacteristics.
Dose Response
Pearson’s r correlation coefficients were used to explore associations between program imple-mentation metrics and change in outcomes from pretest to posttest among students assignedto the treatment condition. Change in outcomes was not significantly correlated with the
Table 2. Demographic characteristics by condition.
Female n (%) 64 (51) 61 (49)SPED n (%) 11 (9) 12 (10)ELL status n (%) 31 (25) 28 (22)Age M (SD) 6.5 (0.5) 6.5 (0.5)
Notes. M D Mean, SD D Standard deviation. Age was computed as of the beginning of the study (10/1/2013.)
652 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
number of sessions completed (rs ranged from ¡.10 to .07, ps > .270), number of repeatedsessions (rs ranged from ¡.09 to .09, ps > .338), number of practice opportunities (rs rangedfrom ¡.04 to .18, ps > .070), or item response accuracy (rs ranged from .02 to .16, ps > .111).
Discussion
The present study was designed to test the promise of the NS1 intervention for improvingthe mathematical outcomes of students at risk for MD. Recognizing the shortage of rigorousstudies of education technology tools, we chose to employ a randomized controlled trial todetermine the statistical and practical effects of NS1, our first research aim. We hypothesizedthat at-risk students in the NS1 treatment condition would significantly outperform their at-risk counterparts in the control condition on proximal measures of whole-number conceptsand skills that were directly targeted in the intervention. This hypothesis was based on ourattention to incorporating evidence-based instructional design elements (Coyne et al., 2011)with an engaging gaming platform.
The results of the ANCOVA analyses suggest that NS1 treatment students outperformedthe control students on the primary proximal measure of whole-number concepts and skills,demonstrating a moderate and practically important effect (approximately a third of a stan-dard deviation difference). Treatment students also significantly outperformed their controlson the first of three interim mastery tests, demonstrating a .22 effect size difference.Although trends favored the treatment students on the second and third interim masterytests, these effects were not significant. This finding is interesting and worth critical
Table 3. Descriptive statistics and ANCOVA results for the outcome measures.
Pretest Posttest Condition effect
Outcome measure/condition M (SD) M (SD) Adj M F p Partial eta2 Hedges’s g
easyCBM-NCTM total raw 1.29 .257 .006 ¡0.13Treatment 21.9 (5.3) 23.9 (6.8) 23.2Control 21.6 (4.4) 23.8 (6.8) 24.1
2-week interim mastery test 5.07 .025 .022 0.22Treatment na 36.0 (11.0) 34.5Control na 31.0 (10.8) 32.1
4-week interim mastery test 2.57 .110 .011 0.17Treatment na 26.3 (12.2) 25.1Control na 22.4 (10.3) 23.2
6-week interim mastery test 0.24 .624 .001 0.06Treatment na 30.0 (9.3) 28.9Control na 27.6 (10.2) 28.3
Notes. M D Mean, SD D Standard Deviation, Adj D Adjusted. Baseline differences between conditions were observed on theen-CBM quantity discrimination and the Profusion-R assessment; therefore, all analyses included pretest scores on thesemeasures as covariates in addition to the pretest score on the target measure. Analyses involving interim mastery testassessments included the primary proximal Profusion-R pretest score as a covariate because the interim measure was notassessed at pretest. na D not assessed at pretest.
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 653
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
reflection. It may be that the NS1 intervention was less engaging over time, or it may be thatthe intervention was less effective in teaching treatment students to master skills and con-cepts in the last half of the program. It is also plausible that item difficulty was not constantacross the interim researcher-developed mastery tests and warrants further research in sub-sequent studies of the NS1 intervention.
As hypothesized, we found no significant differences between treatment and control stu-dents on the distal outcome measures. Although the trend favored NS1 treatment studentson two of the three distal measures, none of the effects were significant, nor were the magni-tude of effects large enough to be considered potentially promising effects, as deemed by theWWC (e.g., Hedges’s g > .25). Our hypothesis for null findings on the distal measures wasbased on three factors: (a) students received an abbreviated version of the full NS1 interven-tion, (b) our study was relatively small and underpowered to detect effects on measures withhistorically low to medium-sized effects in mathematics intervention research, and (c) thedistal measures assess concepts that do not directly align with the content targeted in theNS1 intervention (e.g., measurement, geometry).
Although we do not want to overinterpret the results from this promise study, we do findit interesting that our positive results on proximal outcomes stand in stark contrast to thelargely negative results from previous rigorous evaluation of education technology tools(Dynarski et al. 2007; Campuzano et al., 2009). However, we do believe that the findings ofthe current promise study do align with the slate of intervention studies that demonstratethe positive effect of explicit instruction approaches on the mathematics learning of studentswith early signs of math difficulties (Baker et al., 2002; Gersten, Beckmann, et al., 2009; Gers-ten, Chard, et al.. 2009; Kroesbergen & Van Luit, 2003). Further, the current findings supportour hypothesis that features of explicit instruction that have been demonstrated as effica-cious in print-based curricula for students with or at risk for MD could be carefully inte-grated with education technology to realize the potential for education technology tools(Baker et al., 2002; Clarke et al., 2008; Clarke et al., 2009; Nelson, Fien, Doabler, & Clarke, inpress).
Our second research aim was to test the potential moderating effects of prior mathematicsachievement and ELL status on treatment effects. It was hypothesized that ELL status wouldnot moderate treatment effect because we believed ELLs, like their non-ELL peers, wouldbenefit from the NS1 intervention due to our careful attention to precise mathematical lan-guage and vocabulary within gameplay. As predicted, results indicated that ELL status didnot moderate treatments effects for any of the outcomes, suggesting that ELL students equi-tably responded to the treatment relative to their non-ELL peers. Our second moderationhypothesis posited that at-risk students at the upper end of the at-risk pretest distributionwould benefit more from the NS1 than at-risk students at the lower end of the pretest distri-bution. We anticipated that the entire at-risk sample would benefit from NS1, but becausewe designed the intervention as a Tier 2 supplemental intervention, we conjectured that stu-dents at moderate risk might benefit more than students at significant risk for MD. However,this hypothesis was not supported by the analysis and we found no evidence for pretest scoremoderating treatment impact. Thus, NS1 appeared to be equally effective across the pretestdistribution.
Our third and final research aim was to examine dose response patterns or the degree towhich intervention gains were related to number of sessions completed, number of sessionsrepeated, or number of practice opportunities completed across NS1 gameplay. We
654 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
hypothesized a positive relation between practice opportunities and strength of interventiongains. Surprisingly, we found no statistical relationship between any of the dosage variablesand students’ change in any of the outcomes from pretest to posttest. We believe our opera-tional definition of dosage might be too crude, as currently defined, and that we may need toreconceptualize “dosage” and make it more precise to include the vast amount of gameplaymetrics.
In summary, we believe that the NS1 has demonstrated some promise for improving stu-dent mathematical outcomes related to whole-number concepts and skills. These promisingoutcomes were found in a rigorous research design and in comparison to a relatively strongbusiness-as-usual control condition—all participating schools had fairly sophisticated Tier 2mathematics systems of support for at-risk students. As with any randomized experiment, itis important to consider the nature of the counterfactual in this study. Critical for under-standing the magnitude of an observed treatment effect in randomized experimental designsis grasping the instructional events that occur in the comparison or control condition (Flayet al., 2005; Gersten, Baker, & Lloyd, 2000; Gersten et al., 2005; Shadish, Cook, & Campbell,2002). The measurement of the comparison condition can provide critical information forgenerating and justifying causal inferences. It is therefore particularly important forresearchers who use these types of designs to become knowledgeable about the comparisoncondition and examine the degree to which implementation of instruction differs betweentreatment and control conditions. Lemons, Fuchs, Gilbert and Fuchs (2014) noted that suchimplementation comparisons will likely lie somewhere on a continuum between no differen-ces and stark overlap.
Consider an RCT study that tests the efficacy of a print-based, Tier 2 mathematics pro-gram (Intervention X) with 200 at-risk kindergarten students. The researchers assign 100 ofthe students to the treatment condition (Intervention X) and the remaining 100 to a controlcondition, which provided students with no mathematics intervention services. Using thecontinuum model suggested by Lemons and colleagues (2014), a comparison of instructionalimplementation would reveal no overlap between the two conditions. In this case, if Inter-vention X was soundly designed and implemented with high fidelity, one would expect treat-ment students to significantly outperform their control peers on most mathematicsoutcomes. This is largely in part because the control students were essentially assigned to ano-treatment condition.
However, if the researchers were interested in conducting a more rigorous study of Inter-vention X, they might increase the robustness of the comparison condition. For instance,they might have students assigned to the comparison condition receive a competing evi-dence-based, Tier 2 mathematics program (Intervention Z) that targets identical mathemat-ics content and incorporates similar instructional design components to that included inIntervention X. Given the nature of Intervention Z, one could argue that there is at leastmodest overlap between it and the treatment intervention. Because of this overlap, theresearchers would likely characterize the study’s findings in a different light than studies thatuse a no-treatment condition. For example, if treatment effects were obtained, even minimalones, then those results could arguably be considered substantively important given thatIntervention Z has a previously established evidentiary basis for improving student mathe-matics outcomes. If treatment students performed equivalent to or no worse than their con-trol peers, then the researchers might likewise characterize the findings as educationallymeaningful.
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 655
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
While the control condition in our study implemented a host of intervention materialsand instructional practices, one could argue that it represents a more robust comparisonthan a no-treatment control condition. For example, six of the classrooms used EverydayMathematics, a program the WWC has deemed as having “potentially positive effects” onstudent mathematics achievement. The noted mathematics content overlap between NS1and the control condition encourages us to use a more molecular approach in our futurework to distinguish the similarities and differences between NS1 and the mathematics inter-ventions commonly used in today’s classrooms. We would expect these procedures to helpus better understand the meaning of our observed effects.
Limitations and Directions for Future Research
We have several limitations worth noting in the present study. First, the purpose of the study wasto examine the promise of the program—and not to document program efficacy. Although weemployed a rigorous experimental design, we included a relatively modest sample and were suffi-ciently powered to detect effects on proximal measures, but not effects on distal measures.Because this promise study was nested within an iterative design framework, we continued mak-ing revisions to the NS1 intervention after the conclusion of the study to ready it for subsequentformal efficacy testing. Therefore, the positive, significant outcomes should only be viewed aspreliminary support for the NS1 supplemental intervention. In addition, our moderation resultsmust likewise be viewed as preliminary findings and further testing is warranted to examine ifimportant subgroups respond similarly, or not, to the NS intervention.
A second factor that affects the generalizability of the findings is the setting in which thecurrent study took place. We tested the promise of NS1 in nine schools from two school dis-tricts in the Pacific Northwest and the sample of students was not entirely representative ofU.S. schoolchildren. Although the percentage of White students (61% in study, 60% U.S.)and Asian students (5% study sample, 4% U.S.) was commensurate with the larger popula-tion of U.S. students, we had a lower than average representation of Black students (5% studysample, 17% U.S.) and higher than average representation of Hispanic students (21% studysample, 17% U.S.; NCES, 2014). Therefore, it is unknown whether similar results would befound in a more representative sample of U.S. students or a sample that included higherthan average Black students, for example.
A final limitation of the current study is that we have only begun to sort through the largeamount of gameplay metrics to pose interesting research questions (e.g., patterns of playthat predict responsiveness) and to verify certain assumptions for gameplay sessions (e.g.,treatment integrity and adherence). For example, there may be interesting patterns of stu-dent engagement that may associate with responsiveness (or nonresponsiveness) that arenot readily identified by direct observation. Our initial foray into operationalizing and mea-suring dosage as practice opportunities within gameplay sessions was not predictive of pre–post gains, and therefore, a more sophisticated conceptualization of dosage may be necessaryto examine dose response or other interesting research questions related to responsiveness.In addition, conceptualizing and documenting such important constructs as treatment fidel-ity and treatment adherence could be transformed in the context of educational gaminginterventions. Research studies commonly focus on teachers alone when collecting informa-tion on the fidelity of intervention implementation. Rather than regarding students as
656 H. FIEN ET AL.
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
passive recipients of treatment, we encourage the field to begin to view students as activeparticipants of interventions, particularly technology-based interventions.
To address some of these limitations, we propose to further extend these researchactivities by conducting a series of rigorous studies to document the efficacy of NS1across more diverse samples and settings of students. Toward that end, we are submit-ting an Efficacy and Replication application to the Institute of Education SciencesNational Center on Special Education’s Education Technology topic area. We proposeto implement the fully featured 12-week NS1 intervention with a much larger numberof students from more diverse schools set in three regions of the United States (i.e.,Las Vegas, NV, Boston, MA, and Portland, OR). We will be adequately powered todetect both proximal and distal outcome measures. Further, we propose to examinewhether the significant outcomes demonstrated in the current study extend to diversesettings, and we propose to examine whether the treatment effect varies for importantsubgroups of students (e.g., ELL students, students with MD). In this way, we hope to(a) increase the use of rigorous methods endorsed and promulgated by the WWC and(b) provide invaluable information for practitioners and researchers seeking to imple-ment evidence-based supplementary technology-based interventions in mathematics forstudents at risk for mathematics difficulties.
Conflict of Interest
The author(s) declared the following potential conflicts of interest with respect to the research, author-ship, and/or publication of this article: Hank Fien, Christian T. Doabler, Nancy J. Nelson, and Scott K.Baker are eligible to receive a portion of royalties from the University of Oregon’s distribution andlicensing of certain Numbershire-based works. Potential conflicts of interest are managed through theUniversity of Oregon’s Research Compliance Services. An independent external evaluator and coau-thor of this publication completed the research analysis described in the article.
Funding
This research was supported in part by Project NumberShire 1, Grant No. EDIES11C0026, a subcon-tract with ThoughtCycle through the U.S. Department of Education, Institute of Education Sciences,Small Business Innovation Research Program. The opinions expressed are those of the authors and donot represent the views of the Institute or the U.S. Department of Education.
ARTICLE HISTORY
Received 22 May 2015Revised 3 November 2015Accepted 9 November 2015
EDITORS
This article was reviewed and accepted under the editorship of Carol McDonald Connor and SpyrosKonstantopoulos.
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 657
Dow
nloa
ded
by [
Uni
vers
ity o
f O
rego
n], [
Ben
Cla
rke]
at 1
6:02
10
Aug
ust 2
017
References
Alonzo, J., Tindal, G., Ulmer, K., & Glasgow, A. (2006). EasyCBM online progress monitoring assess-ment system. Eugene: University of Oregon.
Anderson, D., Lai, C.-F., Nese, J. F. T., Park, B. J., S�aez, L., Jamgochian, E.,… Tindal, G. (2010). Techni-cal adequacy of the easyCBM primary-level mathematics measures (grades K-2), 2009–2010 version(Technical Report No. 1006). Eugene, OR: Behavioral Research and Teaching, University ofOregon.
Archer, A. L., & Hughes, C. A. (2011). Exploring the foundations of explicit instruction. In A. L.Archer & C. A. Hughes (Eds.), Explicit instruction: Effective and efficient teaching (pp. 1–22). NewYork, NY: The Guilford Press.
Atkins, D. E., Bennett, J., Brown, J. S., Chopra, A., Dede, C., Fishman, B.,…Williams, B. (2010). Trans-forming American education: Learning powered by technology. Retrieved from http://www.ed.gov/sites/default/files/netp2010.pdf.
Baker, S. K., Fien, H., & Baker, D. L. (2010). Robust reading instruction in the early grades: Conceptualand practical issues in the integration and evaluation of Tier 1 and Tier 2 instructional supports.Focus on Exceptional Children, 42(9), 1–20.
Baker, S. K., Gersten, R. M., & Lee, D.-S. (2002). A synthesis of empirical research on teaching mathe-matics to low-achieving students. Elementary School Journal, 103, 51–73.
Berch, D. B. (1998, April). Mathematical cognition: From numerical thinking to mathematics educa-tion. Paper presented at the National Institute of Child Health and Human Development, Bethesda,MD.
Bloom, H. S., Bos, J. M., & Lee, S.-W. (1999). Using cluster random assignment to measure programimpacts. Evaluation Review, 23, 445–469. doi:10.1177/0193841x9902300405
Bodovski, K., & Farkas, G. (2007). Do instructional practices contribute to inequality in achievement?The case of mathematics instruction in kindergarten. Journal of Early Childhood Research, 5, 301–322.
Bryant, B. R., & Bryant, D. P. (2008). Introduction to the special series: Mathematics and learning dis-abilities. Learning Disability Quarterly, 31, 3–11. doi:10.2307/30035521
Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading and mathematicssoftware products: Findings from two student cohorts (Executive summary No. NCEE 2009-4042).Washington, DC: Institute of Education Sciences, National Center for Education Evaluation andRegional Assistance.
Cawley, J. F., Parmar, R., Foley, T., Salmon, S., & Roy, S. (2001). Arithmetic performance of students:Implications for standards and programming. Exceptional Children, 67, 311–328.
Chard, D. J., & Jungjohann, K. (2006). Scaffolding instruction for success in mathematics learning,intersection: Mathematics education sharing common grounds. Houston, TX: Exxon-MobilFoundation.
Clarke, B., Baker, S. K., Chard, D., Smolkowski, K., & Fien, H. (2008). Early learning in mathematics:Efficacy in kindergarten classrooms (Institute of Education Sciences Grant). Retrieved from https://ies.ed.gov/funding/grantsearch/details.asp?IDD599.
Clarke, B., Baker, S. K., & Fien, H. (2009). Foundations of mathematical understanding: Developing astrategic intervention on whole number concepts (Institute of Education Sciences Grant). Retrievedfrom https://ctl.uoregon.edu/research/projects/foundations-mathematical-understanding-developing-strategic-intervention-whole
Clarke, B., Baker, S. K., Smolkowski, K., Doabler, C. T., Strand Cary, M., & Fien, H. (2015). Investigat-ing the efficacy of a core kindergarten mathematics curriculum to improve student mathematicslearning outcomes. Journal of Research on Educational Effectiveness, 8, 303–324. doi:10.1080/19345747.2014.980021
Clarke, B., Doabler, C. T., Nelson, N. J., & Shanley, C. (2015). Effective instructional strategies for kin-dergarten and first-grade students at risk in mathematics. Intervention in School and Clinic, 50,257–265. doi:10.1177/1053451214560888
Clarke, B., Doabler, C. T., Strand Cary, M., Kosty, D. B., Baker, S. K., Fien, H., & Smolkowski, K.(2014). Preliminary evaluation of a Tier-2 mathematics intervention for first grade students: Utiliz-ing a theory of change to guide formative evaluation activities. School Psychology Review, 43, 160–177.
Clarke, B., & Shinn, M. R. (2004). A preliminary investigation into the identification and developmentof early mathematics curriculum-based measurement. School Psychology Review, 33, 234–248.
Clements, D. H. (2007). Curriculum research: Toward a framework for “research-based curricula.”Journal for Research in Mathematics Education, 38, 35–70.
Coyne, M. D., Kame’enui, E. J., & Carnine, D. (2011). Effective teaching strategies that accommodatediverse learners (4th ed.). Upper Saddle River, NJ: Pearson Education.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323, 66–69. doi:10.1126/science.1167311
Doabler, C. T., Baker, S. K., Kosty, D. B., Smolkowski, K., Clarke, B., Miller, S. J., & Fien, H. (2015).Examining the association between explicit mathematics instruction and student mathematicsachievement. The Elementary School Journal, 115, 303–333. doi:10.1086/679969
Doabler, C. T., & Fien, H. (2013). Explicit mathematics instruction: What teachers can do for teachingstudents with mathematics difficulties. Intervention in School and Clinic, 48, 276–285. doi:10.1177/1053451212473151
Doabler, C. T., Fien, H., Nelson-Walker, N. J., & Baker, S. K. (2012). Evaluating three elementarymathematics programs for presence of eight research-based instructional design principles. Learn-ing Disability Quarterly, 35, 200–211. doi:10.1177/0731948712438557
Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L.,… Sussex, W. (2007).Effectiveness of reading and mathematics software products: Findings from the first student cohort(Report to Congress No. NCEE 2007–4005). Washington, DC: U.S. Department of Education,Institute of Education Sciences.
Flay, B. R., Biglan, A., Boruch, R. F., Gonz�alez Castro, F., Gottfredson, D. C., Kellam, S.,… Jil, P.(2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Sci-ence, 6, 151–175. doi:10.1007/s11121-005-5553-y
Fuchs, L. S., Powell, S. R., Seethaler, P. M., Fuchs, D., Hamlett, C. L., Cirino, P. T., & Fletcher, J. M.(2010). A framework for remediating number combination deficits. Exceptional Children, 76, 135–156.
Gause, M., Fien, H., Baker, S. K., & Clarke, B. (2011). Project NumberShire I: A game-based integratedlearning and assessment system to target whole number concepts. (Subcontract with Thought Cycle,U.S. Department of Education, Institute of Education Sciences, Small Business Innovation ResearchProgram, 2011–2013, Funding Number: EDIES11C0026, $1,049,954).
Gersten, R. M., Baker, S. K., & Lloyd, J. W. (2000). Designing high-quality research in special educa-tion: Group experimental design. Journal of Special Education, 34, 2–18.
Gersten, R. M., Beckmann, S., Clarke, B., Foegen, A., Marsh, L., Star, J. R., & Witzel, B. (2009). Assist-ing students struggling with mathematics: Response to intervention (RtI) for elementary and middleschools (Report No. NCEE 2009–4060). Retrieved from http://www.eric.ed.gov/ERICWebPortal/detail?accnoDED504995
Gersten, R. M., & Chard, D. J. (1999). Number sense: Rethinking arithmetic instruction for studentswith mathematical disabilities. Journal of Special Education, 33, 18–28.
Gersten, R. M., Chard, D. J., Jayanthi, M., Baker, S. K., Morphy, P., & Flojo, J. (2009). Mathematicsinstruction for students with learning disabilities: A meta-analysis of instructional components.Review of Educational Research, 79, 1202–1242. doi:10.3102/0034654309334431
Gersten, R. M., Jordan, N. C., & Flojo, J. R. (2005). Early identification and interventions for studentswith mathematics difficulties. Journal of Learning Disabilities, 38, 293–304. doi:10.1177/00222194050380040301
Hudson, P., & Miller, S. P. (2006). Designing and implementing mathematics instruction for studentswith diverse learning needs. Boston, MA: Pearson Education.
Kilpatrick, J., Swafford, J., & Findell, B. (2001). Adding it up: Helping children learn mathematics.Washington, DC: Mathematics Learning Study Committee, National Research Council.
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 659
Klopfer, E., Osterweil, S., & Salen, K. (2009). Moving learning games forward. Cambridge, MA: TheEducation Arcade.
Kroesbergen, E. H., & Van Luit, J. E. H. (2003). Mathematics interventions for children with specialeducational needs: A meta-analysis. Remedial & Special Education, 24, 97–114. doi:10.1177/07419325030240020501
Lemons, C. J., Fuchs, D., Gilbert, J. K., & Fuchs, L. S. (2014). Evidence-based practices in a changingworld: Reconsidering the counterfactual in education research. Educational Researcher, 43, 242–252. doi:10.3102/0013189x14539189
Lohr, S., Schochet, P. Z., & Sanders, E. A. (2014). Partially nested randomized controlled trials in edu-cation research: A guide to design and analysis (NCER No. 2014–2000). Retrieved from http://ies.ed.gov/ncer/pubs/20142000/pdf/20142000.pdf
Morgan, P. L., Farkas, G., & Maczuga, S. (2015). Which instructional practices most help first-gradestudents with and without mathematics difficulties? Educational Evaluation and Policy Analysis,37, 184–205. doi:10.3102/0162373714536608
Morgan, P. L., Farkas, G., & Wu, Q. (2009). Five-year growth trajectories of kindergarten children withlearning difficulties in mathematics. Journal of Learning Disabilities, 42, 306–321. doi:10.1177/0022219408331037
Morgan, P. L., Farkas, G., & Wu, Q. (2011). Kindergarten children’s growth trajectories in reading andmathematics: Who falls increasingly behind? Journal of Learning Disabilities, 44, 472–488.doi:10.1177/0022219411414010
Murray, D. M. (1998). Design and analysis of group-randomized trials. New York, NY: Oxford Univer-sity Press.
National Association of State Directors of Special Education. (2006). Response to intervention: Policyconsiderations and implementation. Retrieved from http://www.nasdse.org/publications-t577/response-to-intervention-policy-considerations-an.aspx
National Center for Education Statistics. (2013). A first look: 2013 mathematics and reading (NCES2014–451). Washington, DC: National Center for Education Statistics, Institute of Education Sciences.
National Center for Education Statistics. (2014). The condition of education 2014 (NCES 2014–083).Retrieved from http://nces.ed.gov/pubs2014/2014083.pdf.
National Council of Teachers of Mathematics. (2006). Curriculum focal points for prekindergartenthrough grade 8 mathematics: A quest for coherence. Retrieved from http://www.nctm.org/standards/focalpoints.aspx?idD282
National Governors Association Center for Best Practices, Council of Chief State School Officers.(2010). Common Core State Standards for Mathematical Practice. Retrieved from http://www.corestandards.org/the-standards/mathematics
National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the NationalMathematics Advisory Panel. Washington, DC: U.S. Department of Education.
Nelson, N. J., & Doabler, C. T. (2013). TOT: Technology observation tool. Eugene: University ofOregon.
Nelson, N. J., Fien, H., Doabler, C. T., & Clarke, B. (in press). Considerations for realizing the promiseof educational technology. TEACHING Exceptional Children.
Nelson, N. J., Fien, H., Doabler, C. T., Clarke, B., Gause, M., & Baker, S. K. (2014, September). Assess-ing the feasibility and promise of an individualized mathematics gaming intervention for first gradestudents in a randomized-controlled trial. Presented at the Improving Early Math Outcomes forStudents with Disabilities through Intensive Intervention Symposium, Fall Conference of the Soci-ety for Research on Educational Effectiveness, Washington, DC.
Raudenbush, S. W., & Sadoff, S. (2008). Statistical inference when classroom quality is measured witherror. Journal of Research on Educational Effectiveness, 1, 138–154. doi:10.1080/19345740801982104
Roberts, C., & Roberts, S. A. (2005). Design and analysis of clinical trials with clustering effects due totreatment. Clinical Trials, 2, 152–162. doi:10.1191/1740774505cn076oa
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized stud-ies. Journal of Educational Psychology, 66, 688–701. doi:10.1037/h0037350
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designsfor generalized causal inference. Boston, MA: Houghton-Mifflin.
Van de Walle, J. A. (2001). Elementary and middle school mathematics: Teaching developmentally.New York, NY: Longman.
What Works Clearinghouse. (2008). What works clearinghouse: Procedures and standards handbook(Version 2.0). Washington, DC: National Center for Education Evaluation and Regional Assistance,Institute of Education Sciences, U.S. Department of Education.
Woodward, J., Beckmann, S., Driscoll, M., Franke, M., Herzig, P., Jitendra, A.,… Ogbuehi, P. (2012).Improving mathematical problem solving in grades 4 through 8 (NCEE 2012–4055). Retrieved fromhttp://ies.ed.gov/ncee/wwc/publications_reviews.aspx#pubsearch
Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B.,… Yukhymenko, M. (2012). Ourprincess is in another castle: A review of trends in serious gaming for education. Review of Educa-tional Research, 82, 61–89. doi:10.3102/0034654312436980
PROMISE OF THE NUMBERSHIRE LEVEL 1 GAMING INTERVENTION 661
