The Impact of IXL Math and IXL ELA on the California School Dashboard
CALIFORNIA RESEARCH REPORT
IXL LEARNING 777 Mariners Island Blvd., Suite 600, San Mateo, CA 94404
650-372-4040 | www.ixl.com
TABLE OF CONTENTS
_____________________________________________________________________________
Executive Summary 1
Objectives 2
Study Design 3
Methodology 5
Study Sample 5
Analysis 5
Results 5
TheIXLMathEffectontheCaliforniaDashboardforMath 5
TheIXLMathEffectontheCaliforniaDashboardforELA 7
References 8
Appendix 9
California Research Report
1
Executive Summary _____________________________________________________________________________
IXLisapersonalizedlearningplatformdesignedtohelpstudentsbuildacademicskills.ToquantifytheeffectivenessofIXLMathandIXLEnglishLanguageArts(ELA)onimprovingstudentlearning,researchersconductedanevaluationstudyacrossthestateofCaliforniausingaquasi-experimentalcontrolgroupdesign.ThisstudyaimedtounderstandtheimpactofIXLMathandIXLELAontheCaliforniaSchoolDashboardforMathandELA.TheCaliforniaDashboardshowsschoolperformanceonindicatorsincludedinthestate’sschoolaccountabilitysystem.Foreachindicator,aschoolisassignedacolorrating(fromhighesttolowest):Blue,Green,Yellow,Orange,orRed.StudiedsamplesusedIXLbetween2015and2019.Thekeyfindingsinclude:
• SchoolsthatusedIXLMathdemonstratedbettermathperformancethannon-IXLschools,basedontheirhigherprobabilityofachievingBlueorGreenratingsontheCaliforniaMathDashboard.
California Research Report
2
• SchoolsthatusedIXLELAdemonstratedbetterELAperformancethannon-IXLschools,basedontheirhigherprobabilityofachievingBlueorGreenratingsontheCaliforniaELADashboard.
Objectives _____________________________________________________________________________
ThisresearchaimedtoevaluatetheimpactofIXLMathandIXLELAontheCaliforniaSchoolDashboardforMathandEnglishLanguageArtsandLiteracy.Thisevaluationconsideredtworesearchquestions:
RQ1:WhatistheassociationbetweentheusageofIXLMathandschoolacademicperformanceasmeasuredbytheCaliforniaSchoolDashboardforMath?
RQ2:WhatistheassociationbetweentheusageofIXLELAandschoolacademicperformanceasmeasuredbytheCaliforniaSchoolDashboardforEnglishLanguageArtsandLiteracy?
California Research Report
3
Study Design _____________________________________________________________________________
ThisstudyevaluatedtheimpactofIXLMathandIXLELAonstudentacademicperformanceattheschoollevelinelementaryandmiddleschoolsinCalifornia.
TheCaliforniaDashboardshowsschoolperformanceonstateandlocalindicatorsincludedinCalifornia’sschoolaccountabilitysystem.Foreachindicator,aschoolisassignedacolor(i.e.,Blue,Green,Yellow,Orange,orRed)indicatingtheirperformancelevelsbasedonafive-by-fivecoloredtable(Figure1),acombinationoftheschool’scurrentperformancestatus(i.e.,VeryHigh,High,Medium,Low,orVeryLow),andchangestatus(i.e.,IncreasedSignificantlyfromPriorYear,IncreasedfromPriorYear,MaintainedfromPriorYear,DeclinedfromPriorYear,orDeclinedSignificantlyfromPriorYear).Forexample,ifaschool’s2019mathperformancelevelis“Medium”onthe2019SmarterBalancedAssessmentsandthisperformanceis“Maintainedfromtheprioryear,”thisschoolisassignedaYellowratingontheDashboardforMath(Figure1).AsshowninFigure1,Bluerepresentsthehighestperformancelevel,followedbyGreen,whileRedrepresentsthelowestperformancelevel.MoredetailsabouttheCaliforniaDashboardcanbefoundontheCaliforniaDepartmentofEducationwebsite(CaliforniaDepartmentofEducation,2019).
Figure 1. Dashboard determination (copied from California Department of Education, 2019)
California Research Report
4
Thisstudyusedaquasi-experimentalcontrolgroupdesign(Figures2-4)andcomparedtheperformanceofthetreatmentgroupandthecomparisongrouponthe2019,2018,and2017DashboardsforMathandELA,respectively.Forthe2019Dashboardanalysis,thetreatmentgroupincluded1-,2-,3-,or4-yearIXLschoolsthatusedIXLbetween2015and2019(Figure2).Forthe2018Dashboardanalysis,thetreatmentgroupincluded1-,2-,or3-yearIXLschoolsthatusedIXLbetween2015and2018(Figure3).Forthe2017Dashboardanalysis,thetreatmentgroupincluded1-or2-yearIXLschoolsthatusedIXLbetween2015and2017(Figure4).IXLusagebytheschoolsinthisstudyrangedfromlessthanoneminuteperstudent,perweektoover100minutesperstudent,perweek.SchoolsthatdidnotuseIXLduringthestudyperiod(non-IXLschools)wereusedasthecomparisongroup.Itisnotedthatnon-IXLschoolsmayormaynothaveusedotheronlineprogramsduringthestudyperiod.
Figure 2. Study design for the 2019 Dashboard analysis
Figure 3. Study design for the 2018 Dashboard analysis
California Research Report
5
Figure 4. Study design for the 2017 Dashboard analysis
Methodology_____________________________________________________________________________
STUDY SAMPLE
Thisstudyanalyzeddatafromallschoolsthatincludedstudentsinatleastonegradelevelfrom3-8.ResearchersexaminedDashboardperformanceformathandELAattheschoollevel.Intotal,therewere809,704,and695schoolsidentifiedasIXLMathschoolsforthe2019,2018,and2017MathDashboardanalysis,respectively;therewere426,315,and276schoolsidentifiedasIXLELAschoolsforthe2019,2018,and2017ELADashboardanalysis,respectively.Table1intheAppendixshowsthenumberofstudysamplesintotalandbyschoollocation(i.e,city,suburb,town,ruralareas,asdefinedbytheInstituteofEducationSciences).TheDashboardperformance,enrollment,anddemographicdatawereobtainedfromtheCaliforniaDepartmentofEducationwebsiteandtheInstituteofEducationSciences.
ANALYSIS
Thisstudycomputedchi-squarestatisticsandoddsratiostatisticstoexaminewhetherIXLschoolsweremorelikelytoreceiveperformancelevelsofBlueorGreencolorsontheirDashboardforMathandforELAthannon-IXLschools.Thetestsofstatisticalsignificancewereexamined.Ap-valuelessthan.05indicatesstatisticalsignificance—evidencethattheIXLeffectexists.
Results_____________________________________________________________________________
THE IXL MATH EFFECT ON THE CALIFORNIA DASHBOARD FOR MATH
IXLMathusagewasassociatedwithbettermathperformance.Specifically,schoolsthatusedIXLMathhadsignificantlyhigherprobabilitiesofreceivingperformancelevelsofBlueorGreenontheDashboardforMaththannon-IXLschools.
California Research Report
6
2017 Dashboard for MathFigure5showsthe2017DashboardforMathwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLMathusageandthe2017DashboardforMath(x2=32.35,p<.01).Specifically,43%ofIXLschoolsthatusedIXLMathfor1or2yearsduringthe2015-2017schoolyearsreceivedperformancelevelsofBlueorGreenonthe2017DashboardforMath,comparedtojust32%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.6timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2017DashboardforMaththannon-IXLschools(seeTable2intheAppendixfordetails).
2018 Dashboard for MathFigure6showsthe2018DashboardforMathwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLMathusageandthe2018DashboardforMath(x2=27.3,p<.01).Specifically,45%ofIXLschoolsthatusedIXLMathfor1,2,or3yearsduringthe2015-2018schoolyearsreceivedperformancelevelsofBlueorGreenonthe2018DashboardforMath,comparedtojust35%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.5timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2018DashboardforMaththannon-IXLschools(seeTable2intheAppendixfordetails).
Figure 5. 2017 Dashboard for Math with 5 colors (left) and with 2 color categories (right)
Figure 6. 2018 Dashboard for Math with 5 colors (left) and with 2 color categories (right)
California Research Report
7
THE IXL ELA EFFECT ON THE CALIFORNIA DASHBOARD FOR ELA
IXLELAusagewasassociatedwithbetterELAperformance.Specifically,schoolsthatusedIXLELAhadsignificantlyhigherprobabilitiesofreceivingperformancelevelsofBlueorGreenontheDashboardforELAthannon-IXLschools.
2017 Dashboard for ELA Figure8showsthe2017DashboardforELAwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLELAusageandthe2017DashboardforELA(x2=16.58,p<.01).Specifically,44%ofIXLschoolsthatusedIXLELAfor1or2yearsduringthe2015-2017schoolyearsreceivedperformancelevelsofBlueorGreenonthe2017DashboardforELA,comparedtojust32%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.7timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2017DashboardforELAthannon-IXLschools(seeTable3intheAppendixfordetails).
Figure 8. 2017 Dashboard for ELA with 5 colors (left) and with 2 color categories (right)
2019 Dashboard for Math Figure7showsthe2019DashboardforMathwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLMathusageandthe2019DashboardforMath(x2=10.33,p<.01).Specifically,44%ofIXLschoolsthatusedIXLMathfor1,2,3,or4yearsduringthe2015-2019schoolyearsreceivedperformancelevelsofBlueorGreenonthe2019DashboardforMath,comparedtojust38%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.3timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2019DashboardforMaththannon-IXLschools(seeTable2intheAppendixfordetails).
Figure 7. 2019 Dashboard for Math with 5 colors (left) and with 2 color categories (right)
California Research Report
8
2019 Dashboard for ELAFigure10showsthe2019DashboardforELAwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLELAusageandthe2019DashboardforELA(x2=12.30,p<.01).Specifically,47%ofIXLschoolsthatusedIXLELAfor1,2,3,or4yearsduringthe2015-2019schoolyearsreceivedperformancelevelsofBlueorGreenonthe2019DashboardforELA,comparedtojust39%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.4timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2019DashboardforELAthannon-IXLschools(seeTable3intheAppendixfordetails).
Figure 10. 2019 Dashboard for ELA with 5 colors (left) and with 2 color categories (right)
2018 Dashboard for ELAFigure9showsthe2018DashboardforELAwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLELAusageandthe2018DashboardforELA(x2=15.06,p<.01).Specifically,50%ofIXLschoolsthatusedIXLELAfor1,2,or3yearsduringthe2015-2018schoolyearsreceivedperformancelevelsofBlueorGreenonthe2018DashboardforELA,comparedtojust39%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.6timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2018DashboardforELAthannon-IXLschools(seeTable3intheAppendixfordetails).
Figure 9. 2018 Dashboard for ELA with 5 colors (left) and with 2 color categories (right)
California Research Report
9
Appendix_____________________________________________________________________________
Table 1. Characteristics for the state of California and IXL schools
Math ELA
State IXL schools State IXL schools
California Dashboard 2019
• Thenumberofschools 6,511 809 6,420 426
• %ofschoolsincities 42% 34% 42% 34.5%
• %ofschoolsinsuburbs 42% 46% 42% 45.5%
• %ofschoolsintowns 6% 9% 6% 9%
• %ofschoolsinruralareas 9% 11% 9% 11%
California Dashboard 2018
• Thenumberofschools 6,610 704 6,492 315
• %ofschoolsincities 42% 37% 42% 36%
• %ofschoolsinsuburbs 42% 42% 42% 43%
• %ofschoolsintowns 6% 9% 6% 8%
• %ofschoolsinruralareas 9% 12% 9% 13%
California Dashboard 2017
• Thenumberofschools 6,744 695 6,578 276
• %ofschoolsincities 42% 37% 42% 38%
• %ofschoolsinsuburbs 42% 43.5% 42% 45%
• %ofschoolsintowns 6% 8.5% 6% 5%
• %ofschoolsinruralareas 9% 11% 9% 12%
References_____________________________________________________________________________ CaliforniaDepartmentofEducation(2019).2019CaliforniaSchoolDashboardTechnicalGuide. Fromhttps://www.cde.ca.gov/ta/ac/cm/documents/dashboardguide19.pdf
California Research Report
10
Table 2. Analysis on Dashboard for Math
Table 3. Analysis on Dashboard for ELA
Year Schools
% of schools that received Green or Blue on Dashboard
% of schools that received Red, Orange, or Yellow on Dashboard
Chi-square Odds Ratio
2019IXLSchools 44% 56%
10.33* 1.3*Non-IXLSchools 38% 62%
2018IXLSchools 45% 55%
27.33* 1.5*Non-IXLSchools 35% 65%
2017IXLSchools 43% 57%
32.35* 1.6*Non-IXLSchools 32% 68%
Year Schools
% of schools that received
Green or Blue on Dashboard
% of schools that received Red, Orange, or Yellow on Dashboard
Chi-square Odds Ratio
2019IXLSchools 47% 53%
12.30* 1.4*Non-IXLSchools 39% 61%
2018IXLSchools 50% 50%
15.06* 1.6*Non-IXLSchools 39% 61%
2017IXLSchools 44% 56%
16.58* 1.7*Non-IXLSchools 32% 68%
Note. *p<.05. For 2019 analysis, IXL schools refer to schools that used IXL Math for 1, 2, 3 or 4 years. For 2018 analysis, IXL schools refer to schools that used IXL Math for 1, 2, or 3 years. For 2017 analysis, IXL schools refer to schools that used IXL Math for 1 or 2 years.
Note. *p<.05. For 2019 analysis, IXL schools refer to schools that used IXL ELA for 1, 2, 3 or 4 years. For 2018 analysis, IXL schools refer to schools that used IXL ELA for 1, 2, or 3 years. For 2017 analysis, IXL schools refer to schools that used IXL ELA for 1 or 2 years.