CARNEGIEQUANTWAYSUCCESS 1
CARN
EGIEM
ATHPAT
HWAY
S
TECH
NICAL
REP
ORT
AssessingtheEffec%venessofQuantwayAMul&levelModelwithPropensityScoreMatching
October2016
HiroyukiYamada,AngelBohannon,&AliciaGrunow
CarnegieFounda-onfortheAdvancementofTeaching
Stanford,CA
ThisprogramofworkissupportedbyCarnegieCorpora7onofNewYork,TheBill&MelindaGatesFounda1on,TheWilliamandFloraHewle:Founda1on,TheKresgeFounda/on,andLuminaFounda/onincoopera/onwiththeCarnegieFounda-onfortheAdvancementofTeaching.
CARNEGIEQUANTWAYSUCCESS 2
ABSTRACTQuantwayisaCarnegieMathPathwaysini2a2vewhichredesignsthecontent,pedagogy,andstructureoftradi-onaldevelopmentalmathcoursestosimultaneouslytackletradi-onalbarriersofstudentsuccessandsupportabroaderrangeofdevelopmentalstudentsinachievingtheirmathpoten,al.Specifically,Quantwayisaquan,ta,vereasoningsequencethatiscomprisedofasingletermdevelopmentalmathcoursecalledQuantway1andacollege-levelmathcoursecalledQuantway2.Thisstudyassessedtheeffec4venessofthedevelopmentalmathcourse,Quantway1,duringitsfirst6semestersofimplementa:on.Weusedahierarchicallinearmodelingtechniquetoconductpropensityscorematchingacross37studentcharacteris(csinordertocomparethecourseperformanceofQuantway1studentswithmatchedcomparisonstudentsintradi0onaldevelopmentalmathcourses.Quantway1studentsdemonstratedsignificantlyhigheroddsofsuccessthanmatchedcomparisonstudentsinfulfillingdevelopmentalmathcourserequirements.Addi8onally,Quantway1effectswereposi%veacrossallsexandrace/ethnicitysubgroupsaswellasinnearlyallclassroomsandcolleges.ThisstudyprovidedrobustevidencethatQuantway1increasesstudentsuccessinfulfillingdevelopmentalmathrequirementsandadvancesequityinstudentoutcomes.Direc&onsforfutureworkaresuggested.
CARNEGIEQUANTWAYSUCCESS 3
Traditionaldevelopmentalorremedialmathsequencesserveasahugeimpedimentforcommunitycollegestudents,oftenpreventingthemfromobtainingtechnicalcredentialsandassociatedegrees,aswellasblockingtheirtransfertofour-yearinstitutions.Nearly60%ofcommunitycollegestudentsnationwidearerequiredtotakeatleastonedevelopmentalmathematicscourse,and80%ofthesestudentsdonotcompleteacollegemathcoursewithinthreeyears(Bailey,Jeong,&Cho,2010).Studentsspendlongperiodsoftimerepeatingcoursesandaccruingstudentloandebt,ultimatelyleavingcollegewithoutadegree.ThiscrisisincompletionhasnegativeramificationsforcommunitycollegestudentearningsandforAmericanworkforcedevelopmentoverall.Studentswhodonotcompletehigherlevelsofeducationhavesignificantlylowerincomes.In2012,forallpeopleaged25orolder,highschoolgraduatesearnedanannualincomeof$33,904,thosewithassociate’sdegreesearned$40,820,andthosewithbachelor’sdegreesearned$55,432(Johnstone,2013).Lowerlevelsofeducationalattainmentalsohaveadetrimentaleffectontheoveralleconomyastechnologicaladvancesandglobalcompetitionhavecreatedapressforanincreasinglyskilledworkforce.TheGeorgetownCenteronEducationandtheWorkforceestimatesthat65%ofallAmericanjobswillrequireapostsecondaryeducationbeyondhighschoolby2020,buttheUnitedStateswillfallshortofmeetingtheserequirementsby5millionworkersatthecurrentrateofproduction(Carnevale,Smith,&Strohl2013).Communitycollegesplayacriticalroleinworkforcedevelopment,servinghalfofthe6.5millionundergraduatesintheUnitedStates.Additionally,traditionallyunderservedstudentsaredisproportionatelylikelytoencounterdevelopmentalmathasastumblingblockontheroadtocollegecompletion.Thecommunitycollegestudentpopulationismoreraciallydiverse,older,andlowerincomethan4-yearuniversitystudents(Bueschel,2004).Minoritystudentsareplacedinmoredevelopmentalmathcoursesandlesslikelytocompletethesecoursestoachievecollege-levelmathcreditthanwhitestudents(Baileyetal.,2010;Chen,2016).Improvingthesuccessratesofstudentsindevelopmentalmathsequencesisakeyleverforadvancinganequityagenda.Severalaspectsoftraditionaldevelopmentalmathsequenceshavebeenproposedascontributorstonegativestudentoutcomes.Traditionally,studentsmusttakelongmulti-coursesequencesofincreasinglevelsofdifficultytofulfilldevelopmentalmathrequirements.Asequenceofmultipledevelopmentalcourses,suchasbasicarithmetic,pre-algebra,elementaryalgebra,andthenintermediatealgebra,leadsintoacollege-leveltransferableclasssuchaspre-calculus.Thisstructuredrasticallyhindersstudentcompletion.Evenwhenstudentscompleteonecourseinasequence,manyfailtoenrollinsubsequentcourses(Cullinane&Treisman,2010;Baileyetal.,2010).Additionally,theinstructioninmanymathclassroomsdoesnotincorporateresearch-basedcurriculumdesignandpedagogicpracticesthatfosterdeeperstudentlearningandengagement(Mesa,2011).Traditionalmathcoursesemphasizetransmissionofcontentoveramoreparticipatoryapproach(Edwards,Sandoval,&McNamara,2015),factualandproceduralknowledgeoverconceptualknowledge(Mesa,2011),anddonotdemonstratetherelevancyofmathematicalconcepts(Carnevale&Desrochers,2003).Furthermore,traditional
CARNEGIEQUANTWAYSUCCESS 4
developmentalmathcoursesdonotaddresseitherlanguageandliteracyornon-cognitivebarriers(e.g.,mathanxietyandstereotypethreat)thatimpedemanystudents’abilitytolearnmath(Blackwell,Trzesniewski,&Dweck,2007;Haynes,Perry,Stupnisky,&Daniels,2009;Gomez,Rodela,Lozano,&Mancevice,2013).Morerecently,therelevancyofthealgebra-heavycontentoftraditionalmathcurriculumhasalsobeencalledtoquestion.AstudyontheSurveyofWorkplaceSkills,Technology,andManagementPracticesfoundthatonly19%ofemployeesuseanyalgebraintheirwork(Handel,2007).Tospurprogressonthisproblem,CarnegieFoundationfortheAdvancementofTeachingconvenedanetworkedimprovementcommunity(NIC)-anationalcommunityofcommunitycollegeadministratorsandfaculty,andeducationalresearchers(Bryk,Gomez,Grunow,&LeMahieu,2015).Throughanimprovementscienceapproach,theNICredesignedthecontent,pedagogy,andstructureoftraditionalmathsequencestoincreasethenumberofstudentscompletingtheirmathrequirements.Theresultofthisworkwastwoacceleratedalternativestotraditionaldevelopmentalmathsequencesfornon-STEMstudents:StatwayandQuantway.Statwayisanacceleratedyear-longintroductorycollege-levelstatisticscoursethatintegratesdevelopmentalmathcontent.ApreviousstudydemonstratedStatway’sefficacyandimpactonstudentsuccessovertraditionaldevelopmentalmathprograms(Yamada&Bryk,2016).Quantwayprovidesanotheralternativetothetraditionalmathsequencefocusingonquantitativereasoning.AsshowninFigure1,Quantway1isaone-termquantitativereasoningcourseforstudentswhoplacetwolevelsbelowcollege-level,thusenablingthemtocompletetheirdevelopmentalmathrequirementsinasingleterm.StudentswhosuccessfullycompleteQuantway1arepreparedforcollegelevelmathandeligibletoenrollinQuantway21oranothercollege-levelquantitativereasoningcourse.SincetheQuantway1’slaunchin2012,Quantway1’simplementationhasgrownmorerapidlythanStatway,withQuantway1enrollmentgrowingfrom418to1936studentsoverthefirstfouryearsofimplementation(Huang,Hoang,Yesilyurt,&Thorn,2016).BecauseQuantway1fulfillsdevelopmentalmathrequirementsinoneterm,communitycollegescaneasilyintegrateitintotheircurrentdevelopmentalmathofferings.Quantway1hasbeensuccessfulinpromotingstudentsuccessindevelopmentalmathcourses,essentiallydoublingthesuccessratesoftraditionaldevelopmentalmathcoursesinhalfthetime.Notably,Quantway1maintainedthesehighsuccessratesevenasstudentenrollmentmorethanquadrupled.Inthemostrecent2014-2015academicyear,57%ofthestudentsenrolledinQuantway1successfullycompletedthecourseinoneterm.Incomparison,only21%ofabaselinegroupofdevelopmentalmathstudentscompletedthetraditionaldevelopmentalmathcourseinoneyear2(Huangetal.,2016).1Fiveins(tu(onshaveimplementedQuantway2,servingatotalof429studentsover3yearsofimplementa/on(Huangetal.,2016).Forthepurposesofthisstudy,weevaluatedtheefficacyofQuantway1.
2Tocomputethisbaselinesuccessrate,weworkedwithins(tu(onalresearchersfromsixofthefirstQuantwaycolleges.Analysesrevealedthatonly20.6percentofstudentswereabletosuccessfullycompletetheir
CARNEGIEQUANTWAYSUCCESS 5
Despitethesepromisingresults,itisunclearwhetherQuantway1’sdemonstratedimpactisa"ributabletotheprogramitself,orisduetodifferencesinthestudentsthatselectorareplacedintoQuantway1.Buildinguponourdescrip3veresults,thisstudypursuesamorerigorouscausalanalysisofQuantway1throughapropensityscoretechnique,separa7ngtheprogram’seffectfrompoten0alselec0onbias.Theseresultswouldprovideastrongereviden!arybaseonwhichtobasedecisionsaboutadop!ngandscalingQuantway1.
DESIGNOFQUANTWAYQuantway1isdesignedaroundthepremisethatallstudentsarecapableoflearningambitiousmathematicsandsucceedingindevelopmentalmathcourseswiththerightsupports.AssummarizedinFigure2,Quantway1sharesaworkingtheoryofimprovementwithitssisterprogramStatway,aimingtoincreasestudentsuccessthroughworkingonsixkeydrivers:(1)accelerationofdevelopmentalmathrequirements,(2)implementationofaresearch-basedinstructionalsystem,(3)socioemotionalsupports,(4)languageandliteracysupports,(5)facultydevelopment,and(6)participationinaNIC(formoreinformationonthetheoryofimprovement,seeYamada&Bryk,2016).ThewayQuantway1addressesthesedriversdiffersfromthatofStatwayinordertofilladifferentnicheincommunitycollegemathdepartmentsandmeettheneedsofspecificstudents.Below,wewillelaborateonthefeaturesthatdistinguishQuantway1fromotheraccelerateddevelopmentalmathprograms.First,Quantway1acceleratesstudents’abilitytocompletedevelopmentalmath,gettingthemtocollegelevelmathmorequickly.Quantway1focusesonquantitativeliteracy,whichisdescribedas“theabilitytoadequatelyuseelementarymathematicaltoolstointerpretandmanipulatequantitativedataandideasthatariseinanindividual’sprivate,civic,andworklife”(Gillman,2004,p.5).Thesequantitativeliteracyconceptsarecodifiedinasetofrigorouslearningoutcomesthatwerecollaborativelyestablishedandvettedbyacommitteethatincludedrepresentativesfromseveralmathematicalprofessionalsocieties.3BecauseQuantway1’slearningoutcomesprovidestudentswithastrongfoundationinnumericalandquantitativereasoningconcepts,ithasservedaspreparatoryquantitativereasoningcourseformanynon-STEMpathwaysandmajorsandastheculminatingmathcoursefortechnicalcertificateprograms.SomecollegesofferQuantwayasapathwaythroughcollegelevelmathbycombiningQuantway1withQuantway2oranothercollegelevelquantitativereasoningcourse.Quantway’sflexibledesigncanbeeasilyintegratedintocurrentinstitutionalstructuresandmeetsavarietyofcommunitycollegeneeds.
developmentalmathsequencewithinafullyear.Addi8onally,28.5percentachievedthisgoala&ertwoyears,31.6percenta)erthreeyears,and33.3percenta)erfouryears.Formoreinforma7on,seeHuangetal.,2016.3Thesemathema)calsocie)esincludetheNa)onalNumeracyNetwork,AmericanMathema)csAssocia)onofTwo-YearColleges,andtheMathema,calAssocia,onofAmerica.
CARNEGIEQUANTWAYSUCCESS 6
Second,Quantway1’sinstructionalsystemisdesignedtogroundunfamiliarmathconceptsinfamiliarsituationsthroughcontextualization.Quantway1’slessonsuseauthentic,relevantcontextsandrealdatatoincreasestudentmotivationtolearn.Quantway1isorganizedaroundthreeintentionalthemes(citizenship,healthcare,andfinancialliteracy)thatreflecteverydayconceptsandarecriticallyimportantinengaginginsociety.Byillustratingtherealworldapplicationsofmathconcepts,Quantway1cancontributepreviouslyunsuccessfulstudentscanhavemeaningfulandpositiveinteractionswithquantitativereasoningcontent.LikeStatway,theQuantway1instructionalsystemisdesignedtofosterrobustandsustainedmathematicallearning,emphasizingtheteachingofconceptstoimprovebothproceduralandconceptualunderstanding(Hiebert&Grouws,2007).TheQuantway1instructionalmodelisorganizedaroundthreeresearch-basedlearningopportunities–productivestruggle,explicitconnections,anddeliberatepractice.Inproductivestruggle,facultyengagestudentsinsubstantivemathematicaltasksthatencouragestudentstostrugglewithkeymathematicalconceptsandsolveproblemsthatarechallengingbutstillwithinreach(Hiebert&Grouws,2007).Byproductivelystruggling,studentscanmakemeaningofthemathematicalcontentforthemselvesanddevelopstrategiesforengagingwiththecontent.Explicitconnectionsrefertoinstructionthatcreatesopportunitiesforstudentstomakeconnectionsbetweenmathematicalproceduresandunderlyingconceptualknowledge.Deliberatepracticeaimstoimprovestudentperformancethroughaseriesofhighlystructured,increasinglysophisticated,andchallengingtasksthatdeepenfacilitywithkeyconcepts(Edwards&Beattie,2015).Theselearningopportunitiesaresupportedbyinstructionalpracticesthatfacilitatestudentdiscussionandsupportcollaborativelearningaroundrichmathematicalproblems(Edwards&Beattie,2015;Edwardsetal.,2015).Third,Quantway1integratestwotypesofresearch-basedstudentsupportsdesignedtomeettheneedsofdiversestudentlearners–productivepersistence,andlanguageandliteracysupports.Onesetofsupportsisdesignedtopromotestudents’abilitytoproductivelypersistthroughrigorousmathcoursework.Thesocioemotionalintervention,whichwecallProductivePersistence,consistsofacollectionofstudentactivitiesandfacultyactionsthataddressthehigh-leveragenon-cognitivefactorsthatpromotestudenttenacityandeffectivelearningstrategies(Edwards&Beattie,2016).NICmembersworkedtogetherwithsocialpsychologiststoiterativelydevelopthispackageofproductivepersistenceroutines,interventions,andpracticesthatworktopromotegrowthmindset,reducemathanxiety,andincreasestudents’senseofbelonging.Asecondsetofinterventionsisdesignedtosupportstudentsinsuccessfullygrapplingwiththecomplexlanguageandliteracydemandsofmathematics,withitsdifferentformsofrepresentationandelaborategrammaticalforms.Quantway1lessonsembedlanguageandliteracytoolstosupportthecomprehensionandorganizationofinformationinquantitativesituations.Theselessonsarewrittentoavoidliteracybarriersthatdevelopmentalmathstudentscommonlyface(Gomez,Rodela,Lozano,&Mancevice,2013;Gomezetal.,2015).Fourth,becausetheQuantway1curriculumandpedagogysignificantlydifferfromtraditionalmethodsofteaching,Quantway1facultyareinvitedtoparticipateinacomprehensive
CARNEGIEQUANTWAYSUCCESS 7
professionaldevelopmentprogram(Edwardsetal.,2015).ThisFacultySupportProgrampreparesfacultytoteachQuantway1andsupportsthemintheirfirstyearofteaching,andprovidesongoingopportunitiesforinstructionalimprovementandprofessionallearning.Throughonlineresources,facultymentorship,andongoingworkshops,thisprogrampreparesfacultytoeffectivelyimplementPathways’collaborativeinstructionalapproach,learningopportunities,andproductivepersistenceandlanguageandliteracysupports.Fifth,Quantway1facultyandadministratorsparticipateinanetworkimprovementcommunity(NIC)thatprovidesthemwithacollaborativelearningcommunitytosupporttheminteachingandimplementingQuantway1.TheNICsocialstructuresupportscommunitycollegesfacultyandadministratorsincollectivelygeneratinganddisseminatingpracticallearningaboutwhatworks,forwhom,andunderwhatconditionstoreliablydeliverefficacyatscale(Bryketal.,2015).BothQuantway1studentsandfacultyreportthatthecombina(onofthesedesignelementscreatesameaningfullydifferentmathexperiencefromtradi5onaldevelopmentalmathcourses.Inaddi'on,thesuccessratesofQuantway1studentsinthefirstfouryearsofimplementa0onaresignificantlyhigherthanins/tu/onalbaselinesateachofthepar/cipa/ngcolleges(Huangetal.,2016).However,evalua3ngtheeffec3venessoftheQuantway1programrequirescomparingQuantway1studentsuccesstoareasonablecounterfactualthatrepresentshowsimilarstudentswouldhaveperformediftheyhadnottakenQuantway1.Inthisstudy,weusedapropensityscorematchingtechnique(Rosenbaum&Rubin,1983)tocompareQuantway1studentswithsimilarstudentsintradi+onaldevelopmentalmathprogramsinthesameins$tu$on.Weconductedthepropensityscoreanalysiswithinahierarchicallinearmodelingframework(Raudenbush&Bryk,2002)toaccountforthenestedstructureofthedatawithstudentswithinins(tu(onsinthenetwork.Weconductedafollow-upsensi(vityanalysistoexaminewhethertheeffectscouldbeexplainedbyotherunmeasureddifferencesbetweenthetwogroupsofstudents.
Wealsolookedatvariationinperformanceacrossstudents,classroomsandinstitutions.Incontrasttotypicalevaluationsthatreportonlytheaverageimpactofanintervention,thisstudyassessedwhetherQuantway1iseffectiveacrosstherangeofclassroomsandinstitutionsintheNIC.Theseinvestigationsintovariationsupporttheprogram’sabilitytoscalewithefficacy(Bryketal.,2015)andcaninformwhereimprovementeffortsshouldbetargetedinordertofurtherincreasesuccessrates.WealsoexaminedpossibledifferentialeffectsofQuantway1acrosssexandrace/ethnicitysubgroupstodeterminethepotentialoftheprogramtopromoteanequityagendabyimprovingoutcomesacrossallrace/ethnicityandsexsubgroups.
CARNEGIEQUANTWAYSUCCESS 8
METHODS
PARTICIPANTSQuantway1wasfirstimplementedduringthespringof2012.4Theinitialcohortofstudentsspanned8communitycollegesacrossthreestates(Georgia,NewYork,andOhio).Throughoutthe2014-15academicyear,Quantway1served5561studentsfromatotalof14colleges(seeAppendix)acrosseightstates(Georgia,Minnesota,NewJersey,NewYork,Ohio,Washington,WestVirginia,andWisconsin;Huangetal.,2016;Sowers&Yamada,2015).
DATAANDSTUDYDESIGNInstitutionalresearchersfromparticipatingcollegesprovidedbackgrounddataonstudentcharacteristics,courseenrollmentandperformance.Theanalyticsampleofthecurrentstudyconsistedof4,243Quantway1studentsfrom10collegeswhoenrolledinaQuantway1coursebetweenthespringof2012andthefallof2014,and83,887potentialcomparisongroupstudentsfromthecorrespondingsemesters.
First,weidentifiedagroupofcomparisontraditionaldevelopmentalmathstudentswithsimilarcharacteristicstoQuantway1students.Toobtainpropensityscores,wetookahierarchicallinearmodeling(HLM)approach(Hong&Raudenbush,2005,2006;Raudenbush&Bryk,2002;Yamada&Bryk,2016)andconstructedatwo-levelHLMmodelwithatotalof37student-levelcovariatesincludingstudentbackgroundcharacteristicsandpriorcoursetakingandsuccesspatternsduringthetwoyearspriortotheQuantway1term.Weselectedcovariatesbasedonpriorresearchfindingsandadvicefrominstitutionalresearchersintheparticipatingcolleges.Thelistincludesstandardstudentbackgrounddatasuchassexandrace/ethnicity.Ithasbeenshownthatthesecharacteristicstendtodifferentiatestudents’progressindevelopmentalmathsequences(Baileyetal.,2010).Wealsomatchedonstudents’priorcourse-takinghistoryandperformanceinthepasttwoyears.Previousresearchdemonstratedthatstudents’priorcoursetakinghistoryandsuccesspatternsareamorereliableindicatorofstudents’educationalandcareergoalsthantheirdeclaredprogramofstudy(Jenkins&Cho,2012).
Table1presentsallofthecovariatesusedinthepropensityscorematchingandtheirdescriptivestatisticsbeforeandafterpropensityscorematchingwasconducted.Wefoundasubstantialnumberofunknownrecordsforstudents’dateofbirthwhencomputingstudents’ageinyears.Tofactorthesecasesintothepropensitymodel,weconstructedadummyvariableandcodedmissingageas1,otherwise0.Also,weaccountedforsixcohortgroupsbyformulatingasetofdummyvariableswithSpring2014asareferencecategory.ThedescriptivedataontheleftpanelofTable1showsthatoverallQuantway1studentshavehigherproportionsoffemaleandHispanicstudentsthanthenon-Quantway1students.Quantway1studentshadmorecourserecordsinthetwoyearsbeforetakingaQuantway1course,
4 One college was on a quarter system until the fall of 2012 and implemented Quantway 1 for the first time in the winter of 2012.
CARNEGIEQUANTWAYSUCCESS 9
suggestingthatthetermtheytookaQuantway1coursewaslesslikelytobetheirfirstsemesteroryear.Quantway1studentsalsostartedtheirdevelopmentalcourse(s)earlier,andattemptedmoredevelopmentalmathcoursesandcollege-levelcoursesthannon-Quantway1students.
Weconductedpropensityscorematchingseparatelyforeachcohortandcollegebyapplyinganearestneighbormatchingalgorithm(Rosenbaum&Rubin,1985).ThisalgorithmwasappropriateforourstudybecausewewantedtoretainasmanyQuantway1studentsaspossibleandhadalargepoolofnon-Quantway1studentsforcreatingmatches.WeattemptedtofinduptofivematchesperQuantway1student(5:1ratiomatching)tomaximizethebestmatchesfromthenon-Quantway1studentgroupwhilestillmaintainingprecision(Ming&Rosenbaum,2000).Wealsospecifiedacaliperdistanceofupto0.2toreducetheriskofbadnearestneighbormatchesbasedonrecommendationsintheliterature(Austin,2011;Rosenbaum&Rubin,1985).Forpropensityscorematching,weusedthepackageMatchIt(Ho,Imai,King,&Stuart,2011)inR(RCoreTeam,2015).WethenestimatedQuantway1’seffectivenessbycomparingsuccessratesofQuantway1studentswiththeirmatchedcomparisonsusingafour-levelHLMmodelwithabinaryoutcome.SuccesswasdefinedasapassinggradeoragradeofCorhigher5onaQuantway1courseforQuantway1studentsandadevelopmentalmathcourseonelevelbelowcollegelevel(oranothercoursedeemedequivalenttoaQuantway1coursebyfaculty)forthematchedcomparisonstudents.Forthelattergroup,wetrackedcourseoutcomesovertheentireacademicyear(i.e.,trackingcourseoutcomesoverthefallandspringsemestersforthefallcohortsandthespring,summer,andfallsemestersforthespringcohorts).Asdescribedearlier,Quantway1acceleratestraditionaldevelopmentalmathsequencesforstudentsplacedtwolevelsbelowcollegemathematicsinonesemester.Similarstudentsfollowingthetraditionaldevelopmentalmathroutetypicallycompletethedevelopmentalsequenceinoneandahalfyears(Baileyetal.,2010;Cullinane&Treisman,2010).Accordingly,ifcomparisonstudentsinthefallcohortshadfailedadevelopmentalmathcourseonelevelbelowcollegelevelinthefallsemesterbutpasseditthefollowingspringsemester,wecounteditassuccess.Therefore,analysiswasconservative,providingcomparisonstudentstwiceasmuchtimetoreachthesamesuccessbenchmarkasQuantway1students.InthisHLMframework,matchedclusters(level1)werenestedwithinQuantway1students(level2),whowereinturnnestedwithinQuantway1facultymemberclassrooms(level3)withintheircolleges(level4).BecausematchedcomparisonswereconstructedforeachQuantway1student,theirrespectivecomparisonstudentswerealsoassignedthecorrespondingQuantway1facultyID.Thisstrategyallowedustoformeachfacultymember’sclassroomasamini-experimentinwhichthemeanoutcomeoftheirQuantway1studentswascomparedwiththatofsimilarstudentswhopursuedtraditionalmathcourses,inorderto
5 A grade of C- or higher was used for a college that employed a +/– grading scheme.
CARNEGIEQUANTWAYSUCCESS 10
estimatethevariabilityineffectamongfacultywithincolleges.WeusedHLM7(Raudenbush,Bryk,Cheong,Congdon,&duToit,2011)foralloftheHLManalyses.
RESULTSPROPENSITYSCOREMATCHINGToobtainpropensityscores,weconstructedatwo-levelBernoullimodelandestimateditsmodelparametersusingmaximumlikelihoodviapenalizedquasi-likelihoodestimation.ϕilistheprobabilityofstudentienrollinginQuantway1incollegel.Accordingly,ηilisthelog-oddsofthisincidentandformallyexpressedas:
Level-1Model(Student)Prob(QWil=1|βl)=ϕillog[ϕil/(1-ϕil)]=ηilηil=β0l+β1l(COV1il)+…+β37l(COV37il),Level-2Model(College)β0l=γ00+u0l,β1l=γ10,...Β37l=γ370,
whereQWisadummyvariableindicatingwhetheragivenstudentwasenrolledinQuantway1(codedas1)ornot(codedas0),COV1…COV37arethesetofpropensityscorecovariates.6Wematchedatotalof12,448comparisonstudentsto3,992Quantway1students.Table1comparesthedescriptivestatisticsoneachcovariatebeforeandaftermatchingtotheQuantway1group.Table2documentsthebalanceinpropensityscorecohortbycohortforeachcollege.TherewerenosignificantdifferencesinmeanpropensityscoresbetweentheQuantway1andmatchedcomparisonstudentsinanyofthecohortsforeachcollege(seetvalues).Theseresultsprovidestrongevidencethatcomparabilityofthegroupswasachievedonthemeasuredcovariates.Itmaybeworthwhileheretomentionthematchedratiosweaccomplished.AsdescribedearlierintheMethodsection,weattemptedtofinduptofivematchesperQuantway1student.Thematchedratiosinthefarrightcolumnsuggestthatingeneral,weidentified4to5
6 We initially included two covariates of college non-STEM courses (the number of courses attempted and the respective success rate). However, they involved collinearity with other covariates, and accordingly, the model did not converge. Thus, we excluded them from the propensity model.
CARNEGIEQUANTWAYSUCCESS 11
matchesperQuantway1student.ForsomecohortsfromColleges3and8,however,weidentifiedfewermatchesandneededtoexcludesomeQuantway1studentstomaintainthecomparabilityofthegroups.Itappearsthatbothcollegeshavearelativelylargepopulationofstudentswhowereplacedintodevelopmentalmathcoursesandaccordingly,morestudentsatvaryinglevelsofdevelopmentalmath.Therefore,itmaybepossiblethatcertainkindsofstudents(e.g.,thosewhofaileddevelopmentalcoursesmultipletimes)wereadvisedtotakeQuantway1soastolimitthenumberofappropriatestudentsformatching.ESTIMATINGQUANTWAYEFFECTSToestimatedifferencesinsuccessrates,weconstructedafour-levelBernoullimodelandestimateditsmodelparametersusingmaximumlikelihoodviapenalizedquasi-likelihoodestimation.ϕijklrepresentstheprobabilitythatstudentimatchedwithQuantway1studentjassociatedwithfacultymemberk’sclassincollegelsuccessfullycompletedthedevelopmentalmathsequence.Correspondingly,ηijklisthecorrespondinglog-oddsofthisoutcomeandformallyexpressedas:
Level1Model(Student)Prob(SUCCijkl=1|πjkl)=ϕijkl,log[ϕijkl/(1-ϕijkl)]=ηijkl,ηijkl=π0jkl+π1jkl*(QWijkl),Level2Model(QWStudent)π0jkl=β00kl+β01kl(TERMjkl)+β02kl(W12jkl)+β03kl(S12jkl)+β04kl(F12jkl)+β05kl(S13jkl)+β06kl(F13jkl)+β07kl(F14jkl)+r0jkl,π1jkl=β10kl+β11kl(TERMjkl)+r1jkl,Level3Model(Faculty)β00kl=γ000l+u00kl,β01kl=γ010l,β02kl=γ020l,β03kl=γ030l,β04kl=γ040l,β05kl=γ050l,β06kl=γ060l,β07kl=γ070l,β10kl=γ100l+u10kl,β11kl=γ110l,
CARNEGIEQUANTWAYSUCCESS 12
Level4Model(College)γ000l=δ0000+v000lγ010l=δ0100,γ020l=δ0200,γ030l=δ0300,γ040l=δ0400,γ050l=δ0500,γ060l=δ0600,γ070l=δ0700,γ100l=δ1000+v100l,γ110l=δ1100,
whereSUCCrepresentsdevelopmentalmathachievement(1forsuccessfullycompletedand0fornotsuccessfullycompleted),andQWisadummyvariableindicatingwhetherthestudentwasenrolledinQuantway1(codedas1)oroneofthematchedcomparisons(codedas0).AllthecovariatesatLevel2wereincludedasadditionaladjustmentvariablesfortheoutcome.Termisadummyvariableindicatingwhethertheoutcomeformatchedcomparisonstudentswasbasedononesemester(codedas1)ortheentireacademicyear(codedas0).W12toF14aredummyvariablesforthesixcohortgroupsdescribedearlierintheMethodsection.7TheresultspresentedinTable3indicatethatonaverage,Quantway1studentsdemonstratedsignificantlyhigheroddsofsuccess,2.10(95%CI[1.35,3.27]8),insuccessfullycompletingthedevelopmentalmathcoursethanthematchedcomparisonstudents.Thecorrespondingestimatedprobabilitiesofsuccesswere56.78%fortheQuantway1groupand38.51%forthematchedcomparisongroup.VARIATIONINPERFORMANCETheestimatedcoefficientsbetweentheinterceptandtheslopeatbothcollegeandfacultylevelswerenegative(-.70and-.41,respectively),suggestingthatthelowertheoutcomeforthematchedcomparisongroup,thelargertheeffectofQuantway1.Thisinverserelationshipwasstrongeratthecollegelevel.Inaddition,wefoundvariationinQuantway1effectamongcollegesandfacultymembers(0.36and0.23forthecollegeandfacultyvariances).Figures3and4displaythevariationinQuantway1effectsizeatthecollegeandfacultylevels,respectively.Inbothcharts,weaddedthreelinesasreferences.ThecenterlinerepresentstheaverageeffectofQuantway1,andtheupperandlowerlinesrepresenttheupperandlowerboundsoftheaverageeffect(whicharedeviatedintwoSEsfromthecenterline).Figure3demonstratesthattherewerepositiveQuantway1effectsonstudentoutcomesinallbutCollege10(whichshowednoeffectof7 We also constructed the same four-level model with individual students’ propensity scores included in the level 1 model and those six cohort group variables added to the level 2 model for the slope. The results from this model revealed no significant coefficients of these additional covariates and closely mirrored those from the simpler model. For the ease of interpretation, we focus here on the results from the simpler model. 8HLM7generates95%confidenceintervalsofoddsra0os.
CARNEGIEQUANTWAYSUCCESS 13
Quantway1).College8standsoutasapositivedeviantwithaQuantway1effectoutsidetheupperboundoftheaverageeffect.Figure4showsthevariationinQuantway1effectivenessacrosstheclassroomsintheNIC.ThevastmajorityofQuantway1facultyatCollege8drasticallyoutperformedtheaverageQuantway1faculty,suggestinginternalcoherenceatthisinstitution.Incontrast,awiderangeofvariationwasobservedamongfacultymembersatCollege3.Understandingthemechanismsthatenableconsistentlyhighperformanceandinvestigatingthecausesofvariationareareasoffuturestudy.
SUBGROUPANALYSISToexaminepossibledifferentialeffectsofQuantway1bysexandrace/ethnicitysubgroups,weconstructedafour-levelHLMsimilartothosedescribedabove.Inthissubgroupanalysis,however,weappliedeffectcodingtothegroupingvariablesinordertodirectlyrepresentbothmainandinteractioneffectsontheoutcome.ThereferencecategorieswerefemaleandWhite.Eachofthesewascodedas-1.Weexcludedcaseswiththeunknownsexstatus.Figure5presentsthemodel-basedresultstransformedbackintotheirnaturalmetricsofproportionofstudentssuccessfullycompletingadevelopmentalmathsequence.Thismetrictransformationwasmadefortheeaseofinterpretation.PositiveeffectsofQuantway1wereobservedforeachrace/ethnicitygroup.BlackandHispanicmalestudentsthattakeQuantway1exhibitedthelargestincreaseincompletionratesrelativetobaselinelevelsofperformance.SENSITIVITYANALYSISIngeneral,Quantway1effectswerestrongandprevalentforallsubgroups.Thevalidityoftheseeffectswasbasedonanassumptionofastronglyignorabletreatmentassignment.Inotherwords,allrelevantcovariateswereincludedinthepropensityscoreanalysis,sothatthebiasduetounmeasuredcovariatescouldbeignored.Thus,weexaminedthesensitivityoftheestimatedQuantway1effectstopossibleconfoundingbyunmeasuredvariables(Hong&Raudenbush,2005,2006;Lin,Psaty,&Kronmal,1998).Givensomeunmeasuredcovariates(U),theQuantway1effect(δ)canbere-estimatedbyadjustingforhypothesizedhiddenbias(γ(E[U1]-E[U0]))asδ*=δ-γ(E[U1]-E[U0]),whereγistheunmeasuredcovariates’associationwiththeoutcomeandE[U1]-E[U0]istheirassociationwithtreatmentassignment(i.e.,Quantwayornon-Quantwayenrollment).AdaptingtheapproachofHongandRaudenbush(2005,2006),weoperationallydefinedaproxyforγasacoefficientderivedfromafour-levelmodeldesignedtopredicttheoutcomewiththesamesetofcovariatesusedinthepropensityscoreanalysisandE[U1]-E[U0]astheobservedmeandifferencebetweentheQuantwayandnon-Quantwaygroupsonthecorrespondingcovariate.Wethenselectedthelargestpositivevalueoftheproductofthesetwovaluesasthelargestpossiblebias9andobtainedanadjustedQuantwayestimate(δ*).9 We used the sum of the product values for those requiring a set of dummy variables (e.g., cohort group,
race/ethnicity).
CARNEGIEQUANTWAYSUCCESS 14
Accordingly,were-estimatedaQuantway1effectontheoutcomeandconstructeda95%confidenceintervalfortheadjustedestimate.Theadjustedestimatewas.69inlogits(95%CIs[.28,1.11]),andthecorrespondingconfidenceintervaldidnotcontain0oranynegativevalue,therebysupportingthestrongignorabilityassumption.OursensitivityanalysisconcludesthatitisveryunlikelythatourgeneralconclusionregardingQuantway1’spositiveeffectswasinfluencedbytheomissionofunmeasuredconfoundingfactors.DISCUSSIONThisstudyassessedQuantway1’seffectivenessforcommunitycollegestudentsacrosssixsemestersofimplementationusingarigorouscausalanalysis.Apropensityscorematchingtechnique(Rosenbaum&Rubin,1983)withinanHLMframework(Raudenbush&Bryk,2002)allowedustocontrolforpossibleselectionbiasbymatchingQuantway1studentswithcomparablestudentsenrolledintraditionaldevelopmentalmathsequencesacross37studentcharacteristics.Wealsoundertookasensitivityanalysistoexaminethepossibilitythattheestimatedeffectswereinfluencedbyunmeasuredconfoundingvariables.Throughouttheseanalyses,Quantway1studentsdemonstratedsignificantlyhigheroddsofsuccessthanmatchedcomparisonstudents.Ouranalysesalsoindicatedthattheseresultswerenotduetounmeasureddifferencesbetweenthetwogroups.WeconcludethatQuantway1substantiallyimprovesstudentsuccessratesinfulfillingdevelopmentalmathcourserequirements.WhiletypicalevaluationsmaystopatestimatingtheeffectsizeofQuantway1,thisstudyalsosoughttounderstandthevariationoftheQuantway1programacrossdifferentcolleges,faculty,andstudentsubgroups.Inordertoachieveefficacyatscale,Quantway1mustnotonlyproduceapositiveeffectonaverage,butmustalsobeeffectivefordiversestudentpopulationsacrossarangeofdifferentclassroomandinstitutionalcontexts.OuranalysisfoundthatQuantway1effectswerepositiveacrossallsexandracial/ethnicsubgroupsofstudents.Aspreviouslystated,studentsfromtraditionallyunderservedgroups,includingBlacks,Hispanics,andlow-incomestudents,aremorelikelytoparticipateindevelopmentalmathcourses(Chen2016).Becauseineffectivetraditionaldevelopmentalmathcourseshaveadisproportionateimpactonthesetraditionallyunderservedgroups,theseresultssuggestthatQuantway1canplayanimportantroleinincreasingtheoverallnumberoftraditionallyunderservedstudentscompletingtheirmathrequirements.Inaddition,Quantway1hasapositiveeffectinnearlyallclassroomsandcollegesinthenetwork,indicatingthattheprogramcanworkforvariedfacultyindifferentinstitutionalcontexts.Quantway1’sdesignincludeskeyleversthatmayexplainwhyitbettersupportstraditionallyunderservedcommunitycollegestudents.First,Quantway1isaquantitativeliteracycoursethatacceleratesstudentprogresstocollege-levelmathbyofferingdevelopmentalmathrequirementsinasingleterm.Second,Quantway1’sresearch-basedinstructionalsystemcontextualizesmathconceptsandisorganizedaroundthreelearningopportunitiestopromoterichmathematicallearningforabroaderrangeofstudents.AnecdotalaccountsfromstudentsindicatethatQuantway1’suniquemathematicalexperienceshelpthemseethemselvesas
CARNEGIEQUANTWAYSUCCESS 15
mathematicallearnersanddoers.Third,Quantway1aimstosupportsocioemotionalskillsandprovidelanguageandliteracysupportstohelpstudentsgrapplewiththecomplexlanguageofmathematics.Finally,Quantway1’sFacultySupportProgramandtheNICstructuresupportfacultyinimplementingQuantway1’suniquepedagogicalpracticesacrossdifferentfacultyandinstitutionalcontexts.OurresultssuggestthatQuantway1’scomprehensiveandsystematicapproachtotacklingthetypicalbarriersthatdevelopmentalmathstudentsfaceiskeytoitssuccess.Furtherempiricalevidenceisneededtoconnectparticulardesignelementstothepositiveeffectsoftheprogram.Fornow,wecanconcludethattheQuantway1packageisaneffectivealternativetothetraditionaldevelopmentalmathsequenceandacceleratestheabilityofadiverserangeofstudentstocompletetheirdevelopmentalmathrequirementsinavarietyofcontexts.TheseresultsandQuantway1’sflexiblesingle-termstructuredemonstrateitssignificantpotentialtopositivelyimpactnumerousstudentsinavarietyofcommunitycollegecontexts.Itisworthnotingacoupleofthekeylimitationsofthecurrentstudythatwouldbefruitfultopicsforfutureinvestigations.Oneopportunityforfutureresearchistotrackcollege-levelmathachievementbetweenthetwomatchedgroupsoneyearafterQuantway1enrollmenttodetermineifQuantway1studentsperformcomparablyorbetterthanmatchedstudentgroupsinfuturecollege-levelmathcourses.SinceQuantway1isdesignedtonotonlygetstudentsthroughtheirdevelopmentalmathsequencesbuttopreparethemtomeettheircollegemathrequirements,thiswillbeaparticularlyimportantanalysisindeterminingQuantway1’seffectiveness.Aslongitudinaldatabecomeavailable,wealsoplantotracklonger-termoutcomes,suchastransferratesandacademicsuccessofQuantway1studentsin4yearinstitutions.ThesignificantvariationinoutcomesacrossfacultyandcollegesdemonstratedinFigures3and4presentsanotheropportunityforfurtherinvestigation.Thegoalofqualityimprovementistoreducethevariationbetweenclassroomsandcollegesachievingpositiveresultsacrossdiversecontexts.Investigatingpositiveandnegativedeviantsprovidesinsightintothekeysourcesofvariation.College8,forexample,significantlyoutperformstheothercollegesinthenetworkandmaintainshighperformanceacrossalltheclassroomsinthecollege.College10,incontrast,performssignificantlyworsethanothercolleges.Futureresearchshouldexplorewhetherthesecollegesdifferinhowtheyenactthekeydesignelementsdescribedabove,andstudythevariousadaptationsthatthesecollegesmadeinresponsetotheirlocalcontext.DiscoveringandsharingkeypracticesacrossNICcollegeswouldenhancethenetwork’sabilitytoreplicateQuantway1’spositiveoutcomesasitspreadstomorediversecontexts.Inconclusion,byredesigningthecontent,pedagogy,andstructureoftraditionaldevelopmentalmathcourses,Quantway1providesarichmathematicalexperienceforabroaderrangeofdevelopmentalstudents,includinghistoricallyunderservedgroups.TheseeffortshavecontributedtoQuantway1’spositiveimpactonequitableoutcomesbyimprovingcompletionratesforallsexandracial/ethnicsubgroupsandacrossdiversecontexts.Despitegreatadvancesinincreasingdevelopmentalmathcompletionratesonaverage,variationinoutcomes
CARNEGIEQUANTWAYSUCCESS 16
acrossfacultyandcollegesintheNICindicatesthereisstillmuchworktobedoneinadvancingQuantway1’sefficacyreliablyatscale.ByleveragingtheNICstructure,wewillcontinuetoacceleratelearningthroughqualityimprovementtosolvethisdevelopmentalmathcrisis.
CARNEGIEQUANTWAYSUCCESS 17
REFERENCESAustin,P.C.(2011).Optimalcaliperwidthsforpropensity-scorematchingwhenestimating
differencesinmeansanddifferencesinproportionsinobservationalstudies.PharmaceuticalStatistics,10,150-161.
Bailey,T.,Jeong,D.W.,&Cho,S.-W.(2010).Referral,enrollment,andcompletionin
developmentaleducationsequencesincommunitycolleges.EconomicsofEducationReview,29,255–270.
Blackwell,L.S.,Trzesniewski,K.H.,&Dweck,C.S.(2007).Implicittheoriesofintelligence
predictachievementacrossanadolescenttransition:Alongitudinalstudyandanintervention.ChildDevelopment,78,246–263.
Bryk,A.S.,Gomez,L.M.,Grunow,A.,&LeMahieu,P.G.(2015).Learningtoimprove:How
America'sschoolscangetbetteratgettingbetter.Cambridge,MA:HarvardEducationPress.
Bueschel,A.C.(2004).Themissinglink:Theroleofcommunitycollegesinthetransi?on
betweenhighschoolandcollege.InM.W.KirstandA.Venezia(Eds.),Fromhighschooltocollege:Improvingopportuni1esforsuccessinpostsecondaryeduca1on(pp.252–284).SanFrancisco:Jossey-Bass.
Carnevale,A.P.,&Desrochers,D.M.(2003).Standardsforwhat?:TheeconomicrootsofK-16reform.Princeton,NJ:EducationalTestingService.
Carnevale,A.P.,Smith,N.,&Strohl,J.(2013).Recovery:Projectionsofjobsandeducation
requirementsthrough2020.GeorgetownUniversityCenteronEducationandtheWorkforce.
Chen,X.(2016).RemedialcoursetakingatU.S.public2-and4-yearinstitutions:Scope,
experiences,andoutcomes(NCES2016-405).U.S.DepartmentofEducation,Washinton,DC:NationalCenterforEducationStatistics.
Cullinane,J.,&Treisman,P.U.(2010).Improvingdevelopmentalmathema1cseduca1onin
communitycolleges:AprospectusandearlyprogressreportontheStatwayini7a7ve.AnNCPRWorkingPaper.Na#onalCenterforPostsecondaryResearch.
Edwards,A.&Beattie,R.(2015).Promotingstudentlearningandproductivepersistenceindevelopmentalmathematics.NADEDigest,9,30-39.
CARNEGIEQUANTWAYSUCCESS 18
Edwards,A.R.,Sandoval,C.,&McNamara,H.(2015).Designingforimprovementinprofessionaldevelopmentforcommunitycollegedevelopmentalmathematicsfaculty.JournalofTeacherEducation,66,466-481.doi:10.1177/0022487115602313.
Gillman,R.(2004).SIGMAAisFormed.Focus,24(1),5.Gomez,K.,Rodela,K.,Lozano,M.,&Mancevice,N.(2013).Designingembeddedlanguageand
literacysupportsfordevelopmentalmathematicsteachingandlearning.MathAMATYCEducator,5,43-56.
Gomez,K.,Gomez,L.M.,Rodela,K.C.,Horton,E.S.,Cunningham,J.,&Ambrocio,R.(2015).
Embeddinglanguagesupportindevelopmentalmathematicslessons:Exploringthevalueofdesignasprofessionaldevelopmentforcommunitycollegemathematicsinstructors.JournalofTeacherEducation,66,450-465.doi:10.1177/0022487115602127.
Handel,M.J.(2007,May31–June1).Anewsurveyofworkplaceskills,technology,
andmanagementpractices(STAMP):Backgroundanddescriptivestatistics.PaperpresentedattheWorkshoponResearchEvidenceRelatedtoFutureSkillDemandsoftheNationalResearchCounciloftheNationalAcademies,Washington,DC.
Haynes,T.L.,Perry,R.P.,Stupnisky,R.H.,&Daniels,L.M.(2009).Areviewof
attributionalretrainingtreatments:Fosteringengagementandpersistenceinvulnerablecollegestudents.InJ.C.Smart(Ed.),Highereducation:Handbookoftheoryandresearch(pp.227–272).NewYork,NY:Springer.
Hiebert,J.,&Grouws,D.(2007).Theeffectsofclassroommathematicsteachingonstudents’
learning.InF.K.LesterJr.(Ed.),Secondhandbookofresearchonmathematicsteachingandlearning(pp.371–404).Charlotte,NC:InformationAge.
Ho,D.E.,Imai,K.,King,G.,&Stuart,E.A.(2011).MatchIt:Nonparametricpreprocessingfor
parametriccausalinference.JournalofStatisticalSoftware,42,1-28.Hong,G.,&Raudenbush,S.W.(2005).Effectsofkindergartenretentionpolicyonchildren’s
cognitivegrowthinreadingandmathematics.EducationalEvaluationandPolicyAnalysis,27,205-224.
Hong,G.,&Raudenbush,S.W.(2006).Evaluatingkindergartenretentionpolicy:Acasestudyof
causalinferenceformultilevelobservationaldata.JournaloftheAmericanStatisticalAssociation,101,901-910.
CARNEGIEQUANTWAYSUCCESS 19
Huang,M.,Hoang,H.,Yesilyurt,S.,&Thorn,C.(2016).CommunityCollegePathways:2014-2015ImpactReport.Stanford,CA:CarnegieFounda4onfortheAdvancementofTeaching.
Jenkins,D.,&Cho,S.W.(2012).Getwiththeprogram:Acceleratingcommunitycollegestudents’entryintoandcompletionofprogramsofstudy(CCRCWorkingPaperNo.32).NewYork,NY:ColumbiaUniversity,TeachersCollege,CommunityCollegeResearchCenter.
Johnstone,R.(2013).FiscalconsiderationsofStatway®andQuantway®:Weshouldbedoingthis
anyway,buthere’showitmayhelpthebottomline.NationalCenterforInquiry&Improvement(NCII).Retrievedfromhttp://www.inquiry2improvement.com/attachments/article/12/NCII-Carnegie_SW-QW_Fiscal_Considerations_110713-Rob-NCII.pdf.
Lin,D.Y.,Psaty,B.M.,&Kronmal,R.A.(1998).Assessingthesensitivityofregressionresultsto
unmeasuredconfoundersinobservationalstudies.Biometrics,54,948-963.Ming,K.,&Rosenbaum,P.(2000).Substantialgainsinbiasreductionfrommatchingwitha
variablenumberofcontrols.Biometrics,56,118-124.Mesa,V.(2011).Similaritiesanddifferencesinclassroominteractionbetweenremedialand
collegemathematicscoursesinacommunitycollege.JournalonExcellenceinCollegeTeaching,22(4),21-55.
RCoreTeam.(2015).R:Alanguageandenvironmentforstatisticalcomputing[Computer
software].Vienna,Austria:RFoundationforStatisticalComputing.Raudenbush,S.W.,&Bryk,A.S.(2002).Hierarchicallinearmodels:Applica2onsanddata
analysismethods(2nded.).ThousandOaks,CA:SagePublica<ons.Raudenbush,S.W.,Bryk,A.S.,Cheong,Y.F.,Congdon,R.T.,&duToit,M.(2011).HLM7:
Hierarchicallinearandnonlinearmodeling.Lincolnwood,IL:Scien/ficSo1wareInterna'onal.
Rosenbaum,P.R.,&Rubin,D.B.(1983).Thecentralroleofthepropensityscoreinobservationalstudiesforcausaleffects.Biometrika,70,41-55.
Rosenbaum,P.R.,&Rubin,D.B.(1985).Constructingacontrolgroupusingmultivariate
matchedsamplingmethodsthatincorporatethepropensityscore.TheAmericanStatistician,39,33-38.
Sowers,N.,&Yamada,H.(2015).Pathwaysimpactreport.Stanford,CA:CarnegieFoundation
fortheAdvancementofTeaching.
CARNEGIEQUANTWAYSUCCESS 20
Yamada,H.,&Bryk,A.S.(2016).Assessingthefirsttwoyears’effectivenessofStatway®:Amultilevelmodelwithpropensityscorematching.CommunityCollegeReview,44,179-204.
CARNEGIEQUANTWAYSUCCESS 21
APPENDIXListofPar+cipa+ngColleges
• AtlanticCapeCommunityCollege• BoroughofManhattanCommunityCollege• CuyahogaCommunityCollege• EastGeorgiaStateCollege• MadisonCollege• MarshallUniversity• OnondagaCommunityCollege• RidgewaterCollege• RocklandCommunityCollege• SinclairCommunityCollege• SouthGeorgiaStateCollege• UniversityofNorthGeorgia,Gainesville• UniversityofWashington,Bothell• WestchesterCommunityCollege
CARNEGIEQUANTWAYSUCCESS 22
Table1.
Descrip(veSta(s(csofCovariatesintheTwo-LevelPropensityModel
Samplebeforematching Samplea(ermatching
Non-Quantway1 Quantway1
Non-Quantway1 Quantway1
% % % %
Sex
Female* 56 62
60 61
Male 44 37
39 38
Unknown 0 1
1 1
Race/Ethnicity
Asian 3 4
3 4
Black 31 30
32 30
Hispanic 18 26
21 26
White* 36 34
36 33
Mul$racial 1 1
1 1
Other 1 0
0 0
Unknown 9 5
6 5
Anycourserecordsinpasttwoyears
No* 45 38
46 41
Yes 55 62
54 59
Cohortgroup
Winter2012 6 2
3 2
CARNEGIEQUANTWAYSUCCESS 23
Spring2012 16 11
16 11
Fall2012 14 13
20 13
Spring2013 16 20
14 20
Fall2013 16 21
20 21
Spring2014* 17 15
14 15
Fall2014 15 19
14 19
Agemissing 21 14
23 15
M SD M SD
M SD M SD
Age(inyears) 24.98 8.47 24.09 8.18 24.09 7.55 23.97 8.11
Semesterssincefirstdevelopmentalmathcourse 0.92 1.70 1.62 2.46
1.27 2.12 1.37 2.25
Courseload 3.69 1.20 3.95 1.14
3.95 1.19 3.93 1.14
Developmentalmath
Onelevelbelowcollegelevel
Numberofcoursesa-empted 0.09 0.31 0.21 0.55
0.15 0.41 0.17 0.47
Successrate 0.00 0.04 0.00 0.04
0.00 0.04 0.00 0.03
Twolevelsbelowcollegelevel
Numberofcoursesa-empted 0.21 0.48 0.35 0.65
0.26 0.56 0.30 0.58
Successrate 0.11 0.31 0.19 0.38
0.13 0.33 0.17 0.36
Threeormorelevelsbelowcollegelevel
Numberofcoursesa-empted 0.18 0.50 0.16 0.55
0.12 0.45 0.13 0.49
Successrate 0.10 0.29 0.08 0.27
0.07 0.25 0.07 0.25
CARNEGIEQUANTWAYSUCCESS 24
DevelopmentalEnglish
Numberofcoursesa-empted 0.10 0.37 0.08 0.33
0.08 0.35 0.08 0.32
Successrate 0.06 0.24 0.05 0.21
0.05 0.21 0.05 0.21
Developmentalreading
Numberofcoursesa-empted 0.07 0.28 0.11 0.38
0.09 0.32 0.11 0.37
Successrate 0.05 0.21 0.06 0.24
0.05 0.22 0.06 0.23
Developmentalwri/ng
Numberofcoursesa*empted 0.10 0.32 0.11 0.34
0.09 0.32 0.10 0.34
Successrate 0.07 0.25 0.07 0.26
0.06 0.24 0.07 0.26
Collegemath
Numberofcoursesa-empted 0.03 0.18 0.08 0.29
0.05 0.27 0.07 0.28
Successrate 0.01 0.07 0.01 0.08
0.00 0.06 0.01 0.07
Collegenon-math
Numberofcoursesa-empted 2.10 3.57 4.32 6.01
3.29 5.17 3.76 5.42
Successrate 0.33 0.42 0.42 0.42
0.35 0.42 0.40 0.42
CollegeSTEM
Numberofcoursesa-empted 0.30 1.11 0.49 1.30
0.44 1.48 0.45 1.25
Successrate 0.07 0.24 0.09 0.26 0.08 0.25 0.08 0.25
CARNEGIEQUANTWAYSUCCESS 25
Note.Termswith"*"wereusedasreferencecategories(codedas0)whenformula:ngdummyvariables.Agewascomputedinyearsusingadateofbirthand9/1forthefallcohortsand3/1forthespringcohorts(1/1foronewintercohortgroup).Inthecurrentpropensitymodel,wecenteredagearoundage18.Semesterssincefirstdevelopmentalmathcoursetakesaninteger,suchas0,1,2,etc.,where0meansastudenttookadevelopmentalmathcourseforthefirst(meinthesametermastheQuantway1term,1onesemesterbefore,2twosemestersbefore,andsoon.CourseloadreferstothenumberofcoursesastudenttookduringtheQuantway1term.Successratewascomputedbydividingthenumberofcoursescompletedwithapassinapass/failgradingscheme,oraCorhigher(C-ifa+/-gradingschemeisused)bythenumberofcoursesa)empted.
Table2.
BalanceinLogitofthePropensityScorefornon-Quantway1andQuantway1Students
Samplebeforematching Samplea(ermatching
Non-Quantway1
Quantway1
Non-Quantway1
Quantway1
Matchedra#oCollege Cohort
n M SD
n M SD
n M SD
n M SD t
1 2012Spring 585 -2.78 0.48 43 -2.72 0.37 212 -2.74 0.32 43 -2.72 0.37 -0.42 4.93
1 2012Fall
470 -2.41 0.31
34 -2.20 0.54
149 -2.36 0.35
31 -2.31 0.42 -0.59 4.81
1 2013Fall
305 -1.88 0.19
59 -1.79 0.38
249 -1.89 0.15
54 -1.88 0.16 -0.12 4.61
1 2014Spring
273 -2.32 0.26
29 -2.01 0.52
112 -2.23 0.26
25 -2.16 0.34 -0.94 4.48
2 2013Fall 690 -2.34 0.96 69 -2.29 0.80 337 -2.31 0.75 69 -2.29 0.80 -0.16 4.88
2 2014Spring
270 -2.93 0.62
17 -3.02 0.27
85 -3.02 0.27
17 -3.02 0.27 -0.01 5.00
2 2014Fall 402 -2.86 0.41 47 -2.99 0.36 217 -2.95 0.26 46 -2.96 0.32 0.18 4.72
3 2012Spring
4138 -3.38 0.39
72 -3.37 0.27
360 -3.37 0.27
72 -3.37 0.27 -0.05 5.00
CARNEGIEQUANTWAYSUCCESS 26
3 2012Fall
3234 -2.86 0.40
177 -2.74 0.42
875 -2.76 0.38
175 -2.76 0.38 -0.07 5.00
3 2013Spring
3745 -2.24 0.43
584 -1.93 0.75
1057 -2.11 0.53
544 -2.06 0.59 -1.66 1.94
3 2013Fall
2358 -2.11 0.51
408 -1.66 0.93
739 -1.89 0.64
378 -1.84 0.72 -1.28 1.96
3 2014Spring
3242 -2.58 0.54
290 -2.12 0.81
559 -2.20 0.72
287 -2.14 0.78 -1.08 1.95
3 2014Fall
4696 -2.46 0.36
402 -1.90 0.94
368 -2.08 0.69
368 -2.07 0.71 -0.16 1.00
4 2012Spring 594 -3.20 0.66 38 -2.78 1.27 175 -3.03 0.88 36 -2.97 0.97 -0.33 4.86
4 2012Fall
570 -2.88 0.47
45 -2.40 0.83
193 -2.67 0.48
42 -2.54 0.65 -1.22 4.60
4 2013Spring
599 -2.20 0.48
42 -1.91 0.73
188 -2.09 0.50
41 -1.96 0.68 -1.18 4.59
4 2013Fall
681 -2.33 0.46
69 -2.01 0.89
278 -2.28 0.48
63 -2.20 0.66 -0.84 4.41
4 2014Spring
617 -2.78 0.50
49 -2.43 0.81
213 -2.65 0.57
47 -2.51 0.72 -1.16 4.53
4 2014Fall 827 -2.66 0.38 87 -2.48 0.72 408 -2.63 0.41 85 -2.56 0.53 -1.17 4.80
5 2013Fall
712 -3.88 0.58
6 -3.60 0.76
30 -3.61 0.68
6 -3.60 0.76 -0.02 5.00
5 2014Spring
648 -4.58 0.58
19 -3.65 0.93
82 -3.88 0.76
19 -3.65 0.93 -0.99 4.32
6 2012Spring 1601 -4.72 0.45 21 -4.74 0.35 105 -4.74 0.34 21 -4.74 0.35 -0.05 5.00
6 2013Spring
1481 -3.75 0.55
35 -3.44 0.87
167 -3.57 0.63
34 -3.52 0.73 -0.34 4.91
6 2013Fall
1534 -3.82 0.49
49 -3.49 0.78
232 -3.60 0.65
48 -3.54 0.73 -0.53 4.83
6 2014Spring
1436 -4.20 0.45
32 -3.76 0.87
140 -3.88 0.55
29 -3.83 0.58 -0.41 4.83
6 2014Fall 1954 -4.16 0.35 17 -3.44 0.85 71 -3.70 0.63 16 -3.55 0.74 -0.72 4.44
7 2012Spring
3675 -4.79 0.91
63 -4.49 1.02
305 -4.53 0.99
62 -4.49 1.03 -0.08 4.92
7 2012Fall
2929 -4.68 0.58
65 -4.47 0.67
320 -4.52 0.58
65 -4.47 0.67 -0.35 4.92
7 2013Spring
3831 -4.05 0.49
42 -3.92 0.52
204 -3.97 0.45
42 -3.92 0.52 -0.08 4.86
7 2013Fall
3566 -4.21 0.44
68 -4.17 0.36
338 -4.18 0.34
68 -4.17 0.36 -0.02 4.97
CARNEGIEQUANTWAYSUCCESS 27
7 2014Spring
4097 -4.65 0.45
29 -4.29 0.80
127 -4.54 0.47
27 -4.41 0.70 -1.11 4.70
8 2012Winter 4693 -5.45 1.30 70 -3.90 1.55 348 -3.93 1.52 70 -3.90 1.55 -0.16 4.97
8 2012Spring
1955 -4.39 0.91
108 -2.32 1.83
275 -2.91 1.22
97 -2.72 1.36 -1.23 2.84
8 2012Fall
3420 -3.94 0.88
124 -2.14 1.69
443 -2.51 1.30
120 -2.29 1.47 -1.48 3.69
8 2013Spring
3615 -3.36 0.60
129 -1.93 1.36
119 -2.13 1.17
119 -2.10 1.20 -0.19 1.00
8 2013Fall
3230 -3.33 0.62
119 -1.87 1.46
105 -2.23 1.19
105 -2.20 1.20 -0.22 1.00
8 2014Spring
3088 -3.79 0.52
118 -2.49 1.23
165 -3.16 0.74
88 -3.04 0.83 -1.10 1.88
8 2014Fall 3796 -3.61 0.49 158 -2.55 1.37 253 -3.07 0.97 138 -2.89 1.09 -1.60 1.83
9 2012Spring
683 -2.86 0.56
41 -2.88 0.70
195 -2.97 0.56
40 -2.93 0.62 -0.37 4.88
9 2012Fall
776 -2.75 0.34
55 -2.33 1.00
226 -2.71 0.37
50 -2.58 0.54 -1.58 4.52
9 2014Spring
543 -2.59 0.38
30 -2.63 0.38
142 -2.59 0.33
29 -2.62 0.38 0.38 4.90
9 2014Fall
574 -2.49 0.29
71 -2.36 0.45
303 -2.43 0.27
68 -2.39 0.41 -0.91 4.46
10 2012Spring 520 -2.36 0.52 72 -2.41 0.58 343 -2.44 0.52 72 -2.41 0.58 -0.36 4.76
10 2012Fall
376 -2.33 0.38
48 -2.29 0.41
229 -2.32 0.37
48 -2.29 0.41 -0.43 4.77
10 2013Fall
243 -1.94 0.15
39 -1.87 0.16
164 -1.91 0.14
39 -1.87 0.16 -1.33 4.21
10 2014Spring
291 -2.03 0.40
27 -2.13 0.32
134 -2.14 0.31
27 -2.13 0.32 -0.13 4.96
10 2014Fall 324 -2.12 0.31 27 -1.76 0.65 109 -2.05 0.25 22 -2.04 0.26 -0.22 4.95
CARNEGIEQUANTWAYSUCCESS 28
Table3.
Model-BasedEs(ma(onofQuantway1EffectonDevelopmentalMathAchievement
Fixedeffect Coefficient SE t p Oddsra'o
Intercept -0.49 0.15 -3.20 .005 0.61
Term -0.71 0.14 -5.24 <.001 0.49
W12 0.08 0.14 0.55 .585 1.08
S12 0.04 0.07 0.63 .531 1.04
F12 -0.11 0.06 -1.79 .073 0.89
S13 -0.19 0.06 -2.88 .004 0.83
F13 0.11 0.06 1.83 .068 1.12
F14 0.01 0.07 0.21 .834 1.01
Quantway1 0.72 0.21 3.47 .003 2.05
Term 0.30 0.30 1.00 .317 1.35
Randomeffectatlevel4(college) Variance df χ2 p Correla'on
Intercept 0.22 9 311.84 <.001 -0.70
Quantway1 0.35 9 97.93 <.001
Randomeffectatlevel3(faculty) Variance df χ2 p Correla'on
Intercept 0.02 70 112.41 .001 -0.41
Quantway1 0.20 70 182.71 <.001
CARNEGIEQUANTWAYSUCCESS 29
Figure1.Quantway1vs.Tradi*onalmathsequence
Quantway2orotherquan+ta+ve
reasoning
CollegeMathCredit
Quantway1
Elementary
Algebra
Intermediate
Algebra
College
Math
CollegeMathCredit
Semester1 Semester2 Semester3ormore
Tradi&onalMathSequence
CARNEGIEQUANTWAYSUCCESS 30
Figure2.SixkeydriversofQuantway
AcceleratedPathwaythroughCollegeLevelMath
NetworkedImprovementCommunity
FacultyDevelopment
Socioemo'onalSupports
(Produc(vePersistence)
LearningPrinciplesforCurriculumandInstruc6on:
Produc'veStruggle,ExplicitConnec,ons,DeliberatePrac,ce
LanguageandLiteracySupports
Goal:
Increasethenumberofstudentsachievingcollegemathcreditwithinoneyearofcon*nuousenrollment
CARNEGIEQUANTWAYSUCCESS 31
Figure3.Varia,oninQuantway1effectamongcolleges
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
10 6 1 5 3 2 4 9 7 8
QW
eff
ect (
in lo
gits
)
Quantwayeffectbycollege Upperbound Averageeffect Lowerbound
CARNEGIEQUANTWAYSUCCESS 32
Figure4.Varia,oninQuantway1effectamongfacultymembers
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
10_0
2 10
_01
3_13
1_
05
1_04
6_
05
3_26
3_
11
3_07
6_
01
6_02
10
_03
3_27
3_
14
4_03
6_
04
6_03
3_
24
3_02
3_
09
3_16
3_
12
5_01
3_
20
2_01
4_
04
2_02
4_
06
1_01
9_
06
4_07
3_
18
4_05
3_
03
9_05
9_
02
5_02
3_
06
3_15
3_
08
7_02
9_
03
4_01
3_
01
3_22
1_
02
9_01
2_
03
3_10
3_
19
3_25
7_
05
3_17
7_
01
9_04
4_
02
3_28
3_
04
3_21
3_
23
1_03
7_
03
9_07
8_
04
3_05
7_
04
8_05
8_
03
8_02
8_
15
8_08
8_
12
8_09
8_
07
8_10
8_
16
8_06
8_
01
8_14
8_
11
8_13
Qua
ntw
ay e
ffec
t (in
logi
ts)
Quantway effect by faculty Upper bound Average effect Lower bound
CARNEGIEQUANTWAYSUCCESS 33
Figure5.Model-basedsuccessratesbysexandrace/ethnicity
69%
48% 53%
58%
68%
46% 47%
61% 52%
30% 35%
40%
51%
29% 30%
43%
0%
20%
40%
60%
80%
100%
White Black Hispanic Other White Black Hispanic Other
Female Male
Succ
ess R
ate
(Pas
s or
C o
r hi
gher
)
Quantway Matched comparison
CARNEGIEQUANTWAYSUCCESS 34
CarnegieFoundationfortheAdvancementofTeaching51VistaLaneStanford,California94305650-566-5100
CarnegieFounda-onfortheAdvancementofTeachingseekstovitalizemoreproduc2veresearchanddevelopmentineduca2on.Webringscholars,prac//oners,innovators,designers,anddeveloperstogethertosolveprac/calproblemsofschoolingthatdiminishourna'on’sabilitytoeducateallstudentswell.Wearecommi,edtodevelopingnetworksofideas,exper9se,andac9onaimedatimprovingteachingandlearningandstrengtheningtheins2tu2onsinwhichthisoccurs.Ourcorebeliefisthatmuchmorecanbeaccomplishedtogetherthaneventhebestofuscanaccomplishalone.
www.carnegiefoundation.org
Weinviteyoutoexploreourwebsite,whereyouwillfindresourcesrelevanttoourprogramsandpublicationsaswellascurrentinformationaboutourBoardofDirectors,funders,andstaff.
©2016CarnegieFounda,onfortheAdvancementofTeaching