E2MC Evaluation report Final 8-10-15V-2 · grade levels with peers to plan and design lessons. They...

cultivating learning and positive change

www.magnoliaconsulting.org

AnEvaluationoftheExploringEnergy&MatterCollaborative(E2MC)Mathematics

andSciencePartnershipProject

August10,2015

Evaluation of the E2MC Project Magnolia Consulting, LLC, August 10, 2015

ii

Executive Summary Withtheever-increasingneedtoprovidestudentswithhigh-qualityscienceeducation,itisimperativethatteachershaveaccesstoqualityprofessionaldevelopment(PD)thatsupportsthemineffectivelyaddressingthesciencestandardsintheirclassrooms.InresponsetotheneedforqualityPDopportunities,NorthernArizonaUniversitycollaboratedwithPeoriaUnifiedSchoolDistrictandGilbertPublicSchoolstooffertheExploringEnergyandMatterCollaborative(E2MC)project,fundedbytheArizonaDepartmentofEducation’sMathematicsandSciencePartnership(MSP)program.E2MCPDprovidersutilizedtheWestEdMakingSenseofSCIENCE(MSS)MatterandEnergycoursestosupportteachercontentknowledgeandeffectivepedagogicalpractices.InanefforttounderstandtheefficacyofE2MCinmeetingitsobjectives,PeoriaUnifiedSchoolDistrictcontractedwithMagnoliaConsulting,LLC,anindependentevaluationfirm,toconductastudyoftheprogramduringthe2014–2015schoolyear.Participantsinthestudyincluded78teachers(39treatmentand39comparison)andasampleof524studentsinparticipatingteachers’classrooms.Study Design & Methods ThepurposeofthestudywastoevaluatetheeffectivenessoftheE2MCprojectinincreasingteachers’physicalsciencecontentknowledgeandunderstandingofeffectivesciencepedagogy.Thestudyalsosoughttounderstandimpactsonstudentsinparticipatingclassrooms.Evaluatorsusedaquasi-experimentaldesignwithamatchedcomparisongroupforthestudy.Teachermeasuresincludedthefollowing:theReformedTeachingObservationProtocol(RTOP),theDiagnosticTeacherAssessmentsforMathematicsandScience(DTAMS)physicalsciencetest,theUnderstanding

ScienceforTeachingMatterandEnergyAssessments(USTMEA),formativeclassroomobservations,instructionalartifacts,andtreatmentandcomparisonteacherimplementationssurveys.EvaluatorsusedtheUSTMEAforassessingstudentlearninginsampleclassrooms. Program Perceptions and Perceived Impacts

E2MCteacherparticipantsfeltthattheprogramwasofvaluetotheirunderstandingofphysicalscienceconceptsrelatedtomatterandenergy.Teachersvaluedmanyaspectsoftheexperience,particularlyworkingcollaborativelyacrossmiddleandhighschoolgradelevelswithpeerstoplananddesignlessons.Theycameawaywithabetterunderstandingofhowmatterandenergyareconnectedandhowconceptsrelatetootherareasofinstruction.TeachersfelttheMakingSenseofSCIENCEcoursesprovidedvaluableresourcesfortheirinstruction.Theyappreciatedtheconnectionsofliteracytosciencecontent.Theyalsogreatlyvaluedtheopportunitytopracticetheirlearningthroughhands-oninvestigations.TeachersindicatedthatthebiggestbarriertoimplementinglearningfromthePDislackoftimeforplanningandreflectingonlessondesignandimplementation. TeacherswhoparticipatedintheE2MCprofessionaldevelopmentprogramindicatedhigherconfidenceintheirabilitiestousementalmodelsintheirscienceinstruction

KeyQuestion:Howdoteachersperceivethequality,utility,andeffectivenessoftheE2MCprofessionaldevelopment?

KeyQuestion:HowwellprepareddoteachersfeeltoapplyE2MClearningintheirscienceclassrooms?


iii

thandidnonparticipants.Theyalsofeltmoreconfidentintheirabilitytomakeconnectionsbetweenmatterandenergyintheirinstruction.Participantsfeltmostconfidentwithrespecttoguidingstudentstouseevidenceindevelopingexplanations,providingopportunitiestopresentfindingsinascientificformat,supportingstudentstocommunicateandjustifytheirexplanationsofphenomena,andhelpingstudentstounderstandwhichtypesofquestionscanbeansweredthroughscientificinvestigations.ParticipantsratedthemselveshigherthannonparticipantsonmanyoftheinstructionalstrategiestargetedinthePDsessions.Applications to Practice

FormativeobservationsshowedthatE2MCteachersuseavarietyofresourcesandcollaborativelearningstrategiesintheirinstruction.Observedlessonswerestructuredcoherentlyandlogically.Teachersappearedtoneedmoresupportforactivelearningstrategiesandopportunitiestoanswerscientificquestions,whichwereemphasizedinthePDsessions.Teachersaremakingstrongerconnectionsbetweenmatterandenergyintheirinstruction.Unitplansshowedvariedproficienciesataligninginstructiontolearninggoals. Impacts on Instruction

TeacherswhoparticipatedintheE²MCPDprogramshowedsignificantgainsininstructionalpracticealignedtoreformedteachingasevidencedbytheirgainsfrompretesttoposttestontheRTOP.Treatmentteachergainscorrespondedtoalargeeffectsizeof0.51.

E²MCteachersscoredhigheronthepostobservationthandidcomparisonteachers.Differencesbetweentreatmentandcomparisonteacherposttestscorescorrespondedtoasmalleffectsize(etasquared=.07).Impacts on Teacher Content Knowledge

E²MCteachersmadesignificantgainsontheDTAMSfrompretestingtoposttestingontheTotalContentandTotalKnowledgeTypescores.Teachersalsomadesignificantgainsontheenergysubtestandsubscoresforschematicknowledgeandpedagogicalcontentknowledge.E2MCteachersdidnotscoresignificantlyhigherthancomparisonteachersontheposttestTotalContentandTotalKnowledgeTypescores.

19.13

28.23

10

15

20

25

30

Pretest Posttest

E2MC Teacher RTOP Scores Pretest to Posttest

KeyQuestion:HowdoesparticipationintheE2MCprojectimpactteacherpracticewithrespecttoeffectivelessonplanningandimplementation?

KeyQuestion:Doesparticipationintheprojecthaveastatisticallysignificantimpactonteachers’abilitytoimplementeffectivescienceinstructionalstrategiesandpedagogy?

KeyQuestion:DoesparticipationintheE2MCprojecthaveastatisticallysignificantimpactonteacherunderstandingofinterconnectedcoreideasandcrosscuttingconceptsrelatingtomatterandenergy?


iv

E²MCteachersmadestatisticallysignificantgainsontheUSTMEAtestsofmatterandenergycontentknowledgealignedtotheMakingSenseofSCIENCEcourses.Gainscorrespondedtolargeeffectsizesformatter(0.41)andenergy(0.63).Student Performance Results

Onaverage,studentmatterandenergyscoresstatisticallysignificantlyincreasedfrompretesttoposttest,andresultscorrespondedtolargeeffectsizes.

Exploratoryanalysesofstudentandteachercharacteristicsshowthatstudentgradeisstatisticallysignificantlyrelatedtomattergain

scores,withstudentsinhighergradesearningmorethanstudentsinlowergrades.Forteachercharacteristics,teachers’yearsofexperienceisalsoastatisticallysignificantpredictorforenergygainscores,withthestudentsofmoreexperiencedteachersgainingmorethanthestudentsofteacherswithfeweryearsofexperience.Fortheremainingsubgroupanalyses,resultsindicatethatstudentsperformedsimilarlyovertime.Forteacherimplementation,resultswerenotstatisticallysignificantbutcorrespondedtosubstantivelyimportanteffectsizes(>0.25),suggestingthatlargerimplementationscorescorrespondedtolargergains. Conclusions TeacherswhoparticipatedintheE²MCprofessionaldevelopmentfeltthattheprogramdeepenedtheirunderstandingofenergyandmatterconceptsandofeffectivescienceteachingstrategies.Teachersvaluedtheabilitytocollaboratewithpeerstodevelopunitplansandappreciatedthecross-gradecollaborationsbetweenmiddleandhighschoolteachers.TeachersgainedconfidenceindesigningopportunitiesforstudentsOverall,findingsfromthisevaluationstudyfoundthattheE²MCprofessionaldevelopmentprojectshowedmanyofthecharacteristicsofeffectivePD.Useofhigh-qualitycurricularmaterials,suchastheMakingSenseofSCIENCEcourses,supportedstructuredandsystematiclearningforteachersandallowedthemtoexperiencecontentandinvestigationsastheirstudentswouldexperiencethem.PDsessionsemphasizedalignmentofactivities,lessons,andunitstoinstructionalgoalsandpromoteactivelearningintheclassroom.

12.44

18.44

0

5

10

15

20

Pretest Posttest

Student USTMEA Matter Scores Pretest to Posttest

11.15

15.55

0

5

10

15

20

Pretest Posttest

Student USTMEA Energy Scores Pretest to Posttest

KeyQuestion:Dostudentsinparticipatingteachers’classroomsexperiencestatisticallysignificantgainsincontentknowledgeoverthecourseofthestudy?


v Eval

Acknowledgements

ThisstudyrepresentsacollaborativeeffortamongMagnoliaConsulting,NorthernArizonaUniversity’sCenterforScienceTeachingandLearning,PeoriaUnifiedSchoolDistrict,andGilbertPublicSchools.MagnoliaConsultingevaluatorswishtothankeveryonewhohelpedtomakethisworkpossible.Wewouldespeciallyliketoexpressourgratitudetothestudyparticipants,includingE2MCteacherandstudentparticipantsandcomparisonteachersfortheircontributionstodatacollectionefforts,theirinsightsandfeedbackabouttheprogram,andtheirinstruction.WealsoappreciatethesupportofDr.Billie-JoGrant,SeniorEvaluatoratMagnoliaConsultingforstudentassessmentanalysis. Theauthors, CarolHaden,Ed.D.PrincipalEvaluatorAnneCosby,M.S.WResearchAssistantMagnoliaConsulting,LLC1535BlenheimRd.Charlottesville,VA22902(ph)855.984.5540(tollfree)http://www.magnoliaconsulting.org


vi

Table of Contents Introduction.................................................................................................................................................1ProgramDescription...................................................................................................................................2ResearchDesign..........................................................................................................................................4

Study Purpose ............................................................................................................................................ 4Formative Evaluation Questions ..............................................................................................................................4Summative Evaluation Questions ............................................................................................................................4

Measures .................................................................................................................................................... 5Teacher Measures ....................................................................................................................................................5Student Measures ....................................................................................................................................................7

Data Preparation and Analysis .................................................................................................................... 7Study Procedures ........................................................................................................................................ 8

Study Timeframe ....................................................................................................................................................8Settings ...................................................................................................................................................... 9

Participants.................................................................................................................................................11Teacher Attrition ...................................................................................................................................... 11Analysis Sample ........................................................................................................................................ 11Teacher Participants ................................................................................................................................. 11Group Equivalence ................................................................................................................................... 12Student Participants ................................................................................................................................. 12

TreatmentandComparisonTeacherScienceInstruction.......................................................................14Science Instruction ................................................................................................................................... 14Coverage of Matter and Energy in Instruction .......................................................................................... 15Energy and Matter Implementation Scores ............................................................................................... 15

E2MCTeacherProgramPerceptions.........................................................................................................16Professional Development Feedback ......................................................................................................... 16Perceptions of Impacts on Instruction ...................................................................................................... 17Barriers to Implementing Learning from E2MC ........................................................................................ 18Summary of Findings Related to Program Perceptions ............................................................................. 19

Teachers’PerceptionsofAbilitiestoImplementEffectiveStrategiesandContent..............................20Teacher Confidence in Use of Instructional Strategies .............................................................................. 20Ability to Make Connections Between Matter and Energy in the Science Classroom ............................... 22Summary of Findings Related to Perceived Ability to Implement Learning .............................................. 22

E2MCTeacherApplicationofLearningtoInstruction..............................................................................23Formative Lesson Observations ................................................................................................................ 23Integration of Matter and Energy ............................................................................................................. 25Unit Plan Development ........................................................................................................................... 26Summary of Findings Related to Application to Instruction ..................................................................... 26

TeacherPerformanceResultsforInstructionalPractice.........................................................................28Descriptive Analyses Regarding Treatment Teachers’ Instructional Practice ............................................. 28


vii

Paired Samples t-Test Analyses Examining Treatment Teachers’ Changes in Instructional Practice .......... 29Comparison of Teacher RTOP Scores by Condition ................................................................................ 29

Descriptive Analysis Comparing RTOP Scores by Condition ................................................................................29Analysis of Covariance Comparing E2MC and Comparison Teachers Posttest Scores on the RTOP ......................29

TeacherPerformanceResultsforPhysicalScienceContentKnowledge................................................31Descriptive Findings and Inferential Analyses Regarding Teacher DTAMS Gains Among E²MC Program Participants .............................................................................................................................................. 31Descriptive Examination of E²MC Teachers’ DTAMS Total Content Scores and Total Knowledge Type Scores ....................................................................................................................................................... 31Paired Samples t-tests Examining E²MC Teachers’ DTAMS Total Content Scores and Total Knowledge Type Scores ....................................................................................................................................................... 32Descriptive Examination of E²MC Teachers’ DTAMS Matter and Energy Subscores ............................... 33Paired Samples t-tests Examining E²MC Teachers’ Matter & Energy Subscore Gains on the DTAMS ..... 33Descriptive Examination of E²MC Teachers’ DTAMS Knowledge Type Subscores ................................. 34Paired Samples t-tests Examining E²MC Teachers’ Total Knowledge Type Subscore Gains on the DTAMS ................................................................................................................................................................. 35Comparison of Teacher DTAMS Scores by Condition ............................................................................ 35

Descriptive Analysis Comparing DTAMS Scores by Condition ............................................................................35Analysis of Covariance Comparing E2MC and Comparison Teachers Posttest Scores on the DTAMS ..................36Descriptive Examination of E²MC Teachers’ Matter & Energy Score Gains ..........................................................38Paired Samples t-tests Examining E²MC Teachers’ Matter & Energy Score Gains ................................................38

Summary of Teacher Performance Results ................................................................................................ 39

StudentPerformanceResults...................................................................................................................40Descriptive Examination of E²MC Students’ Matter & Energy Score Gains ..........................................................40Multilevel Modeling Analyses Examining E²MC Students’ Matter & Energy Score Gains ....................................41

Relationship Between Student Characteristics and E²MC Students’ USTMEA Gain Scores ..................... 42Descriptive Examination of E²MC Students’ Matter & Energy Score Gains by Student Characteristics ................42Multilevel Modeling Analyses Examining E²MC Students’ Matter & Energy Score Gains by Student Characteristics .......................................................................................................................................................45Descriptive Examination of E²MC Students’ Matter & Energy Score Gains by Teacher Characteristics ................46

Relationship between Teacher Characteristics and E²MC Students’ USTMEA Gain Scores ..................... 49Summary of Student Performance Results ................................................................................................ 50

SummaryandDiscussion..........................................................................................................................51References.................................................................................................................................................53AppendixA:ProfessionalDevelopmentTopics.......................................................................................55AppendixB:MissingDataRatesbyAssessment.....................................................................................57AppendixC:TeacherDemographicsandTestsofEquivalence..............................................................58AppendixD:DemographicsforStudentsParticipatingintheUSTMEAAssessments..........................59AppendixE:TreatmentandComparisonTeachers’ScienceResources...............................................60AppendixF:TreatmentandComparisonTeachers’UseofAssessment................................................61


viii

AppendixG:EnergyandMatterContentCoverage................................................................................62AppendixH:E2MCTeacherPerceptionsofProfessionalDevelopmentSessions...................................64

FiguresFigure 1. Teachers explore endothermic and exothermic reactions during a follow-up PD session. ................. 3Figure 2. Frequency of opportunities for students to apply and use scientific practices. ................................ 15Figure 3. Teachers participate in a sensemaking activity during a PD session. ............................................... 16Figure 4. Treatment and comparison teacher mean ratings of confidence in instructional practices. ............. 21Figure 5. Treatment and comparison teacher ratings of confidence in using mental models in instruction. .. 21Figure 6. Treatment and comparison teacher ratings of confidence in making connections between matter and energy in instruction. ............................................................................................................................. 22Figure 7. Ratings of formative lesson observations of E2MC participants (n=11). ........................................ 24Figure 8. Pretest and posttest RTOP scores for teachers participating in the E2MC project (n=39). .............. 28Figure 9. Pretest and posttest mean RTOP scores for treatment and comparison teachers. ........................... 29Figure 10. Pretest and posttest teacher DTAMS Total Content Scores for E2MC partticipants 9 (n=39). ....... 32Figure 11. Pretest and posttest teacher DTAMS Total Knowledge Type scores for E2MC participants (n=39). ..................................................................................................................................................................... 32Figure 12. Pretest and posttest teacher DTAMS Matter Subscores for E2MC participants (n=39) ................ 33Figure 13. Pretest and posttest teacher DTAMS Energy subscores for E2MC participants (n=39). .............. 33Figure 14.Pretest and posttest teacher DTAMS Scientific Inquiry Subscores for E2MC participants (n=39). 34Figure 15. Pretest and posttest teacher DTAMS Schecmatic Knowledge Subscores for E2MC participants (n=39). ......................................................................................................................................................... 34Figure 16. Pretest and posttest teacher DTAMS PCK Subscores for E2MC participants (n=39). .................. 34Figure 17. Pretest and posttest mean DTAMS Total Content scores for treatment and comparison teachers. 35Figure 18. Pretest and posttest mean DTAMS Total Knowledge Type scores for treatment and comparison teachers. ....................................................................................................................................................... 36Figure 19. Pretest and posttest teacher matter scores for E2MC participants (n=39). ..................................... 38Figure 20. Pretest and posttest teacher energy scores for E2MC participants (n=39). ..................................... 38Figure 21. Unadjusted pretest and posttesnt student USTMEA matter scores for E2MC participants. .......... 41Figure 22. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants. ............ 41Figure 23. Unadjusted pretest and posttest student USTMEA matter scores for E2MC participants by gender ..................................................................................................................................................................... 42Figure 24. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants by FRL. 42Figure 25.Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by ethnicity. ...................................................................................................................................................... 43Figure 26. Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by grade. ..................................................................................................................................................................... 43Figure 27. Unadjusted pretest and posttest student USTMEA energy scores for E2MC participants by gender. ..................................................................................................................................................................... 44Figure 28. Undadjusted pretest and posttest student USTMEA energy scores for E2MC participants by FRL. ..................................................................................................................................................................... 44


ix

Figure 29. Unadjusted pretest and posttest student USTMEA scores for E2MC participants by ethnicity. .... 44Figure 30. Unadjusted pretest and and posttest student USTMEA energy scores for E²MC participants by grade. ........................................................................................................................................................... 45Figure 31. Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by teachers’ years of teaching experience. ........................................................................................................... 47Figure 32. Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by teachers’ level of implementation (percentage score). .................................................................................... 48Figure 33. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants by teachers’ years of teaching experience. ........................................................................................................... 48Figure 34. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants by teachers’ level of implementation (percentage score). .................................................................................... 49TablesTable 1. Timeline of E2MC Study Activities .................................................................................................. 9Table 2. District-level Demographics of Participating E2MC School Districts .............................................. 10Table 3. Implementation Score Descriptive Statistics for Treatment and Comparison Teachers ................... 15Table 4. Rubric for Analyzing Teachers' Unit Plans ..................................................................................... 26Table 5. Results of a Paired Samples t-test of RTOP Scores for Participating Teachers ................................. 29Table 6. Treatment and Comparison Pretest and Posttest Mean Scores on the RTOP ................................. 30Table 7. Results of a Paired Samples t-test of DTAMS Total Knowledge Type Scores for Participating Teachers ..................................................................................................................................................................... 32Table 8. Results of a Paired Samples t-test of DTAMS Matter and Energy Subscores for Participating Teachers ....................................................................................................................................................... 33Table 9. Results of a Paired Samples t-test of DTAMS Knowledge Type Subscores for Participating Teachers ..................................................................................................................................................................... 35Table 10. Treatment and Comparison Teachers’ Average Pretest and Posttest DTAMS Total Content and Total Knowledge Type and Subtest scores. .................................................................................................... 37Table 11. Results of Paired Samples t-tests For USTMEA Assessments for E2MC Teachers .......................... 39Table 12. Mean USTMEA Matter and Energy Pretest-to-Posttest Gain Scores for E²MC Students ............ 41Table 13. Relationship Between Student Characteristics and USTMEA Matter Scores ................................ 46Table 14. Relationship Between Student Characteristics and USTMEA Energy Scores ................................ 46Table 15. Relationship Between Teacher Characteristics and USTMEA Matter Scores ................................ 49Table 16. Relationship Between Teacher Characteristics and USTMEA Energy Scores ................................ 49


1 Eval

Introduction

Thereisnodoubtthatscience—and,therefore,scienceeducation—iscentraltothelivesofallAmericans.Neverbeforehasourworldbeensocomplexandscienceknowledgesocriticaltomakingsenseofitall.

–IntroductiontotheNextGenerationScienceStandards

WiththeintroductionoftheFrameworkforK–12ScienceEducation(NationalResearchCouncil,2012)andtheNextGenerationScienceStandards(NGSSLeadStates,2013),itisimperativethatteachershaveaccesstoqualityprofessionaldevelopmentthatsupportsthemineffectivelyaddressingthestandardsintheirclassrooms.Professionaldevelopmentthatsupportsteachersinexamininginstructionalgoalsandstrategiesthataligntothestandardshasthepotentialtogreatlyimpactteacherpracticeandstudentlearning.

Researchshowsthateffectiveprofessionaldevelopmentexperiencesshareasetofcommoncharacteristics.Experiencesshouldfocusonbuildingspecificcontentknowledge,beofsufficientdurationandintensity,becoherentwithotherschoolpoliciesandpractices,andengageteachersinauthentic,activelearningexperiencesthatenablethemtoexperiencelessonsastheirstudentswouldexperiencethem(Garet,Porter,Desimone,Birman,&Yoon,2001)Additionally,researchershavenotedthateffectivePDusesmaterialsthatareeducativeforteachersandtheirstudentsandarealignedtoclassroompractice(Penuel,Fishman,Yamaguchi,&Gallagher,2007).Wheninnovativematerialsandstrategiesareused,itisalsoessentialthatteachersreceivedirectinstructioninthemodelsofteachingassociatedwithparticularmethodsofdesigninginstruction(Penuel,Gallagher,&Moorthy,2011).

WithrespecttoimplementingtheNGSS,teachersfacechallengessuchasunderstandingthecoherenceoftheirinstructionalgoals,standards,curricula,anddistrictpoliciesandpriorities.Collaborativesensemakingcanhelpteachersmitigateincoherencethatmayaffecttheirimplementationofnewstandards.Thus,incorporatingsensemakingactivitieswhereteacherscanengagecollaborativelywithoneanotherduringPDcancontributetomoresuccessfulimplementationofstandard-alignedpractices(Allen&Penuel,2014).

Tosupportqualityprofessionaldevelopmentforteachersalignedtotheeffectivepracticesnotedabove,theArizonaDepartmentofEducation’sMathematicsandSciencePartnership(MSP)programfundedtheExploringEnergyandMatterCollaborative(E2MC)teacherPDprojectinresponsetotheneedtoimproveteacherandstudentcontentknowledgeinthephysicalsciencesintwoschooldistrictsinArizona.Theprojectalsosoughttoprovideteacherswitheffectivemeansofintegratingreadingandwritingstrategiesintothecontentclassroom. InanefforttoevaluatethequalityandeffectivenessoftheE2MCproject,PeoriaUnifiedSchooldistrictcontractedwithMagnoliaConsulting,LLC,anexternalevaluationcompany,toconductanevaluationoftheproject.Thisreportpresentsfindingsoftheevaluationstudy.


2 Eval

Program Description

ThroughacollaborativeeffortbetweenthedistrictsandtheCenterforScienceTeachingandLearning(CSTL)atNorthernArizonaUniversity(NAU),middleandhighschoolteachersparticipatedinprofessionaldevelopmentdesignedtobuildcontentknowledgeinphysicalsciences(matterandenergy)andpedagogicalknowledgeofeffectivestrategiesforteachingmatterandenergyconceptstotheirstudents.Thespecificcontentfocuswasonmattercycling,energyflowingthroughsystems,andfoundationalandcrosscuttingconceptsforunderstandinghowtheuniverseanditsmanysubsystemswork.Theprofessionaldevelopment(PD)experienceaddressedthefollowingobjectives:Objective1:Improvemiddleandhighschoolteachers’contentknowledgeofphysicalscienceconceptssurroundingmatterandenergybydoingthefollowing:

• developingconceptualunderstandingofinterconnectedcoreideasandcrosscuttingconceptsrelatingtomatterandenergy

• developingandusingmodelsandengaginginargumentfromevidence

Objective2:TeacherswillimprovetheirpedagogicalknowledgeofeffectivewaystoteachphysicalscienceconceptsbyusingtheAZScienceStandardsandthelearningprogressionsintheFrameworkandNGSSto

• analyzestudentworkinordertoassessstudentunderstanding;• modifyinstructiontosupportstudentunderstanding;• supportstudentconceptualunderstandinginscienceusingandapplyingArizona’s

CollegeandCareerReadiness(AzCCR)standardstoread,write,usediscourse,andapplymathematicsconcepts;and

• supportstudentunderstandinganduseofthescienceandengineeringpracticeswithparticularemphasisontheabilitytodevelopandexplainmodelsandengageinargumentfromevidence.

Objective3:IncreasenumbersofhighlyqualifiedandappropriatelycertifiedscienceteachersofphysicalsciencesinPeoriaandGilbertschooldistrictstomeetstudents’needsinGrades6through12.

Teachersparticipatedintwoweeks(70hours)ofintensivecourseworkusingtheMakingSenseofSCIENCE(MSS)MatterandEnergycourses.Allparticipatingteachersspentoneweekofthesummerinstituteonthemattercourseandoneweekontheenergycourse.MSScoursesbuildonmorethanadecadeofresearchanddevelopmentatWestEd,anonprofiteducationalresearchcompany(WestEd,2010;Daehler,etal.2012).DevelopedbytheUnderstandingScienceforTeachingprojectteamatWestEdthroughfundingbyNationalScienceFoundationandInstituteofEducationSciences,MSScoursesarealignedtostateandnationalstandardsandaredesignedtosupportteachersastheyexploretheintersectionofscienceknowledge,teaching,andliteracy,whichenablesthemtosupportstudentsinmakingsenseofessentialsciencecontent.(WestEd,2013;http://we-mss.weebly.com/our-model.html).

Eachdayofeachcourseisdividedintofourcomponentsstructuredtobuildteachers’understandingofatopic’sfoundationalconcepts,includinghowstudentsmightfindtheseconceptschallengingaswellaswaystouseliteracy,models,anddiscoursetosupportstudents’conceptual


3 Eval

understanding.ContentbuildsbetweentheMSSMatterandEnergycourseshelpingteachersseeandapplytheinterconnectednessoffoundationalconceptsthroughvarioussciencedisciplines.ThefourcoursecomponentstoMSScoursesinclude

1. scienceinvestigations,duringwhichteachersworkwithincollaborativegroupstoconductinquiry-basedinvestigationsanddevelopconceptualunderstanding;

2. teachinginvestigations,duringwhichteachersreadandcollaborativelyreflectoninstructionalmovesandstudentthinkingpresentedinateachingcaserelatedtotheconcepttheyareexploring;

3. literacyinvestigations,whichareopportunitiesforteacherstoexaminehowtosupportliteracylearninginscience;and

4. classroomconnections,timeformetacognitiontobringthecontentofthedayandweektogetherintoameaningfulwhole.

Followingtheintensivesummercoursework,teachersparticipatedinfour10-hourfollow-upsessionsthroughoutthe2014–15academicyear.ACSTLPDcoordinatorandacontentfacultypartnerfacilitatedtheFridayeveningandSaturdayfollow-upsessions.Sessionsprovidedadditionalcontentsupportforteachersbasedonneedsidentifiedduringthesummercourseworkandthroughteacherfeedbacksurveys.PDalsofocusedonpedagogicalcontentinordertosupportteachers’learning,reinforcing,consolidating,andimplementingthecontent.Sessionsprovidedteacherswithexpectationstoapplyandprovideevidenceoftheirlearningintheformofartifactssuchaslessondesigns,collaborativeunitdesign,observedclassroominstruction,analysisofstudentwork,andassessmentresults.AppendixApresentstheE2MCprofessionaldevelopmentscheduleandBigIdeasforeachsession.

Figure 1. Teachers explore endothermic and exothermic reactions during a follow-up PD session.


4 Eval

Research Design

TounderstandhowwelltheE2MCprojectmetitsobjectivesforparticipatingteachers,evaluatorsusedaquasi-experimentaldesign,specificallyamatchedcomparison-groupstudytoevaluatetheE2MCproject.Quasi-experimentaldesignsareusedinsocialsciencesresearchwhenrandomassignmenttocomparisonandtreatmentgroupsisnotfeasible(Fitzpatrick,Sanders,&Worthen,2004).

Tominimizethreatstovalidityduetothelackofrandomassignmentofparticipantstocomparisonandtreatmentgroups,participantswerematchedonkeycharacteristicsbelievedtoinfluencethekeyoutcomesoftheproject.Evaluatorsusedpropensityscorematchingtechniquestocreateacomparisongroupofteachersfromthepoolofavailablecomparisonteachers.Whenrandomassignmenttostudyconditionisnotpossible,propensityscorematchingprovidesastatisticalmethodforbalancingtwononequivalentgroupsonobservedcovariates,thusprovidingamoreaccurateunderstandingoftheeffectofatreatment(Luellen,Shadish,&Clark,2005).Demographicvariablesselectedformatchingincludedyearsofteachingexperience,gradelevel,subjecttaught,andhighestdegreeattained.Resultsoftestsforgroupequivalencyarepresentedintheparticipantsectionofthereport. Study Purpose

ThepurposeofthisstudywastoevaluatetheeffectivenessoftheE2MCprofessionaldevelopmentprograminmeetingitsobjectives.Evaluationincludedaformativecomponenttoprovidetimelyfeedbacktoprogramproviderstoallowformonitoringandadjustmentofprogramdelivery.Summativeevaluationgathereddatatomeasureimpactsonparticipatingteachers’contentknowledge,classroompractice,andconfidenceinincorporatingeffectiveinstructionalstrategiesintotheirscienceclassrooms.Thefollowingquestionsguidedtheevaluationstudy:Formative Evaluation Questions

1. Howdoteachersperceivethequality,utility,andeffectivenessoftheprofessionaldevelopmentandinstructionalmaterialsinbuildingtheircapacityforeffectivescienceinstructionofinterconnectedcoreideasandcrosscuttingconceptsrelatingtomatterandenergy?

2. AsaresultofE2MCprofessionaldevelopment,howwellprepareddoteachersfeeltoimplementtargetedcontentknowledgeandpedagogicalpracticesintheirscienceclassrooms?

Summative Evaluation Questions

1. HowdoesparticipationintheE2MCprojectimpactteacherpracticewithrespecttoeffectivelessonplanningandimplementation?

2. Doesparticipationintheprojecthaveastatisticallysignificantimpactonteachers’abilitytoimplementeffectiveinstructionalstrategiesandpedagogyinthescienceclassroom?


5 Eval

3. Doesparticipationintheprojecthaveastatisticallysignificantimpactonteacherunderstandingofinterconnectedcoreideasandcrosscuttingconceptsrelatingtomatterandenergy?

4. Dostudentsinparticipatingteachers’classroomsexperiencestatisticallysignificantgainsincontentknowledgeoverthecourseofthestudy?

Measures

Thestudyemployedamixed-methoddesignusingqualitativeandquantitativemethodstoaddressthekeyevaluationquestions.Themeasuresaredescribedinthefollowingsections. Teacher Measures

Toaddresstheevaluationquestionsrelatedtoteacheroutcomes,evaluatorsusedacombinationofmethods.Theseincludedcontentassessments,classroomobservations,teachersurveys,teacherinterviews,andprofessionaldevelopmentsessionobservations. Diagnostic Teacher Assessments in Mathematics and Science (DTAMS)

TheUniversityofLouisville’sCenterforResearchinMathematicsandScienceTeacherDevelopmentdevelopedtheDTAMStoassessteachers’contentknowledgeinmathematicsandscience.Thephysicalscienceassessmentcontains20multiplechoiceandfiveopen-endeditemsonthefollowingphysicalscienceconcepts:propertiesofmatter,motionandforces,andenergy.Itemsarealsocategorizedintothetypesofknowledgetheyrepresent:

• declarativeknowledge(definitionsandfacts)• scientificinquiryandprocedures• schematicknowledge(deepunderstandingofscienceconcepts,laws,andtheories)• pedagogicalcontentknowledge(strategicknowledgeforscienceteaching).

TotalscoresaregivenforContent(25points)andforKnowledgeType(35points).Subscoresare

reportedfortheTotalKnowledgeTypeandTotalContent1.Forthepurposesofthisproject,39treatmentand39comparisonteacherscompletedthepretestinthespringof2014priortothestartoftheprofessionaldevelopment,andtheposttestwasinthespringof2015. Reformed Teaching Observation Protocol (RTOP)

TheRTOPwasdevelopedtoassessthedegreetowhichscienceandmathematicsinstructionrepresents“reformed”teaching(i.e.,usinginstructionalstrategiesthatencouragecollaborativelearning,focusonaninquiry-basedapproach,andprovideopportunitiesforstudentstoexplainandjustifytheirwork)(Piburn,2000).TheRTOPcontains25itemsinfivecategories:(1)LessonDesignandImplementation,(2)Content:PropositionalKnowledge;(3)Content:ProceduralKnowledge;(4)ClassroomCulture:CommunicativeInteractions;and(5)ClassroomCulture:Student/TeacherRelationships.IndividualitemsaremeasuredonaLikertscalefrom0=notobservedto4=very

1Informationonthescienceassessmentscanbefoundonlineat:http://louisville.edu/education/centers/crmstd/diag-sci-assess-middle.


6 Eval

descriptive.PossibletotalscoresontheRTOPrangefrom0to100points.Oftheteachersinthestudy,39treatmentteachersand39comparisonteacherswereobservedpriortothebeginningofprofessionaldevelopmentinAprilof2014study,and38treatmentand39comparisonteacherswereobservedattheendofthestudyinMarch–Aprilof2015.Understanding Science for Teaching Matter and Energy Assessments (Teachers)

WestEdandHellerResearchAssociatesdevelopedtheUnderstandingScienceforTeachingMatterandEnergyAssessmentsinalignmentwiththeMakingSenseofSCIENCEPDcourses.Theassessmentscontainacombinationof30multiple-choiceandthreeopen-endedresponsestounderstandtheextentofknowledgeofmatterandenergyconcepts.Evaluatorsadministeredtheassessmentstotreatmentteachersasapre/postmeasureofcontentknowledge.ThesemeasuresarealignedwiththeMSScurriculumonmatterandenergyandthusallowforavalidmeasureofthelearninggainsfromparticipationinthecurriculumandPDexperience.

Professional Development Feedback Surveys

E2MCparticipantscompletedevaluationsattheendofsummerinstituteandattheendofeachfollow-upPDsession.FeedbacksurveysconsistedofacombinationofLikert-scaledandopen-endedquestionsaboutworkshopquality,utility,delivery,pacing,andformataswellaspreparednesstoimplementthelearningintheclassroom. Treatment Teacher Implementation Surveys

Treatmentteacherscompletedanonlineimplementationsurveytwotimesoverthecourseoftheacademicyearfollowingthesummerinstitute.SurveyscontainedamixofLikert-scaleitems,closedresponseitems,andopen-endeditems.Thepurposesofthesurveysweretoassess(a)teacherself-confidenceinimplementinginstructionalstrategies,(b)teacherconfidenceincontentmatteraddressedinthePD,(c)theextentofimplementationofthemajorprojectcomponentsandstrategies,(c)barrierstoimplementation,and(d)overallprogramfeedback.

Comparison Teacher Survey

Comparisonteacherscompletedanonlinesurveyinthespringofthe2014–2015academicyear.Thepurposeofthesurveywastounderstandhowcomparisonteacherswereimplementingunitsonmatterandenergyinclassroom.ThesurveycontainedamixofLikert-scaleitems,closed-responseitems,andopen-endeditems.Comparisonteachersrespondedtoquestionsaboutthenatureoftheirscienceinstruction,availabilityanduseofexemplaryresources,confidenceinteachingstandards-basedscience,confidenceinenergyandmattercontent,andbarrierstoeffectivescienceinstruction. Professional Development Observations


7 Eval

Evaluatorsobservedasampleofsummerworkshopdaysandfollow-upprofessionaldevelopmentsessions.Observationsfocusedonunderstandinghowcontentwasdelivered,opportunitiesforteacherparticipationandincorporationofeffectivePDstrategies. Formative Classroom Observations

Theexternalevaluatorandaprofessionaldevelopmentproviderobservedasampleofteachersimplementingmatterorenergylessonsintheirclassrooms.ObserversutilizedanobservationprotocoldevelopedfortheprojectalignedtothekeyelementsofeffectivepedagogyaddressedthroughthePDsessions. Instructional artifacts

ProjectPDstaffexaminedinstructionalartifactsfromparticipatingteachers,whichincludedlessonandcollaborativeunitplans.PDstaffevaluatedinstructionalartifactsonfivedimensions:(1)investigationofaBigIdeaoressentialquestionrelatedtomatterand/orenergy,(2)alignmentoflearninggoalstowardtheBigIdeaoressentialquestion,(3)alignmentofinvestigationstolearninggoals,(4)alignmentofformativeandsummativeassessmenttounitobjectives,and(5)overallunitassessment.Student Measures

Tounderstandimpactsofteachers’participationinE2MContheirstudents’understandingofphysicalscienceconcepts,evaluatorsutilizedcontentassessmentsinmatterandenergyalignedtotheprofessionaldevelopmentsessions.Understanding Science for Teaching Matter and Energy Student Assessments (Students)

EvaluatorsassessedapurposefulsampleofstudentsfromE2MCparticipantclassroomswiththeUSTMEAwithinoneweekpriortoandoneweekafterteachersimplementedunitsonmatterandenergy.Evaluatorschoseclassroomsofteacherswhowouldbeteachingwholeunitsoneithermatterorenergyasalignedtotheirgrade-levelstandards.StudentsinsevenclassroomstooktheUSTMEAstudentassessmentformatter,andstudentsin10classroomstooktheassessmentforenergy.

Data Preparation and Analysis

QuantitativedatawereenteredintoSPSS21forcleaningandanalyses.Toensureaccurateandcompletedata,evaluatorsuseddatacleaningandpreparationprotocols,whichincludedcalculatingandexaminingdescriptivestatistics,addressingmissingdata,andidentifyingoutliers.Foranalysisofcovariance(ANCOVA),evaluatorsscreeneddatatocheckforassumptionsassociatedwiththeanalysismethods.Theseincludedexaminingthelinearrelationshipbetweenthedependentvariableandcovariatesandhomogeneityofregressionslopes.Missingdatarateswere12%orlessforalladministrationsoftheteacherandstudentassessments(seeTableB-1,AppendixB).Ratesofmissingdatahigherthan40%compromisefindings,andtherefore,missingdataratesinthisstudyarewellwithintheacceptablerange(Puma,Olsen,Bell,&Price,2009).Evaluatorsaddressedmissingdatabyusingmultipleimputationstouseallavailabledataandmaximizethestudy’spower.For


8 Eval

statisticaltests,evaluatorsconsideredfindingsstatisticallysignificantusinganalphalevelof.05.However,theUSDOEconsidersMSPfindingssignificantatthep<.15levelandthereforewenotewhereresultsmeetthisbenchmark.

EvaluatorsimporteddatafrominterviewsandobservationsintoHyperRESEARCH,a

qualitativedataanalysissoftwarethataidsintheexaminationandcodingofemergentthemes,patterns,commonalities,anddifferences.Evaluatorsthenanalyzedqualitativedatausingthetechniquesofanalyticinduction(Erickson,1986).Onceevaluatorsdevelopedassertionsaboutthedata,theyconductedrepeatedsearchestoconfirmanddisconfirmassertionstoprovidethewarrantforthefindings.

Study Procedures

EvaluatorsconductedastudyorientationfortreatmentandcomparisonteacherspriortothestartofthestudyinAprilof2014.Thestudyorientationexplainedthepurposesofthestudy,data-collectionmeasuresandtimeframe,protectionofparticipantconfidentiality,theparticipantconsentprocess,andthereportingschedule.Allteachersprovideddemographicinformationandinformedconsentatthestartoftheirparticipation.

Evaluatorscompliedwithresearchreviewprocessesinparticipatingdistricts.InformedconsentwassoughtforstudentswhoparticipatedintheUSTMEAassessments.Studentswithoutparentalconsentwerenotincludedinthestudysample. Study Timeframe

Thestudybeganinthespringof2014uponawardbytheArizonaDepartmentofEducation.ObserversconductedinitialobservationsofparticipantandcomparisonteachersusingtheRTOPprotocolinAprilandMayof2014priortoE2MCsummerprofessionaldevelopmentinstitute.ParticipantandcomparisonteacherscompletedtheDTAMSpreassessmentinAprilof2014.TreatmentteachersalsocompletedthepreassessmentfortheUSTMEAinAprilof2014.E2MCparticipantsattendedatwo-weeksummerworkshopinJuneof2014.Follow-upPDsessionsoccurredthroughouttheacademicyear.ObserversconductedpostobservationsusingtheRTOPprotocolinMarchandAprilof2015.TreatmentandcomparisonteacherstooktheposttestDTAMSassessmentinAprilof2015,andtreatmentteacherstooktheUSTMEApostassessmentinAprilof2015.Table1presentsthestudytimelineanddatacollectionpoints.


9 Eval

Table 1. Timeline of E2MC Study Activities

STUDYACTIVITY

Apr

201

4

May

201

4

Jun20

14

July201

4

Aug

201

4

Sept

201

4

Oct201

4

Nov

201

4

Dec

201

4

Jan20

15

Feb20

15

Mar201

5

Apr

201

5

May

201

5

Studyorientation w

Summerinstitute

Professionaldevelopmentfollow-upsessions

w w w w

RTOPobservations w w w w

DTAMSassessment w w w

USTMEAteacherassessment

w w

USTMEAstudentassessment

w w w w

Instructionalobservations w w

Participantsurvey w w

Comparisonteachersurvey w

Endstudy w

Settings

ParticipantsinthestudyincludedteachersfromPeoriaUnifiedSchoolDistrictandGilbertPublicSchoolsinArizona.Bothdistrictsareinlargesuburbanareas.Districtsareofcomparablesizeanddemographics.Table2presentsdemographicsforthetwoparticipatingdistricts.


10

Table 2. District-level Demographics of Participating E2MC School Districts

GilbertPublicSchools

PeoriaUnifiedSchoolDistrict

Geographiclocationandcitydescription1

Suburb:Large

Suburb:Large

Totalnumberofschools2 45 41

Totalstudentenrollment 38,007 36,736

Student/Teacherratio 18.59 19.86EthnicbreakdownHispanic/LatinoNon-Hispanic/LatinoCaucasianAfricanAmericanAsian/PacificIslanderOther/twoormoreraces

14.80%85.20%83.79%2.40%2.84%10.97%

20.47%79.53%79.82%3.34%2.23%14.61%

Englishlanguagelearners 1.05% 2.27%

StudentswithIEPs 12.47% 13.56%Studentswithfree&reduced-pricelunchstatus3

26.90% 41.47%

*DataNotAvailable1.USCensusRegionsandDivisionsoftheUnitedStates;http://www.census.gov/geo/maps-data/maps/pdfs/reference/us_regdiv.pdf2.Schooldatabasedonthe2012–2013schoolyear:NationalCenterforEducationStatisticshttp://nces.ed.gov/


11

Participants Thissectiondescribesstudyparticipants,includingalltreatmentandcomparisonteachers

andstudentsinclassroomsassessedwiththeUSTMEA.Thesectionpresentsteacherattritionanalysesandstudentandteacherdemographics.Teacher Attrition Evaluatorsdeterminedattritionratesbycomparingthenumberofteachersparticipatingatthestartofthestudyandthenumberwhocontinuedthroughtheendofthestudy.Thestudybeganwith90totalteachers.Ofthese,fivecomparisonteacherswereremovedusingpropensityscorematchingtoensuregroupequivalenceandequalsamplesizesbetweentreatmentandcomparisonteachers.Theseteacherswerenotincludedintheattritionanalyses,resultinginastartingsampleof85teachers.Attheconclusionofthestudy,78teachersremained,yieldinganoverallattritionrateof8.23%.Evaluatorscalculateddifferentialattritionbydeterminingtheratesforbothtreatmentandcomparisongroupsandfindingthedifferencebetweenthetwo.Theattritionrateforthetreatmentgroupwas13.33%,andtheattritionrateforthecomparisongroupwas2.23%,yieldingadifferentialattritionrateof11.10%.Achi-squaretestdidnotrevealstatisticallysignificantdifferencesinattritionratesbystudycondition.Themajorityofattrition(71.43%)occurredinonedistrict.Oftheseventreatmentteacherscountedinattrition,fivewithdrewfromtheprogram,andtwodidnotmeetthe90%benchmarkforstudyparticipation,havingmissedfollow-upPDsessions.Analysis Sample

Tobeincludedinthefinalanalysissample,eachteacherneededtobeaparticipantintheE2MCMSPprojectthroughtheendofthestudyaswellashavemetthe90%benchmarkforstudyparticipation.Asnoted,ofthe85startingteachers,fivewithdrewfromthestudy,andtwodidnotmeettheparticipationbenchmark,resultinginafinalsampleof78teachers.Thesubsequentsectionfurtherdescribesthecharacteristicsoftheteachersincludedintheanalysissampleaswellasthecharacteristicsoftheparticipatingstudents.Teacher Participants Thestudysampleconsistedof78teachers(39treatmentand39comparison),with55.13%ofteachersrepresentingPeoriaUnifiedSchoolDistrictand44.87%representingGilbertPublicSchooldistrict.E2MCteachersrepresented25schoolsinthetwodistricts,including21elementary/middleschoolsandfourhighschools.Themajorityofteacherswerefemale(75.64%),and24.35%weremale.Themajorityoftreatmentteachersreportedteachingmiddleschool(76.92%)andtheremaining23.08%taughthighschool.Similarly,74.36%ofcomparisonteachersreportedteachingmiddleschool,and25.64%taughthighschool.Alargemajorityoftreatmentteachersteachscience(84.62%),followedbyintegratedscience(7.69%).Themajorityofcomparisonteachersalsoreportedteachingscience(82.05%).

Mosttreatmentteachersreportedholdingamaster’sdegree(66.67%),and33.33%reportedholdingabachelor’sdegree.Themajorityofcomparisonteachersalsoreportedholdingamaster’s


12

degree(56.41%),whiletheremaining43.59%reportedholdingabachelor’sdegree.TableC-1inAppendixCpresentsthedemographicinformationfortreatmentandcomparisonteachers.Group Equivalence

Evaluatorsusedchi-squareteststodeterminethegroupequivalenceregardingteacherdemographiccharacteristics.TableB-1inAppendixBpresentstheresultsofthechi-squareanalysesandindicatesthattreatmentandcomparisongroupswerecomparableongender,gradetaught,subjecttaught,andhighestdegreeobtained.

Acrossbothconditions,teachershadbeenteachingforanaverageof11.86years(rangeof1–31years).Treatmentteachersreportedteachingforanaverageof11.69years(range1–28years),andcomparisonteachersreportedteachingforanaverageof12.46years(range1–33years).Treatmentteachersreportedteachingattheircurrentgradeforanaverageof7.06years(range1–18)andattheircurrentschoolforanaverageof7.74years(range1–23).Comparisonteachersreportedteachingattheircurrentgradeforanaverageof7.82years(range1–26)andattheircurrentschoolforanaverageof6.33years(range1–20).

Evaluatorsconductedindependentsamples(t-tests)todetermineiftreatmentand

comparisonteachersdifferedsignificantlyontheiryearsofteachingexperience.Theseanalysesrevealednostatisticallysignificantdifferencebetweentreatmentteachers(M=11.69,SD=6.92)andcomparisonteachers(M=12.46,SD=8.31);t(76)=0.44,p=0.66.

Finally,evaluatorsconductedindependentsamplest-teststodetermineiftreatmentand

comparisonteachersdifferedsignificantlyonpretestsfortheRTOPandtheDTAMS.TherewasnosignificantdifferenceinscoresontheRTOPtotalscoreatpretestingfortreatmentteachers(M=19.13,SD=17.98)andcomparisonteachers(M=20.67,SD=11.89);t(76)=1.53,p=0.657).TherewasnosignificantdifferenceinscoresontheDTAMSTotalKnowledgeTypescoreatpretestingfortreatmentteachers(M=15.79,SD=4.52)andcomparisonteachers(M=15.28,SD=4.87);t(76)=0.482,p=0.631).TherewasnosignificantdifferenceinscoresontheDTAMStotalcontentscoreatpretestingfortreatmentteachers(M=14.33,SD=3.50)andcomparisonteachers(M=13.85,SD=3.69);t(76)=0.598,p=0.552).Thesefindingssuggestthattreatmentandcomparisonteacherswerecomparableintheirknowledgeandpracticeatthestartofthestudy.

Student Participants

Thestudentsampleincluded524students.Studentswereincludedintheanalysissampleiftheyhadeitherenergyormatterassessmentdata.Ofthestudentswhocompletedanassessment,51.34%werefromPeoriaschooldistrict,and48.66%werefromGilbertschooldistrict.Studentparticipantswerefrom17schoolsacrossthetwodistricts.Thelargestpercentagesofstudentswereinthesixthgrade(48.76%),followedbytheeighthgrade(33.08%).Moststudentswerefemale(51.75%),and48.25%weremale.Regardingethnicity,65.63%ofstudentswereclassifiedasCaucasian;2.93%asAfricanAmerican;23.24%asHispanic;and8.20%asAsian,AmericanIndianorAlaskanNative,PacificIslander,NativeAmerican,multiracial,orother.Ofthestudentsforwhomdemographicdatawereavailable,33.78%qualifiedforfreeandreduced-pricedlunch(FRL),and6.81%wereclassifiedasspecialeducation(SPED)students.Only2.54%ofparticipantswerecategorizedEnglishlanguagelearners(EL),and2.33%ofstudentswereclassifiedasSection504students.Studentswere


13

administeredeithertheMSSEnergyortheMSSMatterAssessment.TableD-1inAppendixDpresentsthedemographicinformationforstudentsassessedonenergyandmatterseparatelyaswellasthetotaltestedstudentpopulation.


14

Treatment and Comparison Teacher Science Instruction

Ontheimplementationsurveys,E2MCteachersandcomparisonteachersreportedontimespentteachingscience,timespentpreparingandplanningsciencelessons,resourcesusedinscienceinstruction,andstudentassessment.Teachersalsoreportedontheenergyandmatterconceptstheytaughtintheirscienceclassrooms.Science Instruction

Teachersreportedthenumberofminutesperdaytheyspentonatypicalsciencelesson

duringthepastschoolyear.Treatmentteachersreportedspendinganaverageof208minutesperdaywitharangeof0to450minutesperday.Comparisonteachersreportedanaverageof72minutesperdaywitharangeof0to300minutesperday.Whenaskedhowmanyminutesperweekwerespentonplanningandpreparingsciencelessons,treatmentteachersreportedspendingmoretimeonaverageplanningsciencelessonsthancomparisonteachers,withameanof236minutesperweek(range=0–600minutesperweek).Comparisonteachersreportedspendinganaverageof184minutesperweek(range=0–600).

Treatmentandcomparisonteachersreportedtheirfrequencyofuseofvariousresourcesfortheirsciencelessons.Themostfrequentlyusedresourcesforbothtreatmentandcomparisonteachersincludedlessonstheyhavecreatedandlessonscreatedbycolleaguesintheirschoolordistrict,followedbytextbooksandsupplementaltextmaterials.TablesE-1andE-2inAppendixEshowtreatmentandcomparisonteachers’frequencyofuseofresourcesforscienceinstruction.Teachersinbothgroupssupplementtheircurriculumwithadditionalmaterials,includingonlineresources,A–ZReading,andCommonCorematerials(n=21).Lesscommonresponsesincludevideopodcasts(n=3),adaptingmaterialsfromprofessionaldevelopment(n=3),ScienceStudiesWeekly(n=1),andScienceWeeklyScholastic(n=1).

Treatmentandcomparisonteachersreportedontheuseofformativeassessmentatthebeginningandduringanewunitofstudyaswellasthefrequencywithwhichtheycollaboratetoexaminestudentwork(seeTablesF-1andF-2,AppendixF).Overall,treatmentandcomparisonteachersreportedsimilaruseofformativeassessmentsatthebeginningofanewunitofstudy,with41.02%oftreatmentteachersreportingtheyusethemalwaysorfrequently,and44.11%ofcomparisonteachersreportingthesamefrequencyofuse.

Inregardstousingaformativeassessmentduringaunitofstudy,71.79%oftreatment

teachersreportedafrequencyofalwaysorfrequently,comparedto55.89%ofcomparisonteachers.Finally,23.08%oftreatmentteachersreportcollaboratingwithcolleaguestoexaminestudentworkalwaysorfrequently,comparedto23.52%ofcomparisonteachers.

Treatmentandcomparisonteacherssharedhowoftentheyprovideopportunitiesforstudentstouseandapplyscientificpracticestoproblemsorinvestigations.AsshowninFigure2,afterparticipatinginE2MC,treatmentteachersweremorelikelythancomparisonteacherstousethispracticeonaweeklybasis.


15

Figure 2. Frequency of opportunities for students to apply and use scientific practices.

Coverage of Matter and Energy in Instruction

Inordertounderstandwhichmatterandenergyconcepts/BigIdeasteachersincorporated

intotheirclassroominstruction,treatmentandcomparisonteachersselectedtopicsthattheycoveredintheirscienceclassesontheimplementationsurveys.Sciencestandardsvarybygradelevel,anditwasnotexpectedthatteachersshouldcoverallmatterand/orenergytopicsintheirscienceinstruction.Thesurveyquestionsregardingcontentcoveragewereintendedtogainanunderstandingofwhatteacherstaughtduringthestudyperiod.BigIdeasaretakendirectlyfromtheMakingSenseofSCIENCEMatterandEnergycourses.CoverageofmatterandenergyconceptsbyconditionarepresentedinAppendixG.Asshowninthetables,E2MCparticipantstendedtocovermorematterandenergyconceptsintheirinstruction.Energy and Matter Implementation Scores

Evaluatorsconvertedtopiccoverageformatterandenergyintoanimplementationscoreforeachbycalculatingthepercentageofcoverageoftopics.Thisscorewasobtainedbydividingthenumberofenergyormattertopicsbythetotalnumberofpossibletopicscoveredinthatsubjectarea.Evaluatorsusedtheimplementationscoreinanalysisofstudentachievementdatatounderstandwhethertherewasarelationshipbetweencontentcoverageandstudents’scoresontheUSTMEAassessment.Overall,meanimplementationscoreswereslightlyhigherfortreatmentteacherscomparedtocomparisonteachers.Table3providesdescriptivestatisticsontheseimplementationpercentagesfortreatmentandcomparisonteachers. Table 3. Implementation Score Descriptive Statistics for Treatment and Comparison Teachers

Treatment(n=37)

Comparison(n=35)

Range Mean SD Range Mean SD

Energy 5%–90% 51.76% 23.96 0–100% 40.57% 23.10

Matter 0–100% 48.50% 34.43 0–100% 37.65% 29.52

14.71%

21.05%

35.29%

42.11%

26.47%

28.95%

23.53%

7.89%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Comparison

Treatment

More than once per week Once per week 2-3 times per month Once per month


16

E2MC Teacher Program Perceptions

EvaluatorscollectedformativeandsummativefeedbacktounderstandhowwelltheE2MC

professionaldevelopment(PD)projectmettheneedsofparticipatingteachers.Teachersprovidedfeedbackonthesummerinstituteandonthreeofthefollow-upprofessionaldevelopmentsessionsthroughsurveys.FeedbackattheendofeachsessionallowedPDproviderstomonitorandadjustinstructionatsubsequentsessions.Additionally,teachersprovidedfeedbackontheoverallE2MCprojectonthemid-projectandend-of-projectimplementationsurveysandthroughinterviewsduringPDsessionsandafterobservations.

Professional Development Feedback

AttheendofeachPDsession,teachersratedtheformat,organization,pace,materials,

delivery,andoverallsessionona5-pointscalerangingfrom1=poorto5=excellent.AppendixHpresentsmeanratingsforthesecharacteristicsbysession.AsshowninTableG-1,teachersratedallaspectsoftheE2MCPDsessionshighly.Overallsessionratingsrangedfromameanof4.19to4.61,indicatingahighlevelofsatisfactionwiththeofferings. Acrossallsessions,teachersfeltthatthePDalignedwellwiththeirinstructionalgoalsandwiththestandardstheywererequiredtoteachattheirgradelevel.Teachersparticularlyvaluedtheopportunitiestocollaboratenotonlywithgrade-levelpeersbutalsowithteachersacrossgradelevels.Asoneparticipantnoted,“Cross-grade-levelcollaborationandwithin-grade-levelcollaborationwasahighlight[ofthePD]becauseoftheopportunitytosharethelearning.”

Onthefinalimplementationsurvey,teacherssharedtheaspectsoftheE2MCprogramthatweremostusefulandvaluableintheirinstruction.Themostcommonresponseregardedincreased

Figure 3. Teachers participate in a sensemaking activity during a PD session.

“Misconceptionswereclearedupatmanylevels,whichisextremelyimportantingoingforwardwithpreciseteaching.” –E2MCsummerworkshopparticipant

KeyQuestion:Howdoteachersperceivethequality,utility,andeffectivenessoftheprofessionaldevelopmentinbuildingtheircapacityforeffectivescienceinstructionofinterconnectedcoreideasandcrosscuttingconceptsrelatingtomatterandenergy?


17

understandingofenergyandmatterandfeelingmoreconfidentteachingthesetopics.Asoneparticipantcommented,

Thishasbeenagreatwaytoenhancemyteachingofscienceconcepts,toprovidemewithveryspecifictoolsforteachingtomisunderstandings,andtohelpmystudentsmakesenseofscienceconcepts.Thecontentandactivitiesallowedmetoseemyownmisunderstandingsandtogaindepthandbreadthinmythinking,therebyshiftingmyteachingpracticestodothesame.

Anotherstated,“Thecourseshavedeepenedandstrengthenedmyunderstandingsonthesetopics,andinturnIamamoreeffectiveteacher.”

Teachersalsonotedthatparticipatinginboththeenergyandmattercoursesenhancedtheirunderstandingofhowmatterandenergyareconnected.Ateachernoted,“Havingthecontentknowledgeisamazing—andIknowIammakingtheconnectionbetweenmatterandenergymuchmorethaninthepast.”Onthefinalimplementationsurvey,manyteachersexpressedthattheyvaluedlearningmodeling

andimplementingthisininstruction.Oneteacherexpressedthissentimentwhenhe/shewrote,“Ilovedtheemphasisonmodeling,andithasreallyhelpedwithinstructionandfiguringoutwhatkidsknowanddon'tknow”.Othercommonresponsesincludedlearningnewteachingmethodsforthesetopics(n=6),theScienceFormativeAssessmentbook(n=6);hands-onexperiments(n=4);

collaboratingwithcolleagues(n=4);theliteracycomponentsoftheprogram(n=2);questioningtechniques(n=2);andunderstandingclaims,evidence,andreasoning(n=2).Withrespecttomakingconnectionstoliteracy,atypicalteachercommentwas,“Linkingliteracyconceptswithscience[wasvaluable]becauseIbelievetheseskillsareweakinsixthgraders,andIcameawaywithgreatideasandstrategies.”

Finally,surveyparticipantswereprovidedtheopportunitytoshareanyotherfeedbackaboutthe

E2MCprogram.Overall,teachersfoundparticipationtobeveryvaluableandagreatlearningexperience.Oneteachersummedupthissentimentbysaying,

Thisisabrilliantprogramthatgivesteacherstheopportunitytoexpandtheirknowledge,notonlyinsciencebutalsoinplanning,presenting,modelingandteaching.Ibelievethatthisclassisvitalfordevelopingconfidentandcapableelementaryschoolteachersthatwillbeexcited,ratherthanfearful,aboutteachingscience.

Perceptions of Impacts on Instruction

TeachersexplainedtheimpactsthatparticipationintheE2MCprogramhashadontheirscienceinstruction.Themostcommonresponsewasthattheprogramincreasedcontentknowledgeofenergyandmatter(n=15).Manyteachersreportedthatparticipatinginthisprogramhaschanged

“Iamleavingwithamuchdeeperunderstandingofthehierarchyofscientificcontentknowledge,whichisrelevanttoallofthecurriculumIamteaching.” –E2MCsummerworkshopparticipant


18

theirapproachtoteachingbyaddingmoreinquiry-basedlessonsandactivities(n=9).Additionalanswersincludeincreasedconfidenceinteachingmatterandenergytopics(n=6),incorporatingmodelingintoinstruction(n=6),moreuseofformativeassessments(n=2),andincorporatingmorehands-onactivitiesintheclassroom(n=2).Withrespecttomodeling,oneteachercommented,“Ihavefoundnewwaystofocusonmodels,theirbenefits,andlimitations…Iammorecomfortableteachingmystudentstothinklikescientists.”

Additionally,teacherswerealsoaskedwhatimpactstheirparticipationhadonstudentlearningandtoprovideevidencetosupporttheiranswers.Overall,themostcommonresponsewasthatstudentshaveanincreasedunderstandingofenergyandmattercontentasevidencedbytheirassignmentsandhigherassessmentscores(n=18).Asoneteacherstated,

Studentshaveadeeperunderstandingofconceptsinvolvingenergy,andtheyhaveahigherlevelofconcerninregardstotakingownershipoftheirwork.Thequalityhasimprovedandtheirexcitementhasgoneup.Studenttestscoresfromthisyeararehigherthanlastyearinthesamecontentareas.

Otheranswersincludedthatstudentinterestincreasedasteacherconfidenceincreasedin

teachingthesetopicsandthatstudentshavetheabilitytocreatemodelstoshowtheirunderstanding.Twoindividualsindicatedthatthattheyhavenotobservedimpactsontheirstudentsasofyet.

Onthefinalimplementationsurvey,teacherswereaskedtosharehowtheyplantobuildonwhattheyhavelearnedanddevelopedthroughtheE2MCprogramforthenextschoolyear.Themostcommonresponsetothisitemisthatteachersplantousetheformativeassessmentswithstudents(n=10),followedbyplanningtocollaboratewithcolleaguestoincorporateenergyandmatterintothecurriculum(n=6).Otherresponseswereasfollows:

• integratingenergyandmatterintootherscienceunits(n=4)• usingtheExperimentswithPlansUnit(n=3)• incorporatingliteracystrategiesintoteaching(n=3)• usingmorehands-onlabsintheclassroom(n=2)• continuingtouseClaims,Evidence,andReasoning(CER)withstudents(n=2)

Barriers to Implementing Learning from E2MC

Onthemid-projectandend-of-projectimplementationsurveys,participatingteacherssharedthebarrierstheyfacetousingwhattheyhavelearnedintheE2MCprogramfortheirscienceinstruction.Themostcommonlyreportedbarrierisalackoftimetobothimplementthisnewinformationintothecurriculum,andteachittostudents.Asoneteacherstated,“Timeisthebiggestbarrier.Tomakechangestocurrentlessonsandcurriculumtimeisneeded.Weneedtimetospendlookingatalltheresourceandtoprocesstheinformation.”Thesecondmostcommonlyreportedbarrieristhattheenergyandmattercurriculuminformationisnotalignedwithstatestandards(n=9).Oneteacherexpressedthissentimentbysaying,“Ithinkourcurriculumandstatestandardsmakeithard.WeknowNGSSarebetter[thanstatestandards],andthedepthofknowledgetaughtisbetter,butthatdoesn'tfitourstateanddistrictatthistime.”Otherreportedbarriersincludethefollowing:

• aneedformaterialsandresource(n=7)


19

• difficultyunderstandingthecontent(n=3)• wantingmorelabstouseintheclassroom(n=3)• feelingoverwhelmedwiththeamountofnewinformation(n=3)

Lesscommonresponsesincludedaneedforadvocatestopromotesciencelessonsinelementaryschool(n=1),andsciencekitsatschoolnotaligningwiththeE2MCprogram(n=1).Summary of Findings Related to Program Perceptions TeacherswhoparticipatedintheE2MCprofessionaldevelopmentprogramfeltthattheprogramwasofvaluetotheirunderstandingofphysicalscienceconceptsrelatedtomatterandenergy.Teachersvaluedmanyaspectsoftheexperience,particularlyworkingcollaborativelyacrossmiddleandhighschoolgradelevelswithpeerstoplananddesignlessons.Theycameawaywithabetterunderstandingofhowmatterandenergyareconnectedandhowconceptsrelatetootherareasofinstruction.TeachersfelttheMakingSenseofSCIENCEcoursesprovidedvaluableresourcesfortheirinstruction.Theyappreciatedtheconnectionsofliteracytosciencecontent.Theyalsogreatlyvaluedtheopportunitytopracticetheirlearningthroughhands-oninvestigations.Impactsoninstructionincludedincreasedconfidenceinteachingmatterandenergyandenhancedabilitytohelpstudentscreatementalmodelstoexplaintheirthinking,amongothers.TeachersindicatedthatlackoftimeforplanningandreflectingonlessondesignandimplementationisthebiggestbarriertoimplementinglearningfromthePD.


20

Teachers’ Perceptions of Abilities to Implement Effective Strategies and Content

Tounderstandhowparticipatingteacherscomparedtononparticipatingteachersintheir

perceptionsoftheirabilitiestoimplementbestpracticesinscienceinstructionattheendofthestudy,evaluatorsaskedaseriesofquestionsonthefinalimplementationsurveys.ThestrategiesarethosetargetedthroughtheMSScoursesandinprofessionaldevelopmentsessions.

Teacher Confidence in Use of Instructional Strategies

Ontheimplementationsurveysfortreatmentandcomparisonteachers,evaluatorsaskedteacherstoratetheirconfidenceinincorporatingvariousinstructionalpracticesintotheirscienceinstruction(1=notatallconfident;3=somewhatconfident,5=extremelyconfident).Overall,treatmentteachersreportedhigherlevelsofconfidenceusingvariouspracticesthancomparisonteachers.Ofnote,comparisonteachershadhighermeanratingsofconfidenceintheirabilitytoengagelearnersinscientificallyorientedquestionsandguidingstudentstoconsideralternativeexplanationsthantreatmentteachers(Figure4).

KeyQuestion:AsaresultofE2MCprofessionaldevelopment,howwellprepareddoteachersfeeltoimplementtargetedcontentknowledgeandpedagogicalpracticesintheirscienceclassrooms?


21

Figure 4. Treatment and comparison teacher mean ratings of confidence in instructional practices.

Teachersratedtheirconfidenceinincorporatingtheuseofmentalmodelsintotheirinstruction,astargetedintheprofessionaldevelopmentsessions(1=notatallconfident;3=somewhatconfident,5=extremelyconfident).Treatmentteachersreportedhighermeanratingsofconfidenceinvariouspracticesrelatedtotheuseofmentalmodelsascomparedtothecomparisonteachers(Figure5).

Figure 5. Treatment and comparison teacher ratings of confidence in using mental models in instruction.

3.29

3.83

3.89

3.46

3.63

4.00

3.69

3.77

3.38

3.68

3.76

3.44

3.56

3.94

3.97

3.74

1 2 3 4 5

Guiding students to consider alternative explanations to observed phenomena

Providing opportunities for students to communicate/ justify their proposed

explanations

Providing opportunities for students to present information in scientifically accepted

formats

Helping students to identify faulty reasoning, or reasoning not based on evidence

Guiding students to develop their own means of investigating scientific questions

Guiding students to use evidence in responding to questions and developing

explanations

Engaging learners in scientifically oriented questions

Helping students understand the types of questions that can be investigated through

scientific inquiry

Comparison Treatment

3.43

3.17

3.40

3.60

3.38

3.06

2.88

2.94

1 2 3 4 5

Assessing student models

Finding activities/investigations that will build student understanding to modify or explain something in the model

Identifying observable phenomena that students can explain through a mental model

Helping students create mental models to explain their thinking



22

Ability to Make Connections Between Matter and Energy in the Science Classroom Formanyoftheparticipatingteachers,participatinginMSSMatterandEnergycoursesallowedthemtobetterunderstandtheconnectionsbetweenmatterandenergycontent.PDsessionsspecificallyprovidedsupportforunderstandinghowmatterandenergywererelated.Forexample,whendiscussingchangesinstatesofmatter,energywasidentifiedasthedrivingforceforthosephasechanges.Thus,connectionsweremadeexplicit.Tounderstandwhetherparticipatingteachersfeltmoreconfidentinmakingconnectionsbetweenmatterandenergyintheirinstructionascomparedtononparticipants,evaluatorsaskedthemtoratetheirlevelofconfidenceontheimplementationsurvey(1=notatallconfident;3=somewhatconfident,5=extremelyconfident).Overall,treatmentteachersreportedhighermeanconfidenceratingsinmakingconnectionsbetweenmatterandenergy(Figure6).

Figure 6. Treatment and comparison teacher ratings of confidence in making connections between matter and energy in instruction. Summary of Findings Related to Perceived Ability to Implement Learning

TeacherswhoparticipatedintheE2MCprofessionaldevelopmentprogramindicatedhigherconfidenceintheirabilitiestousementalmodelsintheirscienceinstructionthandidnonparticipants.E2MCteachersalsofeltmoreconfidentintheirabilitytomakeconnectionsbetweenmatterandenergyintheirinstruction.Participantsfeltmostconfidentwithrespecttoguidingstudentstouseevidenceindevelopingexplanations,providingopportunitiestopresentfindingsinascientificformat,supportingstudentstocommunicateandjustifytheirexplanationsofphenomena,andhelpingstudentstounderstandwhichtypesofquestionscanbeansweredthroughscientificinvestigations.Participantsratedthemselveshigherthannonparticipantsonmanyoftheinstructionalstrategiestargetedintheprofessionaldevelopmentsessions.

3.57

3.15

1 2 3 4 5

How conf ident a re you in mak ing connections between matter and energy in your instruction?



23

E2MC Teacher Application of Learning to Instruction

TounderstandhowteachersimplementedlearningfromtheE2MCintheirinstruction,evaluatorsandprofessionaldevelopment(PD)staffconductedformativeobservationsofasampleofteacherswhowereteachingunitsonmatterorenergyduringthecourseofthestudy.ObservationswereofteacherswhowerealsoassessingtheirstudentswiththeUSTMEAassessments.Tounderstandhowteacherswouldapplytheirlearningtofutureinstruction,PDstaffsupportedteachersindevelopingunitplansformatterandenergythatwerealignedtograde-levelstandards.ThissectionpresentsfindingsrelatedtoapplicationofE2MClearningtoinstruction.Formative Lesson Observations Evaluatorsandprofessionaldevelopmentprovidersobserved11E2MCteachersusingarubricthatscoredinstructionon10dimensions.ObserversassignedratingsonaLikertScalerangingfrom0=noevidence,to5=highlydescriptive.ThepurposeoftheobservationswastoprovideformativefeedbacktoteachersimplementingmatterandenergylessonsandtoinformareasinneedofmoresupportforsubsequentPDsessions.Thefollowingarethe10dimensionsoftherubric:

• Studentgrouping(opportunitiesforcollaborativelearning)• lessonstructure(activitiesarecoherentandlogicallystructured)• useofresources(varietyofresourcesused)• hands-on/minds-on(opportunitiestophysicallyandmentallyengageincontent)• inquiry(opportunitiestoengageinscientificthinkingandprocesses)• cognitivedepth(focusonBigIdeasfromthediscipline)• scientificdiscourse(“talkingscience”)• explanation/justification(extenttowhichteacherexpectsstudentstojustifytheiranswers)• assessment(formalandinformalassessmentofstudentlearning)• connections/applications(connectingsciencetoreal-worldexperiences).

Acrossobservations,teachersscoredhighestforuseofvariedresources,collaborativestudentlearningopportunities,andlessonstructure(Figure7).

KeyQuestion:HowdoesparticipationintheE2MCprojectimpactteacherpracticewithrespecttoeffectivelessonplanningandimplementation?


24

Figure 7. Ratings of formative lesson observations of E2MC participants (n=11).

Observersnotedthatvariedandeffectiveuseofresourcesseemedtocontributetogreater

studentengagement.Teacherswereobservedusingshortreadingsandexplanatoryvideosalongwithinvestigationsrequiringstudentstousetechnologyandsciencepractices.Manyclassroomsconsistentlyutilizedsciencenotebooks.Theteacherswhohadmoreeffectivelessonsincorporatedthefollowing:coreconceptsfromtheMSSEnergyandMattercourses,coreideasfromAFrameworkofK–12ScienceEducation,orthedisciplinarycoreideaprogressionsinAppendixEoftheNextGenerationScienceStandards.ThosethatreliedmoreheavilyonfactsandperformanceobjectivesfromtheArizonaScienceStandardshadlessonsthatwerenotasclearlydesignedtodevelopconceptualunderstanding.Thoseteachersscoringa3or4onthiscomponenthadmodifiedlessonsfromtheenergyandmattercoursestousewiththeirstudents.Theyfollowedasmallgroupinvestigationwithwholegroupsharingofdataandsensemakingandhadfewerincidentsofmisconceptionsorconfusionfromstudents.

Observersnotedthatseveralteacherswerestrugglingtodesignaseriesoflessonsaround

crosscuttingconceptsorBigIdeas.Postobservationinterviewsprovidedevidencethatteacherswerethinkingaboutthebiggerpicture.Theywereabletoexplainhowpreviouslessonstaughtpreparedstudentsforwhattheyweredoingthatdayandhowfuturelessonswillconnecttoandbuildonthislearning.Manyteacherswereabletoarticulatewhattheyexpectedstudentstoknowandbeabletodoattheendofaunittodemonstrateunderstanding.Teachersappearedtobe

1.36

1.45

1.82

2.00

2.09

2.09

2.18

2.73

2.91

3.00

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

Hands-On/Minds-On

Inquiry

Real-World Connections

Cognitive Depth

Assessment

Scientific Discourse

Justification/Evidence

Lesson Structure

Collaborative Grouping

Use of Resources

Mean Ratings of Formative Observations


25

makingstrongerconnectionsbetweenmatterandenergybyincludingenergyconceptsinmatterunits.

Teachersscoredlowestoninquiry-basedlessonsandopportunitiesforhands-onlearning.It

isimportanttonotethatobservationsrepresentedasnapshotofinstructionwithinalargerunit.Itwasapparentduringteacherinterviewsthatsomeobservedlessonsservedasformativeorsummativeassessmentsofprevioushands-onlessons.Intermsofinquiry-basedapproachestoinstruction,fewteachersgavestudentsmorecontrolofthelearningexperience,andonlyfourprovidedopportunitiestoanswerscientificquestions.

Asaresultofformativeobservations,professionaldevelopmentprovidersadjusted

subsequentPDsessionstocontinuetoemphasizeandsupportinstructionaroundcoreideas.PDsessionsalsoemphasizedopportunitiesduringinstructionforstudentstomakeconnectionsbetweenwhattheyarelearningonedaytothenext.PDproviderssharedthatteacherswhowereusingand/ormodifyinginvestigationsfromtheMSScourseswereseeingincreasedstudentunderstandingoftargetedconcepts.PDprovidersprovidedadditionalsupportintheseareasasteachersworkedcollaborativelyonunitplans.Integration of Matter and Energy Asnotedinthereportsectiononprofessionaldevelopment,akeylearningforparticipatingE2MCteacherswasdrawingconnectionsbetweenenergyandmatterconceptsininstruction.Onthefinalimplementationsurvey,teachersexplainedhowtheymakeconnectionsbetweenmatterandenergyintheirinstruction.

Severalteachersindicatedthattheymaketheconnectionswithrespecttophasechangesofmatter.Atypicalresponsewas,“InthematterunitIwasabletotalkmoreabouthowphasechangesoccurbecauseofthechangesinenergy.StudentsshowedthisimprovementthroughtheirPhaseChangeLabanalysesandconclusions.”Otherstalkedaboutmakingconnectionsinunitsonearthsystems.Oneteachercommented,“[Imadeconnectionsby]discussingtheconceptofenergybeforetheearthsystemsunit.Sowhentheclasslearnedaboutearthphenomena(earthquakes,volcanoes,etc.),theycoulddiscussdifferenttypesofenergyandtheirtransfers.”Otherteachersincorporatedconnectionsbetweenmatterandenergyintolessonsonfoodwebs.Onestated,“I'vemostlymadetheseconnectionsthroughteachingphysicslessonsandfoodchainsandwebs.NotonlyhaveIdiscussedtheseconnectionsinlessons,though,butIhaveusedtheinformationI'velearnedtoanswerstudentquestionsaswell.”Ateamofmiddleschoolteacherssharedthattheyarechangingtheorderoftheirunitsduetotheunderstandingoftheconnectionsbetweenmatterandenergy.Onecommented,

Ihavetotallychangedthetimingandmethodofinstruction.Insteadofan“energy”unit(thatwasactuallyrenewableandnonrenewablefuels),Iwillbeteachingwhatmatterandenergyareandhowtheyrelatetoeverythingintheuniverseasmyveryfirstunit,

“EverythingonEarthiscomprisedofmatter,andeverysystemonEarthrequiresorcontainsenergy.Matterandenergyaredirectlylinkedandcontinuallyaffectoneanother.” –E2MCparticipant


26

andthenaswedowatercyclesandweather,wewillrelatebacktothingslikeconvectioncurrentsintheatmosphere,intheocean,andasheatenergy.

Unit Plan Development

E2MCteachersworkedcollaborativelytodevelopunitplansonenergyandmatterforuseintheirscienceclassrooms.Teachersworkedingrade-levelteamswithintheirdistrictstoidentifyBigIdeasandessentialquestionsandtoaligntheirlearninggoalswiththoseBigIdeas.Teachersidentifiedformativeandsummativeassessmentsandscientificinvestigationstoguidestudentlearningalignedtothelearninggoals.Professionaldevelopmentprovidersdevelopeda10-pointrubrictoanalyzeunitplansalignedtotheabovecriteria(Table4).Table 4. Rubric for Analyzing Teachers' Unit Plans

Dimension ScoringcriteriaTotal

possibleDesignaroundBigIdea,oressentialquestion

Yes=1point No=0point 1

AlignmentoflearninggoalstoBigIdea

Noalignment=0

Inadequatealignment=1

Goodalignment=2

Strongalignment=3

3

Alignmentofinvestigationswithlearninggoals

Noalignment=0


Goodalignment=2

Strongalignment=3

3

Alignmentofformativeandsummativeassessmentswithlearninggoals

Noalignment=0


Goodalignment=2

Strongalignment=3

3

Overallscore 10 Scoresonunitplansvariedacrossthetwodistricts.OfthefourPUSDplanningteams,overallunitplanscoresrangedfrom7to10,indicatinghighalignmentofinvestigationsandassessmentstotheunitBigIdeas.ThefiveGilbertteamsscoredloweroverallontheirunitplanswithscoresrangingfrom2to4,indicatinganeedformoresupportindevelopingBigIdeasoressentialquestionsandthenaligninginstructionaldesignwiththoseideas.

Summary of Findings Related to Application to Instruction FormativeobservationsshowedthatE2MCteachersuseavarietyofresourcesandcollaborativelearningstrategiesintheirinstruction.Observedlessonswerestructuredcoherentlyandlogically.Teachersappearedtoneedmoresupportforactivelearningstrategiesand

KEYFINDINGS:

E2MCteachersaremakingstrongerconnectionsbetweenmatterandenergyintheirinstruction.

Unitplansshowthatsometeacherteamsaremoreproficientatdesigninginstructionthatisalignedtotheirlearninggoals.


27

opportunitiestoanswerscientificquestions,whichwereemphasizedinthePDsessions..Teachersaremakingstrongerconnectionsbetweenmatterandenergyintheirinstruction.Unitplansshowedvariedproficienciesataligninginstructiontolearninggoals.


28

Teacher Performance Results for Instructional Practice

EvaluatorsexaminedchangesinpedagogyusingtheReformedTeachingObservation

Protocol(RTOP)fortreatmentandcomparisonteachers.ThefollowingsectionsofthereportpresentfindingsrelatedtotheimpactsofparticipatingintheE2MCprojectoninstruction.

TounderstandhowparticipationintheE2MCprojectaffectedinstructionalpractice,

evaluatorsexaminedscoresontheRTOPobservationprotocol.RTOPassessesthedegreetowhichteachersuseinstructionalstrategiesthatencouragecollaborativelearning,areinquirybased,andprovideopportunitiesforstudentstoexplainandjustifytheirwork(Piburn,2000).TheRTOPcontains25itemsinfivecategories:(1)LessonDesignandImplementation,(2)Content:PropositionalKnowledge;(3)Content:ProceduralKnowledge;(4)ClassroomCulture:CommunicativeInteractions;and(5)ClassroomCulture:Student/TeacherRelationships.Totalscorescanrangefrom0to100points. Descriptive Analyses Regarding Treatment Teachers’ Instructional Practice

Evaluatorsexaminedteachers’pretestandposttestchangesinpedagogy(asevidencedbyRTOPscores)todeterminewhetherornotteacherswhoparticipatedintheE2MCprojectdemonstratedchangesintheirinstructionalpracticestobemorealignedwith“reformed”teaching. Evaluatorsassessedparticipatingteachers’instructionalpracticeusingtheRTOPatpretestandposttest.Figure8presentstreatmentteachers’pretestandposttestRTOPscores.Unadjustedmeanssuggestthatonaverage,teachersparticipatingintheE2MCprojectdemonstratedincreasedcapacityforreformedteachingstrategies.

Figure 8. Pretest and posttest RTOP scores for teachers participating in the E2MC project (n=39).

19.13

28.23

10

15

20

25

30

Pretest Posttest

Teacher RTOP Scores Pretest to Posttest

KeyQuestion:Doesparticipationintheprojecthaveastatisticallysignificantimpactonteachers’abilitytoimplementeffectiveinstructionalstrategiesandpedagogyinthescienceclassroom?


29

Paired Samples t-Test Analyses Examining Treatment Teachers’ Changes in Instructional Practice Evaluatorsconductedapairedsamplest-testtoexaminewhetherteacherswhoparticipatedintheE2MCprojectmadesignificantgainsintheirinstructionalpracticeasmeasuredbytheRTOP.Resultsindicatethatparticipatingteachersmadesignificantgainsfrompre-toposttestingininstructionalpractice(Table5).Todeterminethemagnitudeofthegainscores,evaluatorscalculatedstandardizedeffectsizesbydividingthepretest-to-posttestdifferencebythepreteststandarddeviation.Effectsizesgreaterthan+/–0.25wereconsideredsubstantivelyimportant2.EffectsizeforgainsinRTOPscoresindicatesalargeeffect. Table 5. Results of a Paired Samples t-test of RTOP Scores for Participating Teachers Outcomevariable(n=39)

Pretestmean

PretestSD

Posttestmean

PosttestSD

Meandiff.

t-Value(df=38) p-value Effect

size

RTOPScore 19.13 17.97 28.23 19.14 9.10 2.43 0.02* 0.51* Statistically significant at the p <.05 level

Comparison of Teacher RTOP Scores by Condition

Evaluatorsconducteddescriptiveanalysesandinferentialanalysestocomparethedegreetowhichparticipatingandnonparticpatingteachersusedeffectivesciencepedagogicalstrategies.Evaluatorscomparedtreatmentandcomparisonteachers’scoresontheposttestRTOPobservationprotocol. Descriptive Analysis Comparing RTOP Scores by Condition

Figure9displaystreatmentand

comparisonteachers’RTOPpretestandunadjustedposttestscores.ThesedescriptivefindingssuggestthatRTOPscoresincreasedfortreatmentteachersandslightlydecreasedforcomparisonteachers.

Analysis of Covariance Comparing 2 What Works Clearing House guidelines; http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Figure 9. Pretest and posttest mean RTOP scores for treatment and comparison teachers.

19.13 28.23

20.67 19.62

0

20

40

Pretest Posttest

RTOP Scores Treatment Comparison


30

E2MC and Comparison Teachers Posttest Scores on the RTOP

TounderstandwhetherparticipatingteachersintheE2MCPDprojectmadegreatergainsininstructionalpracticethannonparticipatingteachers,evaluatorsconductedaone-waybetween-groupsanalysisofcovariance.Theindependentvariablewasthestudycondition(treatmentorcomparison),andthedependentvariablewastheposttestRTOPscore.Evaluatorsusedparticipants’scoresonthepretestRTOPobservationasacovariateinthisanalysis.Previousresearchshowsthatpretestcovariatestendtoexplainthemostvarianceintheoutcomeofinterestandthataddingcovariatesbeyondthepretestdoesnottypicallyexplainmuchadditionalvarianceintheoutcome(Bloom,Richburg-Hayes,&Black,2007;Hedges&Hedberg,2007). Resultsindicatedthatafteradjustingforpreinterventionscores,thereweresignificantdifferencesbetweentreatmentandcomparisonteachers,withtreatmentteachersscoringhigherontheposttestthancomparisonteachers,F(1,75)=5.60,p=0.021.Effectsizewassmallasindicatedbyapartialetasquaredvalueof0.07.Table6presentspre-andposttestunadjustedmeansandstandarddeviationsforthetwogroupsandtheadjustedpostttestmeans.

Table 6. Treatment and Comparison Pretest and Posttest Mean Scores on the RTOP

Condition

Pretestunadj.RTOPmean

PretestSDPosttestRTOP

unadj.mean

PosttestSD

Estimatedmarginalposttestmean

Std.error

T(n=39) 19.13 17.97 28.23 19.14 19.45 2.67C(n=39) 20.67 11.89 19.62 14.24 28.39 2.67

KEYFINDINGS:

Onaverage,E2MCteachersgained9.10pointsfrompretesttoposttestontheRTOP.E2MCteachersscoredhigherontheposttestthancomparisonteacherswhenadjustingforpretestscores.


31

Teacher Performance Results for Physical Science Content Knowledge

TounderstandhowparticipationE2MCprojectaffectedteachers’understandingofphysical

scienceconceptsrelatedtomatterandenergy,evaluatorsusedmultiplemeasures.TheseincludedtheDiagnosticTeacherAssessmentsinMathematicsandScience(DTAMS),specificallythetestforphysicalscience,andtheUnderstandingScienceforTeachingMatterandEnergyAssessmentsalignedtotheprofessionaldevelopmentcoursesinmatterandenergy.

Descriptive Findings and Inferential Analyses Regarding Teacher DTAMS Gains Among E²MC Program Participants

Toaddresskeyevaluationquestionsregardingteacherlearningformatterandenergy

concepts,evaluatorsfirstexaminedscoresontheDTAMStestforphysicalscience.TheDTAMSphysicalsciencetestprovidesscoresforthefollowingsubjectknowledge:propertiesandchangesofpropertiesinmatter,transferofenergy,andmotionandforcesaswellasaTotalContentscorecombiningallthreecontentstrands.Theassessmentalsoprovidesscoresforspecificknowledgetypes:declarativeknowledge(definitionsandfacts),scientificinquiryandprocedures,schematicknowledge(deepunderstandingofscienceconcepts,laws,andtheories),andpedagogicalcontentknowledgeaswellastotalknowledgetypescore.EvaluatorsexaminedtheTotalKnowledgeTypescoreandTotalContentscoresaswellassupplementaryanalysesofscoresformatterandenergy(thefocusofprofessionaldevelopmentefforts)andforspecificknowledgetypes.Evaluatorsconductedavarietyofanalysesincludingdescriptiveandinferentialstatisticsaswellasthecalculationofeffectsizesforprogramparticipants.

Descriptive Examination of E²MC Teachers’ DTAMS Total Content Scores and Total Knowledge Type Scores

ToevaluateE²MCteachergains,evaluatorscalculatedchangesinscoresfrompretesttoposttestontheDTAMSTotalContentscoresandTotalKnowledgeTypescores(seeFigures10and11).ForE²MCteachersinthisstudy,descriptivestatisticssuggestthatDTAMSTotalContentscoreandTotalKnowledgeTypescoreincreasedfrompretesttoposttest.Figure10presentsthepretest-to-posttestDTAMSTotalContentscoresforE²MCparticipantsandFigure11presentsthepretest-to-posttestTotalKnowledgeTypescores.

KeyQuestion:DoesparticipationintheE2MCprojecthaveastatisticallysignificantimpactonteacherunderstandingofinterconnectedcoreideasandcrosscuttingconceptsrelatingtomatterandenergy?


32

Paired Samples t-tests Examining E²MC Teachers’ DTAMS Total Content Scores and Total Knowledge Type Scores

Evaluatorsconductedapairedsamplest-testtoexaminewhetherteacherswhoparticipatedintheE2MCprojectmadesignificantgainsintheirknowledgeasmeasuredbytheDTAMS.ResultsshowthatonaverageTotalContentscoreswhichincludematter,energyandforcesandmotionssubtests,increasedfrompretesttoposttestby1.75points,andTotalKnowledgeTypescoresincreasedfrompretesttoposttestby3.29points.Resultsofpairedsamplest-testsindicatethatgainswerestatisticallysignificant.

Todeterminethemagnitudeofthegainscoresevaluatorscalculatedstandardizedeffect

sizesbydividingthepretest-to-posttestdifferencebythepreteststandarddeviation.Effectsizesgreaterthan+/–0.25wereconsideredsubstantivelyimportant.3Thesegainswerestatisticallysignificantandcorrespondedtolargeeffectssizes(d=0.50forTotalContentscoresandd=0.73forTotalKnowledgeTypescores)(Table7). Table 7. Results of a Paired Samples t-test of DTAMS Total Knowledge Type Scores for Participating Teachers OutcomeVariable

(n=39)PretestMean

PretestSD

PosttestMean

PosttestSD

MeanDiff.

t-value(df=38)

p-Value

EffectSize

TotalContentScore

14.33 3.50 16.08 3.76 1.75 3.74 0.001* 0.50

TotalKnowledgeTypeScore

15.79 4.53 19.08 5.60 3.29 5.64 <.001* 0.73

* Statistically significant at the p <.05 level

3 What Works Clearing House guidelines; http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Figure 10. Pretest and posttest teacher DTAMS Total Content Scores for E2MC partticipants 9 (n=39).

Figure 11. Pretest and posttest teacher DTAMS Total Knowledge Type scores for E2MC participants (n=39).

14.33 16.08

5

10

15

20

25

Pretest Posttest

Teacher DTAMS Total Content Scores Pretest to Posttest

15.79 19.08

5

10

15

20

25

Pretest Posttest

Teacher DTAMS Total Knowledge Type Scores Pretest to Posttest


33

Descriptive Examination of E²MC Teachers’ DTAMS Matter and Energy Subscores

Becausetheprofessionaldevelopmentcoursesspecificallytargetedmatterandenergy

contentknowledge,evaluatorsconductedsupplementaryanalysesonmatter(totalpossiblepoints=6,andenergy(totalpossiblepoints=13)subscoresoftheDTAMS.EvaluatorscalculatedchangesinscoresfrompretesttoposttestontheDTAMSmatterandenergyscores(seeFigures12and13).ForE²MCteachersinthisstudy,descriptivestatisticssuggestthatmatterandenergysubscoresscoresincreasedfrompretesttoposttest.Figure12presentsthepretest-to-posttestDTAMSmattersubscoreforE²MCparticipantsandFigure13presentsthepretest-to-posttestenergysubscore.

Paired Samples t-tests Examining E²MC Teachers’ Matter & Energy Subscore Gains on the DTAMS

Evaluatorsconductedapairedsamplest-testtoexaminewhetherteacherswhoparticipatedintheE2MCprojectmadesignificantgainsintheirmatterandenergycontentknowledgeasmeasuredbytheDTAMS.Resultsshowthatonaverage,mattersubscoresincreasedfrompretesttoposttestby0.10points,andenergysubscoresincreasedfrompretesttoposttestby1.62points.Resultsofpairedsamplest-testsindicatethatgainsformatterwerenotstatisticallysignificantbutgainsforenergywere(Table8).Effectsizecalculationsshowasmalleffectformattergainsandalargeeffectforenergy.Becausesubscoresarecalculatedonfewernumbersofitems,readersshouldinterpretthesefindingswithcaution.

Table 8. Results of a Paired Samples t-test of DTAMS Matter and Energy Subscores for Participating Teachers Outcomevariable(n=39)

Pretestmean

PretestSD

Posttestmean

PosttestSD

Meandiff.

t-value(df=38)

p-value

Effectsize

Matter 4.77 1.27 4.87 0.86 0.10 0.47 0.64 0.08Energy 6.33 1.95 7.95 2.56 1.62 4.26 <.001* 0.83* Statistically significant at the p <.05 level

Figure 12. Pretest and posttest teacher DTAMS Matter Subscores for E2MC participants (n=39)

Figure 13. Pretest and posttest teacher DTAMS Energy subscores for E2MC participants (n=39).

4.77 4.87

0

2

4

6

Pretest Posttest

Teacher DTAMS Matter Subcores Pretest to Posttest

6.33 7.95

0 2 4 6 8

10 12

Pretest Posttest

Teacher DTAMS Energy Subscores Pretest to Posttest


34

Descriptive Examination of E²MC Teachers’ DTAMS Knowledge Type Subscores

Becausetheprofessionaldevelopmentcoursesspecificallysupportedscientificinquiry(knowinghowtodoscience),schematicknowledge(deepunderstandingofscienceconceptslaws,theories,andprinciples)andpedagogicalcontentknowledge(PCK)(strategicknowledgeforscienceteaching),evaluatorsconductedsupplementaryanalysesonthesesubscoresofDTAMSknowledgetypes.Possiblepointsforthesesubscoreswere5pointsforscientificinquiry,15pointsforschematicknowledge,and10pointsforpedagogicalcontentknowledge.ForE²MCteachersinthisstudy,descriptivestatisticssuggestthatonaverage,teachers’scoresforschematicknowledgeandpedagogicalcontentknowledgeincreasedfrompretestingtoposttesting,whilescoresforscientificinquiryremainedthesame.Figures14,15,and16presentthepretest-to-posttestDTAMSTotalKnowledgeTypesubscoresforthesethreeknowledgetypes.

Figure 14.Pretest and posttest teacher DTAMS Scientific Inquiry Subscores for E2MC participants (n=39).

Figure 15. Pretest and posttest teacher DTAMS Schecmatic Knowledge Subscores for E2MC participants (n=39).

Figure 16. Pretest and posttest teacher DTAMS PCK Subscores for E2MC participants (n=39).

3.62 3.62

0 1 2 3 4 5 6

Pretest Posttest

Teacher DTAMS Scientific Inquiry Subscores Pretest to Posttest

7.10 9.03

0 2 4 6 8

10 12

Pretest Posttest

Teacher DTAMS Schematic Knowledge Subscores Pretest to

Posttest

1.46

3.00

0

1

2

3

4

5

Pretest Posttest

Teacher DTAMS PCK Subscores Pretest to Posttest


35

Paired Samples t-tests Examining E²MC Teachers’ Total Knowledge Type Subscore Gains on the DTAMS

Evaluatorsconductedapairedsamplest-testtoexaminewhetherteacherswhoparticipatedintheE2MCprojectmadesignificantgainsintheirschematicknowledgeandpedagogicalcontentknowledgeasemphasizedinthePD.Resultsindicatethatgainswerestatisticallysignificantforbothknowledgesubtypesandeffectsizesindicatealargeeffect(Table9). Table 9. Results of a Paired Samples t-test of DTAMS Knowledge Type Subscores for Participating Teachers OutcomeVariable

(n=39)PretestMean

PretestSD

PosttestMean

PosttestSD

MeanDiff

t-Value(df=38)

p-Value

EffectSize

Schem.Knowledge 7.10 2.24 9.03 2.44 1.93 5.60 <.001* 0.86PCK 1.46 1.41 3.00 2.25 1.54 5.12 <.001* 1.09

* Statistically significant at the p <.05 level

Comparison of Teacher DTAMS Scores by Condition

EvaluatorsconducteddescriptiveandinferentialanalysestocomparethelearningachievementforteacherswhoparticipatedinE2MCprofessionaldevelopmentandthosewhodidnot.Evaluatorscomparedtreatmentandcomparisonteachers’scoresontheposttestDTAMSTotalContentscoresandTotalKnowledgeTypescores. Descriptive Analysis Comparing DTAMS Scores by Condition

Figures17and18displaytreatmentandcomparisonteachers’DTAMSpretestandunadjusted

posttestscoresforTotalContentscoresandTotalKnowledgeTypescores.Theunadjustedmeansdonotaccountforthepretestdifferencesamongstudyparticipants.ThesedescriptivefindingssuggestthatRTOPscoresincreasedfortreatmentandcomparisonteachers.Table10presentspretestandposttestmeansandstandarddeviations.

Figure 17. Pretest and posttest mean DTAMS Total Content scores for treatment and comparison teachers.


36

Analysis of Covariance Comparing E2MC and Comparison Teachers Posttest Scores on the DTAMS

TounderstandwhetherteacherswhoparticipatedintheE2MCprofessionaldevelopmentprojectmadegreatergainsinTotalContentscoresthanthosewhodidnotparticipate,evaluatorsconductedone-waybetween-groupsanalysesofcovariance.Theindependentvariablewasthestudycondition(treatmentorcomparison),andthedependentvariablewastheposttestDTAMScontentscoreandseparately,theposttestTotalKnowledgeTypescore.Evaluatorsusedparticipants’scoresonthepretestDTAMSTotalContentscoreandpretestTotalKnowledgeTypescoresascovariatesintheseanalyses. ResultsindicatedtherewasnosignificantdifferencebetweentreatmentandcomparisonteachersonTotalContentscoresatposttestingatthep<.05significancelevel,F(1,75)=2.67,p=0.11.ForTotalContentscorestherewasastrongrelationshipbetweenpretestandposttestscoresasindicatedbyapartialetasquaredvalueof.58.

TherewasnosignificantdifferencebetweentreatmentandcomparisonteachersforposttestTotalKnowledgeTypescores,F(1,75)=3.14,p=0.084.Effectsizesweresmallasindicatedbyapartialetasquaredvalueof0.034forTotalContentscoresand0.040forTotalKnowledgeTypescores.ForTotalKnowledgeTypescorestherewasastrongrelationshipbetweenpretestandposttestscoresasindicatedbyapartialetasquaredvalueof.65.Table10presentspre-andposttestmeansandstandarddeviationsforthetwogroupsandtheadjustedposttestmeans. Evaluatorsconductedseparatesupplementaryanalysescomparingtreatmentandcomparisonteachers’posttestsubscoresforpedagogicalcontentknowledge(PCK),schematicknowledge,andscientificinquiryknowledgetypes.Resultsofanalysesofcovarianceindicatethatafteradjustingforpretestscores,therewasnosignificantdifferenceinposttestscoresforpedagogicalcontentknowledgebetweentreatmentandcomparisonteachers,F(1,75)=1.72,p=.19.Resultsindicatethatafteradjustingforpretestscores,therewasnosignificantdifferenceinposttestscoresforscientificinquiry,F(1,75)=2.46,p=.124.Resultsindicatethatafteradjustingforpretest

4 Evaluatorssetthealphalevelforsignificanceatp<.05forstudyfindings.However,forthepurposesoffederalreporting,USDOEconsidersMSPfindingswithp-valueslessthan0.15assignificant.

Figure 18. Pretest and posttest mean DTAMS Total Knowledge Type scores for treatment and comparison teachers.

14.33 16.08 13.85 14.79

0 5

10 15 20 25

Pretest Posttest

DTAMS Total Content Scores

Treatment Comparison

15.79 19.08

15.28 17.26

0 5

10 15 20 25

Pretest Posttest

DTAMS Total Knowledge Type Scores

Treatment Comparison


37

scores,treatmentteacherscoredsignificantlyhigheratposttestingforschematicknowledge(deepunderstandingofscienceconcepts,laws,andtheories)thancomparisonteachers,F(1,75)=6.12,p=.02.Effectsizewassmall(partialetasquared=.05).Table10presentspreandposttestmeansandadjustedposttestmeansforthesubtests. Evaluatorscomparedtreatmentandcomparisonposttestmatterandenergysubscores.ANCOVAresultsindicatethatafteradjustingforpretestscorestherewasnosignificantdifferenceinposttestscoresformatter,F(1,75)=3.64,p=.06.Afteradjustingforpretestscores,therewasnowasnosignificantdifferenceinposttestscoresforenergy,F(1,75)=1.92,p=.17.Table10presentspreandposttestmeansandadjustedposttestmeansforthesubtests.Table 10. Treatment and Comparison Teachers’ Average Pretest and Posttest DTAMS Total Content and Total Knowledge Type and Subtest scores.

Condition

Pretestmean

PretestSD

Posttestmean

PosttestSD

Estimatedmarginalposttestmean

Std.error

TotalContentScores T(n=39) 14.33 3.50 16.08 3.76 15.89 .39C(n=39) 13.85 3.69 14.79 3.67 14.99 .39TotalKnowledgeTypeScores T(n=39) 15.79 4.52 19.08 5.60 18.83 .53C(n=39) 15.28 4.87 17.26 5.55 17.50 .53PedagogicalContentKnowledge T(n=39) 1.46 1.41 3.00 2.30 2.98 .28C(n=39) 1.44 1.54 2.46 2.25 2.47 .28SchematicKnowledge T(n=39) 7.10 2.24 9.03 2.44 8.98 .30C(n=39) 7.00 2.50 7.90 2.67 7.94 .30ScientificInquiry T(n=39) 3.62 .96 3.62 1.16 3.60 .16C(n=39) 3.51 .91 3.23 .87 3.25 .16Matter T(n=39) 4.77 1.27 4.87 .86 4.83 .14C(n=39) 4.49 1.30 4.41 1.02 4.45 .14Energy T(n=39) 6.33 1.95 7.95 2.56 7.81 .32C(n=39) 5.95 2.04 7.05 2.27 7.19 .32

KEYFINDINGS:

Onaverage,E2MCteachersgained1.75pointsfrompretesttoposttestontheDTAMSTotalContentscoreand3.29pointsfrompretesttoposttestontheDTAMSTotalKnowledgeTypescore.E2MCteachersmadesignificantgainsfrompretestingtoposttestingontheDTAMSEnergysubtestandonsubtestsforschematicknowledgeandpedagogicalcontentknowledge.


38

Descriptive Findings and Inferential Analyses Regarding Teacher USTMEA Gains Among E²MC Program Participants

WestEddevelopedtheUnderstandingScienceforTeachingMatterandEnergyAssessments

toaligntotheMakingSenseofSCIENCEMatterandEnergycoursesusedintheE2MCprofessionaldevelopmentsessions.Assuch,evaluatorsusedtheseassessmentsasameasureofchangeinparticipatingteachers’contentknowledgetargetedthroughtheprofessionaldevelopmentprogram.Useoftheseassessmentsprovidedameansoftriangulatingfindingsrelatedtogainsinteachercontentknowledgebyusinganassessmentalignedspecificallytothecontentofthecourses.Toaddresskeyevaluationquestionsregardingteacherlearningformatterandenergyconceptsevaluatorsconductedavarietyofanalysesincludingdescriptiveandinferentialstatisticsaswellasthecalculationofeffectsizes. Descriptive Examination of E²MC Teachers’ Matter & Energy Score Gains

Tomeasureteachergainsinmatterandenergy,evaluatorsadministeredtheUSTMEAtoall

treatmentteachers.ToevaluateE²MCteachergains,evaluatorscalculatedchangesinscoresfrompretesttoposttest(seeFigures18and19).ForE²MCteachersinthisstudy,descriptivestatisticssuggestthatUSTMEAmatterandenergyscoresincreasedovertheschoolyear(pretesttoposttest).Figure19presentsthepretest-to-posttestUSTMEAmatterscoresforE²MCparticipantsandFigure20presentsthepretest-to-posttestUSTMEAenergyscoresforE²MCparticipants.

Paired Samples t-tests Examining E²MC Teachers’ Matter & Energy Score Gains

TodeterminewhetherE²MCteachermatterandenergyUSTMEAgainswerestatisticallysignificant,evaluatorsconductedpairedsamplest-tests.Todeterminethemagnitudeofthegainscoresevaluatorscalculatedstandardizedeffectsizesbydividingthepretest-to-posttestdifference

Figure 19. Pretest and posttest teacher matter scores for E2MC participants (n=39). .

Figure 20. Pretest and posttest teacher energy scores for E2MC participants (n=39).

23.67 25.82

10

15

20

25

30

Pretest Posttest

Teacher USTMEA Matter Scores Pretest to Posttest

21.18 23.33

10

15

20

25

30

Pretest Posttest

Teacher USTMEA Energy Scores Pretest to Posttest


39

bythepreteststandarddeviation.Effectsizesgreaterthan+/–0.25wereconsideredsubstantivelyimportant.5

Resultsshowthatonaverage,bothmatterandenergyscoresincreasedfrompretesttoposttestby2.15points.Thesegainswerestatisticallysignificant(p=<.001)andcorrespondedtolargeeffectssizesformatter(d=0.41)andenergy(d=0.63)(Table11).

Table 11. Results of Paired Samples t-tests For USTMEA Assessments for E2MC Teachers Outcomevariable(n=39)

Pretestmean

PretestSD

Posttestmean

PosttestSD

Meandiff.

t-value(df=38) p-value Effect

size

Matter 23.67 5.15 25.82 3.96 2.15 4.45 <.001* 0.41Energy 21.18 3.39 23.33 2.37 2.15 4.93 <.001* 0.63

Summary of Teacher Performance Results TeacherswhoparticipatedintheE²MCprofessionaldevelopmentprogramshowedsignificantgainsininstructionalpracticealignedtoreformedteachingasevidencedbytheirpretestandposttestscoresontheRTOP.E²MCteachersscoredhigheronthepostobservationthandidcomparisonteachers. E²MCteachersmadesignificantgainsontheDTAMSfrompretestingtoposttestingontheTotalContentandTotalKnowledgeTypescores.Teachersalsomadesignificantgainsontheenergysubtestandsubscoresforschematicknowledgeandpedagogicalcontentknowledge.E2MCteachersdidnotscoresignificantlyhigherthancomparisonteachersontheposttestTotalContentandTotalKnowledgeTypescoresatthep<.05significancelevel.Treatmentteachersmadegreatergainsinschematicknowledge,thedeepunderstandingofscienceconcepts,laws,andtheories,thandidcomparisonteachers.

TheUSTMEAisalignedtotheMakingSenseofSCIENCEcourses.E²MCteachersmadestatisticallysignificantgainsontheUSTMEAtestsofmatterandenergycontentknowledge.Gainscorrespondedtolargeeffectsizesformatter(0.41)andenergy(0.63).

5 What Works Clearing House guidelines; http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

KEYFINDING:

Onaverage,E2MCteachersgained2.15pointsfrompretesttoposttestontheUSTMEAmattertestandontheUSTMEAenergytest.


40

Student Performance Results

Toexaminewhetherstudentsinasampleofparticipatingteachers’classroomsmadesignificantgainsincontentknowledgewithrespecttophysicalscienceconceptsrelatedtomatterandenergy,evaluatorsexaminedscoresfrompre-toposttestingontheUnderstandingScienceforTeachingMatterandEnergyAssessmentsforstudents.Thissectionpresentsfindingsrelatedtostudentachievement.

Descriptive Findings and Inferential Analyses Regarding Student USTMEA Gains Among E²MC Program Participants

Toaddresskeyevaluationquestionsregardingstudentlearningformatterandenergy,evaluatorsconductedavarietyofanalysesincludingdescriptiveandinferentialstatisticsaswellasthecalculationofeffectsizes.Inthisstudy,studentswerenestedinteachers’classrooms,makingmultilevelmodelingthemostappropriateanalytictechniqueforconductinginferentialanalyseswithstudentoutcomedata.Evaluatorsanalyzeddatafromthe2014–15schoolyeartoassesschangesinstudentmatterandenergyachievementusingtheUnderstandingScienceforTeachingMatterandEnergyAssessments(USTMEA).

Descriptive Examination of E²MC Students’ Matter & Energy Score Gains

Tomeasurestudentgainsinmatterandenergy,evaluatorsadministeredtheUSTMEAtoalltreatmentstudents.ToevaluateE²MCstudentgains,evaluatorscalculatedchangesinscoresfrompretesttoposttest(seeFigures20and21).Theunadjustedmeanswerecalculatedfordescriptivepurposestoshowtrendsandwerenotintendedtodetermineifdifferenceswerestatisticallysignificant.Thescoresareconsidered“unadjusted”ororiginalscoresbecausetheydonottakeintoaccountthevarianceassociatedwithstudent-andteacher-levelfactors.

ForE²MCstudentsinthisstudy,descriptivestatisticssuggestthatUSTMEAmatterandenergyscoresincreasedovertheschoolyear(pretesttoposttest).Figure21presentsthepretest-to-posttestunadjustedUSTMEAmatterscoresforE²MCparticipants,andFigure22presentsthepretest-to-posttestunadjustedUSTMEAenergyscoresforE²MCparticipants.

KeyQuestion:Dostudentsinparticipatingteachers’classroomsexperiencestatisticallysignificantgainsincontentknowledgeoverthecourseofthestudy?


41

Multilevel Modeling Analyses Examining E²MC Students’ Matter & Energy Score Gains

TodeterminewhetherE²MCstudentmatterandenergyUSTMEAgainswerestatisticallysignificant,evaluatorsconductedmultilevelmodelinganalyses,whichtakeintoaccountthevarianceassociatedwithstudentandteacher-levelfactors.Specifically,evaluatorsranseparatemodelsforeachoutcomeofinterest(matterandenergy)withstudentsnestedinteachers,andtheoutcomeofinterestwasthepretest-to-posttestgainscore.Todeterminethemagnitudeofthegainscores,evaluatorscalculatedstandardizedeffectsizesbydividingtheadjustedpretest-to-posttestdifferencebythepreteststandarddeviation.Effectsizesgreaterthan+/–0.25wereconsideredsubstantivelyimportant.

Resultsshowthatonaverage,matterscoresincreasedfrompretesttoposttestby5.92points,andenergyscoresincreasedfrompretesttoposttestby4.15points.Thesegainswerestatisticallysignificant(p=<.01)andcorrespondedtolargeeffectssizesformatter(d=1.42)andenergy(d=1.19)(Table12).Table 12. Mean USTMEA Matter and Energy Pretest-to-Posttest Gain Scores for E²MC Students

Outcomevariable CoefficientStandarderror

t-Value

Approx.df

p-Value

Effectsize

Matter 5.92 1.33 4.45 6 <.01* 1.42

Energy 4.15 1.08 3.83 9 <.01* 1.19* Statistically significant at the p <.05 level

Figure 21. Unadjusted pretest and posttesnt student USTMEA matter scores for E2MC participants.

Figure 22. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants.

12.44

18.44

0

5

10

15

20

Pretest Posttest

Student USTMEA Matter Scores Pretest to Posttest

11.15

15.55

0

5

10

15

20

Pretest Posttest

Student USTMEA Energy Scores Pretest to Posttest

KEYFINDINGS:

Onaverage,studentsgained5.92pointsfrompretesttoposttestonmatter.

Onaverage,studentsgained4.15pointsfrompretesttoposttestonenergy.


42

Relationship Between Student Characteristics and E²MC Students’ USTMEA Gain Scores

ToexplorehowstudentcharacteristicsrelatedtoE²MCmatterandenergyachievement,evaluatorsconductedexploratoryanalysestoexaminewhetherUSTMEAgainscoresformatterandenergywererelatedtogender,freeandreduced-pricelunch,grade,andethnicity.EvaluatorscalculateddescriptivestatisticsandusedmultilevelmodelingtoconductinferentialanalysesfortheUSTMEAgainscores,whichaccountforteacherandstudentvariation.Descriptive Examination of E²MC Students’ Matter & Energy Score Gains by Student Characteristics

First,evaluatorsexaminedunadjusteddescriptivestatisticsforeachstudentcharacteristic.Theunadjustedmeanswerecalculatedfordescriptivepurposestoshowtrendsandwerenotintendedtodetermineifdifferenceswerestatisticallysignificant(seeFigures22–23).Thescoresareconsidered“unadjusted”ororiginalscoresbecausetheydonottakeintoaccountthevarianceassociatedwithstudent-andteacher-levelfactors.Evaluatorsfirstpresentthedescriptiveresultsformatter,followedbyenergy.Matter

ForE²MCstudentsinthisstudy,descriptivestatisticssuggestthatUSTMEAmatterscoresforbothmalesandfemalesincreasedovertheschoolyear(pretesttoposttest),withmalesscoringslightlymorethanfemalesatposttest.Figure23presentsthepretestandposttestunadjustedUSTMEAmatterscoresforE²MCparticipantsbygender.

Next,evaluatorscalculateddescriptivestatisticsformatterscoresforfreeandreduced-price

lunch(FRL).DescriptivestatisticssuggestthatstudentswhodidnotqualifyforFRLgainedmorefrompretesttoposttestthanstudentswhodidqualify.Bothgroupsincreasedovertheschoolyear(pretesttoposttest),withstudentswhodidnotqualifyforFRLscoringmorethanstudentswhodid

Figure 23. Unadjusted pretest and posttest student USTMEA matter scores for E2MC participants by gender

Figure 24. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants by FRL.

12.40

18.06 12.47

18.77

0

5

10

15

20

Pretest Posttest

Student USTMEA Matter Scores Gender

Female Male

13.17

19.81

10.54

14.80

0

5

10

15

20

Pretest Posttest

Student USTMEA Matter Scores FRL

No Yes


43

qualifyatpretestandposttest.Figure24presentsthepretestandposttestunadjustedUSTMEAenergyscoresforE²MCparticipantsbyFRL.

Next,evaluatorscalculateddescriptivestatisticsformatterscoresbyethnicity.Descriptive

statisticssuggestthatallstudents’matterscoresincreasedfrompretesttoposttest.ResultssuggestthatCaucasianstudentsgainedmorethannon-Caucasianstudentsfrompretesttoposttest.Figure25presentsthepretestandposttestunadjustedUSTMEAmatterscoresforE²MCparticipantsbyethnicity.

Figure 25.Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by ethnicity.

Lastly,evaluatorscalculateddescriptivestatisticsformatterscoresbygrade.Descriptivestatisticssuggestthatallstudents’matterscoresincreasedfrompretesttoposttest.Resultssuggestthateighthgradestudentsgainedmorethansixthgradestudentsfrompretesttoposttest.Figure26presentsthepretestandposttestunadjustedUSTMEAmatterscoresforE²MCparticipantsbygrade.

Figure 26. Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by grade.

Energy

Similartothematterresultsabove,descriptivestatisticssuggestthatUSTMEAenergyscores

forgenderincreasedovertheschoolyear(pretesttoposttest),withmalesscoringslightlymorethanfemalesatpretestandposttest.Figure27presentsthepretestandposttestunadjustedUSTMEAenergyscoresforE²MCparticipantsbygender.

13.29 10.35

19.57 15.51

Pretest Posttest

8.96 13.76

10.04

21.62

6th Grade 8th Grade

Pretest Posttest


44

Next,evaluatorscalculateddescriptivestatisticsforenergyscoresforFRL.Descriptivestatisticssuggestthatbothgroupsincreasedovertheschoolyear(pretesttoposttest),withstudentswhodidnotqualifyforFRLscoringmorethanstudentswhodidqualityforFRLatpretestandposttest.Figure28presentsthepretestandposttestunadjustedUSTMEAenergyscoresforE²MCparticipantsbyFRL.

Next,evaluatorscalculateddescriptivestatisticsforenergyscoresbyethnicity.Descriptive

statisticssuggestthatallstudents’energyscoresincreasedfrompretesttoposttest.ResultssuggestthatCaucasianstudentsgainedmorethannon-Caucasianstudentsfrompretesttoposttest.Figure29presentsthepretestandposttestunadjustedUSTMEAenergyscoresforE²MCparticipantsbyethnicity.

Figure 29. Unadjusted pretest and posttest student USTMEA energy scores for E2MC participants by ethnicity.

Lastly,evaluatorscalculateddescriptivestatisticsforenergyscoresbygrade.Descriptive

statisticssuggestthatformostgrades,energyscoresincreasedfrompretesttoposttest,withsixthgradeshowingthelargestgains.However,fortenthgradestudents,scoresdecreasedfrompretesttoposttest.Figure30presentsthepretestandposttestunadjustedUSTMEAmatterscoresforE²MCparticipantsbygrade.

11.79 10.27

16.51 14.35

Caucasian Non-Caucasian

Pretest Posttest

Figure 27. Unadjusted pretest and posttest student USTMEA energy scores for E2MC participants by gender.

Figure 28. Undadjusted pretest and posttest student USTMEA energy scores for E2MC participants by FRL.

10.89

15.42 11.54

15.96

0

6

12

18

Pretest Posttest

Student USTMEA Energy Scores Gender

Female Male

11.47

15.80

10.67

15.29

0

6

12

18

Pretest Posttest

Student USTMEA Energy Scores FRL

No Yes


45

Figure 30. Unadjusted pretest and and posttest student USTMEA energy scores for E²MC participants by grade. Multilevel Modeling Analyses Examining E²MC Students’ Matter & Energy Score Gains by Student Characteristics

ToexplorehowstudentcharacteristicsrelatedtoE²MCstudents’matterandenergyachievement,evaluatorsusedmultilevelmodelingtoaccountforstudentvariationfortheUSTMEAassessment,whichwasadministeredatpretestandposttest.Toaccountforvariationinstudentperformance,thisexploratoryanalysisexaminedwhetherstudentperformanceontheUSTMEAmatterandenergyassessmentswasrelatedtostudent-levelcovariates:gender,FRL,grade,andethnicity.

Subgroupanalysesformatterandenergygainscoresforgender,FRL,andethnicity,were

notstatisticallysignificant,andeffectsizesweresmall(seeTables13and14).Forgender,theresultsformattersuggestthatfemalesgainedlessthanmalesfrompretesttoposttest,andtheresultsforenergysuggestthatfemalesgainedmorethanmalesfrompretesttoposttest.ForFRL,theresultsformatterandenergysuggestthatFRLstudentsgainedmorethannon-FRLstudentsfrompretesttoposttest.Forethnicity,theresultsformattersuggestthatCaucasianstudentsgainedlessthannon-Caucasianstudentsforpretesttoposttest,andtheresultsforenergysuggestthatCaucasianstudentsgainedmorethannon-Caucasianstudentsfrompretesttoposttest.Becausetheseanalysesdividethesampleintosubgroupsandisconsideredexploratory,readersshouldusecautionwheninterpretingfindings.

11.03 11.19

13.25 12.74

10.44 9.83 8.88

16.99

12.50

16.75

13.00

8.22

12.45 13.50

Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12

Pretest Posttest


46

Table 13. Relationship Between Student Characteristics and USTMEA Matter Scores

PretesttoPosttest


t-value

Approx.df p-value

Effectsize

Gender –0.77 0.53 –1.46 188 0.15 –0.16

FRL 0.19 0.61 0.31 188 0.76 0.04

Grade 3.43 0.48 7.14 188 <0.01* 0.69

Ethnicity –0.66 0.59 –1.13 188 0.26 –0.13* Statistically significant at the p <.05 level Table 14. Relationship Between Student Characteristics and USTMEA Energy Scores

PretesttoPosttest

Outcomevariable Coefficient Standarderror

t-value

Approx.df

p-value Effectsize

Gender 0.70 0.47 1.48 307 0.14 0.13

FRL 0.32 0.53 0.60 307 0.55 0.06

Grade –0.59 0.43 –1.38 307 0.17 –0.11

Ethnicity 0.78 0.50 1.55 307 0.12 0.15

EvaluatorsalsoexaminedtherelationshipbetweengradeandUSTMEAmatterandenergygainscores.Forenergy,resultsbygradewerenotstatisticallysignificant,andeffectsizesweresmall.Formatter,resultswerestatisticallysignificant,andcorrespondedtoasubstantivelyimportant,largeeffectsize(d=0.69).Forenergy,resultssuggestthatstudentsinhighergradesgainedlessthanstudentsinlowergrades.Formatter,resultsshowthatstudentsinhighergradesgainedmorethanstudentsinlowergrades.

Descriptive Examination of E²MC Students’ Matter & Energy Score Gains by Teacher Characteristics

Similartotheexaminationofstudentcharacteristics,evaluatorsexaminedunadjusteddescriptivestatisticsforeachteachercharacteristic.Theunadjustedmeanswerecalculatedfordescriptivepurposestoshowtrendsandwerenotintendedtodetermineifdifferenceswerestatisticallysignificant.Thescoresareconsidered“unadjusted”ororiginalscoresbecausetheydo

KEYFINDING:

Onaverage,studentsinhighergradesmadestatisticallysignificantlylargergainsinmatterfrompretesttoposttestthanstudentsinlowergrades.


47

nottakeintoaccountthevarianceassociatedwithstudent-andteacher-levelfactors.Evaluatorsfirstpresentthedescriptiveresultsformatter,followedbyenergy.

Matter

First,evaluatorscalculateddescriptivestatisticsformatterscoresbyteachers’yearsofexperience.Descriptivestatisticssuggestthatallstudents’matterscoresincreasedfrompretesttoposttest.Resultssuggeststudentsinclassroomswithteacherswith14or24yearsofteachingexperiencegainedmorefrompretesttoposttestthanteacherswithotheryearsofexperience.Figure31presentsthepretestandposttestunadjustedUSTMEAmatterscoresforE²MCparticipantsbyteachers’yearsofteachingexperience.

Figure 31. Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by teachers’ years of teaching experience.

Next,evaluatorscalculateddescriptivestatisticsformatterscoresbyteacherimplementationscores.Asnoted,thisscorewasobtainedbydividingthenumberofenergyormattertopicsbythetotalnumberofpossibleMakingSenseofSCIENCEtopicscoveredinthatsubjectarea.Descriptivestatisticssuggestthatallstudents’matterscoresincreasedfrompretesttoposttest.Resultssuggeststudentsinclassroomswith100%implementationgainedmorethanteacherswithotherimplementationscoresfrompretesttoposttest.Figure32presentsthepretestandposttestunadjustedUSTMEAmatterscoresforE²MCparticipantsbyteachers’levelofimplementation.

14.48 14.06

10.56 13.17

7.63

13.93 13.11

21.76 20.72

10.72

22.07

9.47

23.26 20.46

2 4 9 14 17 24 26

Pretest Posttest


48

Figure 32. Unadjusted pretest and posttest student USTMEA matter scores for E²MC participants by teachers’ level of implementation (percentage score). Energy

Forenergy,evaluatorscalculateddescriptivestatisticsforenergyscoresbyteachers’yearsof

experience.Descriptivestatisticssuggestthatallstudents’energyscoresincreasedfrompretesttoposttest.Resultssuggeststudentsinclassroomswithteacherswith21yearsofteachingexperiencegainedmorethanteacherswithotheryearsofexperiencefrompretesttoposttest.Figure33presentsthepretestandposttestunadjustedUSTMEAenergyscoresforE²MCparticipantsbyteachers’yearsofteachingexperience.

Figure 33. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants by teachers’ years of teaching experience.

Next,evaluatorscalculateddescriptivestatisticsforenergyscoresbyteacher

implementationscores.Descriptivestatisticssuggestthatallstudents’energyscoresincreasedfrompretesttoposttest.Resultssuggeststudentsinclassroomswith65%implementationgainedmorethanteacherswithotherimplementationscoresfrompretesttoposttest.Figure34presentsthepretestandposttestunadjustedUSTMEAenergyscoresforE²MCparticipantsbyteachers’levelofimplementation.

7.63

13.17 10.56

14.48 13.62 13.93

9.47

22.07

10.72

21.76 20.60 23.36

12% 71% 76% 88% 94% 100%

Pretest Posttest

10.83 12.37

9.19 11.42

9.91 11.52

13.43 12.50 12.29 11.66 12.67

17.50 17.61 15.12

16.64

23.54

3 6 9 13 14 18 20 21

Pretest Posttest


49

Figure 34. Unadjusted pretest and posttest student USTMEA energy scores for E²MC participants by teachers’ level of implementation (percentage score).

Relationship between Teacher Characteristics and E²MC Students’ USTMEA Gain Scores

ToexplorehowteachercharacteristicsrelatedtoE²MCstudentachievement,evaluatorsusedmultilevelmodelingtoaccountforteachervariationintheUSTMEAassessment,whichwasadministeredatpretestandposttest.Toaccountforvariationinstudentperformance,theseexploratoryanalysesmodelsexaminedwhetherthestudentperformancewasrelatedtoteacher-levelcovariates:yearsofexperienceandimplementation.ResultsoftheseanalysesarepresentedinTables15and16.Table 15. Relationship Between Teacher Characteristics and USTMEA Matter Scores

PretesttoPosttest

Outcomevariable Coefficient Standarderror

t-value

Approx.df

p-value Effectsize

YearsofExperience 0.10 0.16 0.65 5 0.55 0.02

Implementation 7.83 3.67 2.13 5 0.09 1.58

Table 16. Relationship Between Teacher Characteristics and USTMEA Energy Scores

PretesttoPosttest


t-value

Approx.df p-Value

Effectsize

YearsofExperience 0.33 0.13 2.47 8 0.04* 0.06

Implementation 6.48 11.84 0.55 8 0.60 1.22* Statistically significant at the p <.05 level

Foryearsofexperiencewithmattergainscores,resultswerenotstatisticallysignificantand

correspondedtoasmalleffectsize(0.02).Resultssuggestthatmoreyearsofexperiencecorrespondedtoalargergain.Foryearsofexperiencewithenergygainscores,resultswere

10.34 13.43 12.50 11.30 12.37

10.36 13.14

16.64

23.54

14.90 11.66

17.03

55% 60% 65% 70% 75% 80%

Pretest Posttest


50

statisticallysignificantandalsosuggestedthatmoreyearsofexperiencecorrespondedtolargergains.However,resultscorrespondedtoasmalleffectsize(d=0.06).

Forimplementation,resultswerenotstatisticallysignificantformatterandenergybutcorrespondedtolargeeffectsizes(d=1.58andd=1.22).Resultssuggestthatlargerimplementationscorescorrespondedtolargergains.Alargeeffectsizesuggeststhattheremaybeastrongrelationshipbetweenimplementationandgainscores.However,becausethisstudyhadasmallsamplesize,thestatisticalpowertodetectsignificanteffectswasrelativelylowerthanitwouldhavebeenifthesamplesizehadbeenlarger.Therefore,itispossiblethatifthesamplesizehadbeenlarger,thisfindingmighthavebeenstatisticallysignificant.Summary of Student Performance Results

Overall,resultsshowthatonaverage,matterandenergyscoresstatisticallysignificantly

increasedfrompretesttoposttest,andresultscorrespondedtolargeeffectsizes.Exploratoryanalysesofstudentandteachercharacteristicsshowthatstudentgradeisstatisticallysignificantlyrelatedtomattergainscores,withstudentsinhighergradesearningmorethanstudentsinlowergrades.Forteachercharacteristics,teachers’yearsofexperienceisalsoastatisticallysignificantpredictorforenergygainscores,withthestudentsofmoreexperiencedteachersgainingmorethanthestudentsofteacherswithfeweryearsofexperience.Fortheremainingsubgroupanalyses,resultsindicatethatstudentsperformedsimilarlyovertime.Forteacherimplementation,resultswerenotstatisticallysignificantbutcorrespondedtosubstantivelyimportanteffectsizes(>0.25),suggestingthatlargerimplementationscorescorrespondedtolargergains.


51

Summary and Discussion

Thisquasi-experimentalstudyevaluatedtheE²MCprofessionaldevelopment(PD)projectfundedbytheArizonaDepartmentofEducation’sMathematicsandSciencePartnershipprogramintermsofitsefficacyinincreasingteacherscontentknowledgeinthephysicalsciencesandeffectivepedagogicalpracticesinthescienceclassroom.TheDiagnosticTeacherAssessmentinMathematicsandScience(DTAMS)andtheUnderstandingSciencemeasuredcontentknowledgeforTeachingMatterandEnergyAssessments(USTMEA).EffectivepedagogywasmeasuredthroughclassroomobservationsusingtheReformedTeachingObservationProtocol(RTOP).Thestudyalsoassessedthedegreetowhichstudentsinparticipatingteachers’classroomsshowedevidenceofincreasedunderstandingofphysicalscienceconceptsrelatedtomatterandenergyasmeasuredbytheUnderstandingScienceforTeachingMatterandEnergyAssessments(USTMEA).Participantsinthestudyincluded39treatmentand39comparisonmiddleandhighschoolteachersintwodistrictsinArizona. TeacherswhoparticipatedintheE²MCprofessionaldevelopmentfeltthattheprogramdeepenedtheirunderstandingofenergyandmatterconceptsaswellastheirunderstandingofeffectivescienceteachingstrategies.Teachersvaluedtheabilitytocollaboratewithpeerstodevelopunitplansandappreciatedthecross-gradecollaborationsbetweenmiddleandhighschoolteachers.Teachersgainedconfidenceindesigningopportunitiesforstudentstounderstandtypesofquestionsthatcanbeinvestigatedscientifically,useevidencetodevelopexplanationsandjustifytheirthinking,andcommunicateeffectivelyinscientificformats.FormativeobservationsrevealedareastotargetinsubsequentPDsessions,includinghelpingteacherstodesignactivelearningexperiencesandprovidingstudentswithwaystoinvestigatescientificquestions.

FindingsregardingteacherinstructionrevealedthatteacherswhoparticipatedinE²MCPDdemonstratedstatisticallysignificantgainsineffectiveteachingstrategiesasevidencedbyincreasedscoresontheRTOP,whichcorrespondedtoalargeeffectsizeof0.51.Findingsalsoindicatedthatparticipatingteachersscoredsignificantlyhigheronthepostobservationthandidnonparticipants.

Findingsregardingteachercontentknowledgeshowedthatparticipatingteachersmade

statisticallysignificantgainsincontentknowledgeasevidencedbypretesttoposttestgainsontheDTAMSTotalContentscoreandtheUSTMEAassessments.OntheDTAMStheeffectsizewas0.50forTotalContentscoregains.E²MCparticipants’gainsontheDTAMSenergysubtestwerestatisticallysignificantandcorrespondedtoalargeeffectsizeof0.83.GainsontheDTAMSmattersubtestwerenotstatisticallysignificantandcorrespondedtoasmalleffectsize.OntheUSTMEA,testsofcontentknowledgealignedtotheMSScoursesusedinthePDsessions,teachersmadesignificantgainsinmatterandenergywithcorrespondinglylargeeffectsizes(0.41and0.63respectively).ThestudyrevealednostatisticallysignificantposttestdifferencesinDTAMSscoresbetweentreatmentandcomparisonteachers.

FindingsrevealedthatstudentsinclassroomsofE2MCteachersexperiencedgainsintheir

matterandenergycontentknowledgeasmeasuredbytheirscoresontheUSTMEAmatterandenergyassessments.Gainscorrespondedtolargeeffectsizes:1.42formatterand1.19forenergy.Withoutacomparisongroup,readersshouldinterpretstudentfindingswithcaution.However,theverylargeeffectsizesindicatethatstudentsinparticipatingclassroomsexperiencedimportantgains


52

incontentknowledge.

Inconclusion,theE²MCprofessionaldevelopmentprogramofferedmanybenefitstoparticipants.Theygainedvaluablecontentknowledgeinphysicalsciencesconceptsthatcouldbedirectlyappliedtotheirgrade-levelinstruction.Especiallyofnotewasthelearningregardinghowtomakeconnectionsbetweenenergyandmatterexplicitintheirinstruction.Teachersgainedconfidenceinimplementingeffectiveteachingstrategiesintheirclassrooms,andstudentsinparticipatingteachers’classroombenefitedfromtheknowledgegainedbytheirteachersthroughE²MC.

E²MCteachershadvaluableopportunitiestoplanandcollaborateonunitandlesson

development,somethingnotalwaysavailablewithintheschoolenvironment.Especiallyvaluedwastheopportunitynotonlyforteacherstoworkwithingrade-levelteamsbutalsoformiddleandhighschoolteacherstoworkacrossgrades.Cross-gradecollaborationprovidedopportunitiestounderstandthecontentthatisappropriatetoimplementacrossthe6–12spectrum.

Teachersindicatedthatthegreatestbarriertoimplementingtheirlearningwaslackoftime

outsideofthePDsessionstoplanandcollaboratewithpeers.Therefore,providingsuchopportunitiesthroughqualityprofessionaldevelopmentexperiencesiscriticalinsupportingteacherstoincreasetheirunderstandingofeffectivescienceinstruction.Additionally,thevariationinqualityamongparticipantunitplansspeakstotheneedforongoingfeedbackandprofessionaldevelopmentexperiencesthatallowparticipantstopracticelearningfromPDprogramssuchasE²MC.Teachersclearlybenefittedfromwhattheylearned,buttheneedforongoingsupportwithinthedistrictsiscriticaltosustainingthelearning.

Overall,findingsfromthisevaluationstudyfoundthattheE²MCprofessionaldevelopment

projectshowedmanyofthecharacteristicsofeffectivePD.Useofhigh-qualitycurricularmaterials,suchastheMakingSenseofSCIENCEcourses,supportedstructuredandsystematiclearningforteachersandallowedthemtoexperiencethecontentandinvestigationsastheirstudentswouldexperiencethem.PDsessionsemphasizedalignmentofactivities,lessons,andunitstoinstructionalgoalsandpromoteactivelearningintheclassroom.


53

References Allen,C.,&Penuel,W.(2014).Studyingteachers’sensemakingtoinvestigateteachers’responsesto

professionaldevelopmentfocusedonnewstandards.JournalofTeacherEducation.Downloadedfromhttp://jte.sagepub.com/content/66/2/136,December22,2014,

Bloom,H.S.,Richburg–Hayes,L.,&Black,A.R.(2007).Usingcovariatestoimproveprecisionfor

studiesthatrandomizeschoolstoevaluateeducationalinterventions.EducationalEvaluationandPolicyAnalysis,29(1),30–59.

Daehler,K.,Folsom,J.,Mendenhall,J.,Shinohara,M.(2012).MakingSenseofSCIENCE:Energyfor

TeachersGrades6–8.(TeacherBook&FacilitatorGuide).WestEdErikson,F.(1986).Qualitativemethodsinresearchonteaching.InM.C.Wittrock(Ed.),Handbookof

researchonteaching(3rded.,pp.119–161).NewYork:MacMillan.Hedges,L.V.,&Hedberg,E.C.(2007).Intraclasscorrelationvaluesforplanninggroup-randomizedtrialsineducation.EducationalEvaluationandPolicyAnalysis,29(1),60–87.

Luellen,J.K.,Shadish,W.R.&Clark,M.H.(2005).Propensityscores:Anintroductionandexperimentaltest.EvaluationReview,29(6),530–558.

NationalResearchCouncil.(2012).AFrameworkforK–12ScienceEducation:Practices,CrosscuttingConcepts,andCoreIdeas.CommitteeonaConceptualFrameworkforNewK–12ScienceEducationStandards.BoardonScienceEducation,DivisionofBehavioralandSocialSciencesandEducation.Washington,DC:TheNationalAcademies

NGSSLeadStates.(2013).NextGenerationScienceStandards:ForStates,ByStates.Washington,DC:

TheNationalAcademiesPress.Penuel,W.,Fishman,B.,Yamaguchi,R.,&Gallagher,L.(2007).Whatmakesprofessional

developmenteffective?Strategiesthatfostercurriculumimplementation.AmericanEducationalResearchJournal,44(4),921–958.

Penuel,W.,Gallagher,L.,&Moorthy,S.(2011).Preparingteacherstodesignsequencesofinstruction

inearthsystemsscience:Acomparisonofthreeprofessionaldevelopmentprograms.AmericanEducationalResearchJournal,48(4),996–1025.

Piburn,M.,Sawada,D.,Falconer,K.,Turley,J.Benford,R.,Bloom,I.(2000).ReformedTeaching

ObservationProtocol(RTOP)(No.ACEPTIN-003):ArizonaCollaborativeforExcellenceinthePreparationofTeachers:ArizonaStateUniversity.

Puma,MichaelJ.,RobertB.Olsen,StephenH.Bell,andCristoferPrice(2009).Whattodowhendata

aremissingingrouprandomizedcontrolledtrials(NCEE2009-0049).Washington,DC:NationalCenterforEducationEvaluationandRegionalAssistance,InstituteofEducationSciences,U.S.DepartmentofEducation.


54

U.S.DepartmentofEducation,InstituteofEducationSciences:WhatWorksClearinghouse.(2008,December).WhatWorksClearinghouseproceduresandstandardshandbook(Version2.0).Washington,DC:Author.Retrievedonlinefrom:http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_handbook.pdf

WestEd.2013.MakingSenseofSCIENCE.Retrievedonlinefrom:http://we-mss.weebly.comWestEd.(2010).SummaryofMakingSenseofSCIENCEResearchResults.Retrievedfromhttp://we-

us4t.weebly.com/uploads/8/6/4/9/8649828/mss_research_summary.pdf


55

Appendix A: Professional Development Topics Table A-1. Workshop Schedule and Big Ideas

Date BigIdeas

June16–20,2014&June23–27,2014 MakingSenseofSCIENCEEnergyCourse• Whatenergyisandisnot• Typesofenergy• Conservationofenergy• Energyflowsthroughsystems• Howenergymovesmatter

MakingSenseofSCIENCEMatterCourse• Allmatterhasmassandtakesupspace• Propertiesofmatter• Chemicalandphysicalchanges• Mattercyclesthroughsystems• Particulatenatureofmatter• Chemicalelementsofmatter

Pedagogy• Researchoneffectivescienceinstruction.• Strategiesforreadingandwritingtolearnand

communicatescientificthinking• Evidence-baseddiscourseinscience• Supportforstudentsdevelopingexplanatory

modelsusingclaims,evidence,andreasoning• Teachingforconceptualunderstandingthrough

crosscuttingconceptsandBigIdeas• Developingeffectiveinstructionalunits

September12–13,2014 ScienceCurriculumTopicStudy• Improveunderstandingofsciencecontent• IdentifyBigIdeas,concepts,specificideas,and

importantfactsandterminologythatmakeupatopic

• Clarifythemeaningandintentoflearninggoalswithinatopic

• Identifydevelopmentalissues,difficulties,andmisconceptionsassociatedwithatopic

• Examineandapplyeffectiveteachingandassessmentstrategiesintopicstaught

• Makerelevantconnectionsamongideasinatopic• Developacommonlanguageandknowledgebase

aboutscienceteachingandlearningContentInvestigation:

• Exploringiceballs• Developinginvestigablequestionsinscience• Developingmodelstoexplainphenomena


56

October24–25,2014 ContinuedmatterandenergyintegratedunitdevelopmentStartwiththeendinmind

• Considerwaystosequencelessonsandconceptstaughtinahierarchyformaximumstudentunderstanding

Contentinvestigation:• Understandinghowconvection,conduction,

radiation(heatenergy)movematter• Modelingconvection,conduction,andradiation

January23–24,2015 Contentinvestigation:• Exploringevidenceofchemicalreactions• Developmodelsasexplanationsofphenomena,

supportedwithevidenceandreasoning.Continuedintegratedunitdevelopment

• Shiftinglessons;howtomodifylessonstobemoreinquirybased;benefitsofinquiry-basedlessons

• Developingstorylinestoinsureacoherentbuildingtowardconceptualunderstanding

• Incorporatingformativeassessmentstomonitorlearningandinforminstructionforunits

March27–28,2015 Completing/publishingintegratedunitplans• Teamsrevisedunitplansbasedonfeedbackfrom

instructorsandturnedinelectroniccopies.Postcontentassessmentsforteachers

• MatterandenergyMSSassessments• Namethatconcept.Groupscreatedmodelsto

explainaconceptorconcepts.Otherteamscritiquedthemodelsbyguessingtheconcepttheybelievethemodelexplains.


57

Appendix B: Missing Data Rates by Assessment

Table B-1. Missing Data Rates by Condition

PercentmissingRTOP

Pretest 0.00%Posttest 0.00%

DTAMS Pretest 2.56%Posttest 1.28%

MSSEnergy(Teacher) Pretest 0.00%Posttest 7.69%

MSSMatter(Teacher) Pretest 0.00%Posttest 7.69%

MSSEnergy(Student) Pretest 6.17%Posttest 7.10%

MSSMatter(Student) Pretest 2.50%Posttest 12.00%


58

Appendix C: Teacher Demographics and Tests of Equivalence Table C-1. Teacher Demographics

Comparison Treatment Total Chi-squareteachers teachers teachers results(n=39) (n=39) (n=78)

Characteristics Percent N Percent N Percent NX²

Value

Sig.(alpha=0.05)

Gender .278 .60Female 71.79% 28 79.49% 31 75.64% 59

Male 28.21% 11 20.51% 8 24.36% 19Gradetaught

.00 1.00Middleschool 74.36% 29 76.92% 30 75.64% 59

Highschool 25.64% 10 23.08% 9 24.36% 19Subjecttaught

.882 .83

Biology 10.26% 4 5.13% 2 7.69% 6EnvironmentalScience

2.56% 1 2.56% 1 2.56% 2

Science 82.05% 32 84.62% 33 83.33% 65Integratedscience 5.13% 2 7.69% 3 6.41% 5

HighestDegreeObtained

.488 .49

BA/BS 43.59% 17 33.33% 13 38.46% 30MA/MS 56.41% 22 66.67% 26 61.54% 48


59

Appendix D: Demographics for Students Participating in the USTMEA Assessments

Table D-1. Energy and Matter Student Demographics

Energy Matter Totalstudents students students(n=324) (n=200) (n=524)

Characteristics Percent N Percent N Percent N

Grade

Sixth 61.92% 200 27.50% 55 48.76% 255Seventh 8.05% 26 - - 4.97% 26Eighth 8.67% 28 72.50% 145 33.08% 173Ninth 7.12% 23 - - 4.40% 23Tenth 2.79% 9 - - 1.72% 9Eleventh 8.98% 29 - - 5.54% 29Twelfth 2.48% 8 - - 1.53% 8

Gender Male 49.37% 157 46.43% 91 48.25% 248Female 50.63% 161 53.57% 105 51.75% 266

Ethnicity Caucasian 61.95% 197 71.65% 139 65.63% 336AfricanAmerican 3.14% 10 2.58% 5 2.93% 15Hispanic 27.04% 86 17.01% 33 23.24% 119Other 7.86% 25 8.76% 17 8.20% 42

Freeandreduced-pricelunch

FRL 37.58% 121 27.64% 55 33.78% 176Non-FRL 62.42% 201 72.36% 144 66.22% 345

Englishproficiency ELL 2.52% 8 2.59% 5 2.54% 13Non-ELL 97.48% 310 97.41% 188 97.46% 498

Specialeducation Specialed. 8.46% 27 4.10% 8 6.81% 35Non-SpecialEd.91.54% 292 95.90% 187 93.19% 479

Section504 Section504 2.82% 9 1.54% 3 2.33% 12Non-Sect.504 97.18% 310 98.46% 192 97.67% 502


60

Appendix E: Treatment and Comparison Teachers’ Science Resources Table E-1. Treatment Teacher Frequency of Resource Use

Always Frequently Occasionally Rarely Never % n % n % n % n % n

Sciencetextbookandsupplementalmaterials

- - 15.38% 6 33.33% 13 23.08% 9 28.21% 11

LessonsontheInternet - - 35.90% 14 38.46 15 23.08 9 2.56% 1

Lessonsyouhavecreated 15.38% 6 61.54% 24 20.51% 8 - - 2.56% 1

Lessonscreatedbycolleaguesinyourschoolordistrict

5.26% 2 44.74% 17 39.47% 15 5.26% 2 5.26% 2

FOSSkits 13.16% 5 10.53% 5 2.63% 1 2.63% 1 71.05% 27

GEMSguidesorkits - - - - 5.41% 2 8.11% 3 86.49% 32

Table E-2. Comparison Teacher Frequency of Resource Use


Sciencetextbookandsupplementalmaterials

8.82% 3 14.71% 5 26.47% 9 38.24% 13 11.79% 4

LessonsontheInternet - - 26.47% 9 52.94% 18 14.71% 5 5.88% 2

Lessonsyouhavecreated 23.53% 8 55.88% 19 17.65% 6 - - 2.94% 1

Lessonscreatedbycolleaguesinyourschoolordistrict

2.94% 1 26.47% 9 41.18% 14 17.65% 6 11.76% 4

FOSSkits 3.13% 1 9.38% 3 9.38% 3 - - 78.13% 25

GEMSguidesorkits 3.13% 1 - - 3.13% 1 3.13% 1 90.63% 29

STCkits - - 6.25% 2 6.25% 2 3.13% 1 84.38% 27


61

Appendix F: Treatment and Comparison Teachers’ Use of Assessment Table F-1. Treatment Teacher Frequency of Assessing Student Learning

Always Frequently Occasionally Rarely Never % % % n % n % n % n

Useaformativeassessmentatthebeginningofanewunitofstudy

15.38% 6 25.64% 10 41.03% 16 15.38% 6 2.56% 1

Useaformativeassessmentduringanewunitofstudy 25.64% 10 46.15% 18 25.64% 10 2.56% 1 - -

Collaboratewithcolleaguestoexaminestudentwork 5.13% 2 17.95% 7 41.03% 16 33.33% 13 2.56% 1

Table F-2. Comparison Teacher Frequency of Assessing Student Learning


Useaformativeassessmentatthebeginningofanewunitofstudy 8.82% 3 35.29% 12 35.29% 12 17.65% 6 2.94% 1

Useaformativeassessmentduringanewunitofstudy

17.65% 6 38.24% 13 35.29% 12 8.82% 3 - -

Collaboratewithcolleaguestoexaminestudentwork 11.76% 4 11.76% 4 32.35% 11 26.47% 9 17.65% 6


62

Appendix G: Energy and Matter Content Coverage Table G-1. Treatment and Comparison Teacher Energy Concepts Incorporated into Classroom

ConceptsTaught

Treatment Comparison % n % n

I.Whatisenergy?

Energyisameasureofchangeinasystem 41.67% 15 32.35% 11

Typesofenergy(kinetic,heat,light,sound,electrical,gravitationalpotentialenergy,chemicalpotentialenergy,etc.)

80.56% 29 50.00% 17

Energyisnotmatter;energyisnotaforce 47.22% 17 26.47% 9

Energytransfers(amountofenergypresentinasystemchangesortypeofenergychanges(e.g.,potentialtokinetic)

86.11% 31 67.65% 23

II.Potentialenergy

Whatispotentialenergy 83.33% 30 50.00% 17

Gravitationalpotentialenergy 66.67% 24 41.18% 14

Chemicalpotentialenergy 55.56% 20 14.71% 5

Measuringpotentialenergy 2.78% 1 14.71% 5

III.Heatenergy

Heatv.temperature 27.78% 10 32.35% 11

Changesinheatenergy 41.67% 15 35.29% 12

Conduction 41.67% 15 47.06% 16

Convection 50.00% 18 50.00% 17

Radiation 47.22% 17 44.12% 15

IV.Conservationofenergy

Lawofconservationofenergy 75.00% 27 52.94% 18

V.Energyinecosystems

Whatdoesfoodprovide? 75.00% 27 55.88% 19

Trophiclevels(producers,consumers) 44.44% 16 50.00% 17

Energyandphotosynthesis 58.33% 21 55.88% 19

Foodchainsandfoodwebs 47.22% 17 50.00% 17

Pyramidofenergy 38.89% 14 35.29% 12


63

ConceptsTaught


Periodofbiomass 11.11% 4 29.41% 10

Table G-2. Treatment and Comparison Teacher Counts of Matter Concepts Incorporated into Classroom

TopicsTaught


I.Whatismatter?

Whatisandisn’tmatter 61.11% 22 44.12% 15

Chemicalandphysicalpropertiesofmatter 47.22% 17 35.29% 12

Measuringmassandvolume 55.56% 20 52.94% 18

Calculatingdensity 50.00% 18 26.47% 9

II.Particlesofmatter

Natureofmatter(madeofparticlesandemptyspace)

50.00% 18 35.29% 12

Howparticlesmove 47.22% 17 35.29% 12

Intermolecularbonds 16.67% 6 14.71% 5

Compositionofmatter(elements,compounds,mixedsubstances)

47.22% 17 47.06% 16

Statesofmatter(solid,liquid,gas) 66.67% 24 58.82% 20

III.Changesinmatter

Conservationofmatter(atomsneithercreatednordestroyed)

52.78% 19 44.12% 15

Physicalchangesinmatter 50.00% 18 41.18% 14

Chemicalchangesinmatter 41.67% 15 29.41% 10

IV.Atomicstructure

Atomsdefined(smallestunitofelementthatretainspropertiesofthatelement)

47.22% 17 55.88% 19

Atomicmodel:protons,neutrons,andelectrons 52.78% 19 55.88% 19

Propertiesoftheperiodictableoftheelements 44.44% 16 41.18% 14

V.Matterinaction

Sinkingandfloating 36.11% 13 20.59% 7

Usingdensitytoexplainsinkingandfloating 47.22% 17 20.59% 7


64

Appendix H: E2MC Teacher Perceptions of Professional Development Sessions Teachersratedaspectsofthetwo-weeksummerinstituteandfollow-upprofessionaldevelopmentsessionsona5-pointscalerangingfrom1=poorto5=excellent.FigureD-1presentsthemeanratingsforeachprofessionaldevelopmentsession.

Figure D-1. Average teacher ratings of professional development sessions.

4.42

3.59

4.22

4.28

4.00

4.58

4.53

4.59

4.41

4.55

4.62

4.03

4.44

4.48

4.19

3.97

4.21

4.19

3.86

4.38

3.95

4.61

4.48

4.54

4.47

4.07

4.81

4.64

1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00

Overall PD Session

Format

Organization

Delivery

Pace

Opportunities for Participation

Materlals

Jun-14 Sep-14 Oct-14 Jan-15

Date post:	22-Feb-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

E2MC Evaluation report Final 8-10-15V-2 · grade levels with peers to plan and design lessons. They...

Documents