Models for Evaluating MSP ProjectsWorkshop for State Leaders and Evaluators of
Teacher Professional DevelopmentWashington, DC
April 24, 2008
Norman L. WebbSenior Research Scientist
Wisconsin Center for Education Research
University of WisconsinMadison, Wisconsin
Program StagesPlanningImplementationImpact
First Steps in Initiating an Evaluation
Finding an evaluatorExternal/internalIndependent/within program
Cost for evaluationScope for evaluation
Evaluation for Program Planning
Compatibility between Program Design and Evaluation Design
No causation without manipulation
Experimental and Quasi-Experimental Designs
Pre-Post TestPre-Post Test with a Control GroupPost Test OnlyPost Test with a Control GroupRandom AssignmentMultiple Levels
Teacher – StudentSchool – Student
Trend Analysis—Baseline, Periodic Assessments
Professional Development Program DesignSelection of Teachers
AllVolunteersPurposeful
Selection of SchoolsAllVolunteersPurposefulStratified
Evaluation of Program Theory
Research BasedLogic
Simon, 1997 Mathematics Teaching Cycle
Logic Model Framework for Listing Program Activities
Professional Development Activities
F6)F5)F4)MTLs monthly two-day workshops on leadership
E6)Teachers -bi-weekly grade level meetings
E5)E4)
D6)Teachers -one-week summer institute
D5)Teachers -after school weekly classes
D4)MTLs support other teachers
ACTIVITIES
Curriculum and Instruction
Professional Development Interim Outcomes
L6)Teachers learn to manage reflective teaching classroom
L5)Teachers evaluate students’understanding based on viable model of how mathematical understanding develops
L4)
K6)Teachers make instructional and curriculum decisions based on students’understanding
K5)Teachers gain skills at recognizing student mathematical thinking and experiment with techniques
K4)
J6)J5)Teachers build deeper understanding of content and student thinking
J4)MTLs gain leadership skills and recognize teachers’ growth in reflective teaching
I6)Teachers independently use reflective teaching strategies
I5)Teachers build confidence in reflective teaching of mathematics
I4)MTLs’ beliefs, attitudes, and practices become aligned with reflective teaching
INTERIM
Curriculum and Instruction
Program Implementation and Evaluation
Guskey’s Five Critical Levels of Evaluating Professional Development
1. Participants’ Reactions, 2. Participants’ Learning, 3. Organization Support for Change, 4. Participants’ Use of New Knowledge and
Skills5. Student Learning Outcomes.
Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press. As reported by Monsaas and McGee-Brown, Evaluating Professional Learning Communities Using Mixed, Quantitative and Qualitative Methodology, MSP Evaluation Summit, September 2005. CD available from Consortium for Building Evaluation Capacity, Utah State University.
Some Issues Related to Evaluating the Implementation of Professional Development
Time FrameProof-of-Concept StudiesBreadth vs. DepthFormative Studies Aligned Measurement InstrumentsLiving with ImperfectionAnalytic Horizon Mismatch with Funding
Frequently Used Methods in Evaluation of Professional Development Programs
Case StudiesExperimental DesignsMulti-Variant DesignsTrend Analyses
Frequently Used Tools in Evaluating Professional Development Programs
SurveysObservationsInterviewsFocus GroupsTeacher LogsContent Knowledge Tests
Web Sites with Evaluation Instruments
www.consortium-chicago.orghttp://ed.uwn.edu/CAREI/cetphttp://[email protected]://www.addingvalue.org/
Less Frequently Used Tools
Content Coverage MapsConcept MapsStudent Work Samples
Program Impact
Math Grade 5
-10
-5
0
5
10
15
AlACiMa Cohort I Schools Other schools
Q15%median95%Q3
Value-Added Effect on Mathematics Scores from Grade 4 to Grade 5, 2003-2005
Spanish Grade 5
-10
-5
0
5
10
15
AlACiMa Cohort I Schools Other Schools
Q15%median95%Q3
Value-Added Effect on Spanish Scores from Grade 4 to Grade 5, 2004-2005
0.90-0.120.58-0.07Spanish
0.061.860.681.26Mathematics
Pr > │t │t ValueStandard ErrorAlACiMaParameter Estimate
Grade 5Subject Matter
Value Added Effect of AlAciMaSchools for Grade 5 by Content Area for Cohort 1 2004-2005
600
620
640
660
680
700
720
740
760
2000 2001 2002 2003 2004 2005
Year
Scal
e Sc
ore
4th8th10th
MMSD Mean Math Scale Scores for Grades 4, 8, and 10 2000-2005
0
100
200
300
400
500
600
700
800
900
1000
2000 2001 2002 2003 2004 2005
Year
Scal
e Sc
ore
MMSD Math Mean, First and Third quartile, and 5th and 95th Percentile Scores for Grade 8 2000-2005
n = 1761
n = 1726
n = 1729n = 1697
n = 1666n = 1706
-0.1
0
0.1
0.2
0.3
0.4
2000 2001 2002 2003 2004 2005
Year
Effe
ct S
ize*
Normalized Scale Score Effect Size Between MMSD and WI for Math Grade 8 2000-2005
Evaluators Need to be Realistic
Program design will change(27 of 40 evaluators, 68%)
Collecting student data is challengingAccess to school, teachers, and students can be difficultThere is a trade off between collecting data and writing about resultsQualitative and quantitative data collections requires a balance
What Has Worked for MSP Evaluators
Continual Communication with StakeholdersConducting Case StudiesAnalysis of Teacher-Learning Logs a Value to Stakeholders
Final Thoughts
It is not the direction of the wind, but the set of the sails that guides the boat.