1 Milwaukee Mathematics Partnership Program Evaluation Year 4 Results Carl Hanssen Hanssen Consulting, LLC Cindy Walker University of Wisconsin-Milwaukee November 2007 MTL Meeting This material is based upon work supported by the National Science Foundation under Grant No. 0314898. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).
Transcript
1
Milwaukee Mathematics Partnership
Program EvaluationYear 4 Results
Carl HanssenHanssen Consulting, LLC
Cindy WalkerUniversity of Wisconsin-Milwaukee
November 2007 MTL Meeting
This material is based upon work supported by the National Science Foundation under Grant No. 0314898. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).
2
Evaluation Goals
Help the MMP better serve its constituents and improve its effectiveness
Serve the broader mathematics education community through documentation and dissemination of MMP activities
3
MMP Evaluation Logic ModelStudent
Achievement
Teacher Content& Pedagogical
Knowledge
Math FacultyInvolvement
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
NewCourses
DistrictBuy-in
MPA Ownership
MATCBuy-In
UWMBuy-In
ClassroomPractice
MMPActivities
ProximalOutcomes
DistalOutcomes
4
Presentation Overview
1. 2007 MMP Online Survey Results & Trends
2. MMP Impact on Student Achievement Gains
3. Detailed Work in 10 MPSSchools
5
1. 2007 MMP Online Survey Results & Trends
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
ClassroomPractice
6
Online Survey ResponsesAcademic Year
2004-05 2005-06 2006-07
Math Teacher Leader 124 140 143
Learning Team Member &
Mathematics Teacher167 284 335
LT Member (Administrative, other)
127 165 225
Math Teacher Only 676 1,340 1,388
Total 1,094 2,029 2,091
Responses are aggregated within a school so that each schoolcounts as 1 case in the analysis
7
Online Survey Variables
75+ Survey items 17 Composite Variables
Example Composite Variable:
Alignment How aligned a school’s curriculum is to standards and learning targets
Items I feel the mathematics program my school uses aligns with:
MPS learning targets.
Wisconsin state standards.
Goals of the Comprehensive Math Framework.
State/district assessments (WKCE/Terra Nova).
8
Context for these results
nStage 1.Learning Targets
Stage 2.Align
Targets
Stage 3.Designing
CABS
Stage 4.Examine
CABS
Stage 5.DescriptiveFeedback
Year 1, 2003-04
101 38% 53% 9% 0% 1%
Year 2, 2004-05
97 18% 34% 38% 5% 4%
Year 3, 2005-06
89 13% 26% 41% 18% 2%
Year 4, 2006-07
89 1% 9% 25% 43% 23%
These data show that there has beenprogress toward embracing MMP principles in schools
9
Statistically Significant Improvements
Quantity of PD
Consistency in math instruction
Engaging in activities to align curriculum to learning targets
Engaging in activities using
CABS and student work samples
Engaging in activities to
gauge student progress
Talking about teaching & learningMathematics with others
2.84 3.01
3.06
3.72
3.42
3.60
3.172.88
3.25
3.17
2.99
2.88
Spring 2006 Spring 2007
Eng
agem
ent
10
School Math Focus
Consistent curriculum
+
Teachers working together
+
PD perceived as valuable
PredictsStrongMathFocus
11
Supportive Learning Teams
Predicts
Perceptionof a supportiveLearningTeam
MTL perceived as supportive +Curriculum alignedto targets +Learning Team focuseson math +Teachers working together +PD perceived as valuable
12
Supportive MTL
Predicts
Perceptionof a supportiveMTL
PD perceived as valuable +MTS perceived as supportive +Teachers working together +Learning Team focuseson math -Curriculum aligned to targets
Schools with a supportive MTL likely aligned curriculum to targets last year
13
1. Conclusion
Across the district, schools are reporting higher levels of involvement with MMP
Similarly, schools report more frequently engaging in activities that the MMP encourages and promotes
14
2. MMP Impact on Student Achievement
StudentAchievement
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
ClassroomPractice
15
MMP Impact on 2006 Student Achievement
Are student achievement gains greater in schools that have more fully embraced MMP principles?
16
Sep 04
Sep 05
Sep 06
Sep 07
Sep 08
State TestFall 2004
MMP Online SurveySpring 2005
State TestFall 2005
State TestFall 2006
State TestFall 2007
MMP Online SurveySpring 2006
MMP Online SurveySpring 2007
MMP Online SurveySpring 2008
2004
-200
5S
cho
ol Y
ear
2005
-200
6S
cho
ol Y
ear
2006
-200
7S
cho
ol Y
ear
2007
-200
8S
cho
ol Y
ear
Data Collection Timeline
17
Analytical Approach
Use Student Achievement Data from 2005
+
MMP Online Survey Results from 2006
to explain variability in
Student Achievement Gains from 2005 to 2006
18
Sources of Variability in Student Achievement Scores
VariabilityIn Student
AchievementIn 2006
81%Student
19%School
12%MMP
Alignment
79%Other
52%Student
AchievementIn 2005
48%Other
4th Grade
9%LT
Quality
19
Sources of Variability in Student Achievement Scores
VariabilityIn Student
AchievementIn 2006
78%Student
22%School
9%MMP
Alignment
79%Other
56%Student
Achievement in 2005
44%Other
5th Grade
5%LT
Quality
20
Sources of Variability in Student Achievement Scores
VariabilityIn Student
Achievement in 2006
78%Student
22%School
4%MMP
Alignment
96%Other
50%Student
AchievementIn 2005
50%Other
6th Grade
21
Sources of Variability in Student Achievement Scores
VariabilityIn Student
AchievementIn 2006
76%Student
24%School
10%MMP
Alignment
90%Other
58%Student
Achievement in 2005
42%Other
7th Grade
22
Sources of Variability in Student Achievement Scores
VariabilityIn Student
AchievementIn 2006
79%Student
21%School
7%MMP
Alignment
93%Other
56%Student
Achievement in 2005
44%Other
8th Grade
23
2. Conclusion
Schools that more fully embrace MMP principles are more likely to show gains in student achievement
MMP influence is perhaps felt most strongly by students in the lower grades
24
3. Detailed Work in 10 MPS Schools
StudentAchievement
Teacher Content& Pedagogical
Knowledge
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
ClassroomPractice
Collaboration
25
Ten Case Study Schools
Diverse set of schools School Type
5 K-5 3 K-8 2 6-8
Geography 7 North 3 South
Median students = 430
26
Case Study Data Collection
20 learning team observations—2 in each school
40 classroom observations—4 in each school; 2 teachers observed 2 times each
MKT Assessment for math teachers
SNA Survey for mathteachers and administrators
27
Results of Learning Team Observations
Team Functioning
LeadershipParticipation
Organization/StructureResults
Overall Functioning
MMP Issues
Math Vision Consistency
Math LeadershipMMP Work
Overall MMP
StrengthsParticipation
Organization/Structure
Areas to ImproveMeeting Results
StrengthsMath Leadership
Vision, Consistency
Areas to ImproveMMP Work
28
Authoritarian
Directive leaderLittle discussionReporting out
Two Learning Team Models Emerging
Participatory
Active discussionConsensus building
Planning
Key Observation: to what degree are LTmeetings about learning versus school
administration?
29
Characteristics of High &Low Rated Learning Teams—Team Functioning
Focus on learning Distributed leadership Positional authority is
less important Multiple views are
represented and heard Multiple segments of the
school are represented Written agenda, note
taker, facilitator Explicit action items Participants have hi
knowledge and skill levels
Focus on administration Principal does all the
talking A few individuals
dominate the discussion No agenda or team is
easily distracted from the agenda
Little follow-through on assignments
No clear action items
High Low
30
Characteristics of High & Low RatedLearning Teams—MMP Issues
Consistent curriculum Math is addressed
alongside and in combination with other subjects
Coherent within grades and across grades
MTL clearly in charge with respect to math
Attention to CABS; reference to MMP courses; reviewing student work
Variation in curriculum
Math not addressed at the meeting
No clear math leader—i.e., hard to tell who the MTL is
Confusion about the MMP and CMF
High Low
31
Results of Classroom Observations
General Practice
Identify the Math TaskIs the Math Correct?
Formative Assessment
ComprehensiveMath Framework
UnderstandingComputingApplicationReasoning
Engagement
StrengthsIdentify the math taskCorrect Mathematics
Areas to ImproveFormative assessment*
StrengthsUnderstanding
Reasoning
Areas to ImproveApplication
Engagement
32
Characteristics of Strong & Weak Rated Classroom—General Practice
Math is correct Math task within the
lesson was easy to identify
Math task was discrete and level-appropriate
Encouraging self-assessment and peer-assessment
Establish criteria for proficiency
Promoting problem solving and independent thinking
Incorrect Math Math task was too
complex or obscure Only feedback provided
was if answer was correct
Little teacher involvement in the lesson
Feedback focuses on student behavior
Strong Weak
33
Characteristics of Strong & Weak Rated Classroom Performance—CMF
Student explanations sought
Computation is presented as a means to an end
Problem solving was emphasized
Students had to justify solutions
Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives
Close ended questions are emphasized
Only one way to solve problems presented
Minimal time allowed to share solutions
Students not accountable for responding to questions
Problems not presentedin context
Strong Weak
34
Results of MKT Assessment
Number &Operations
43 item assessment addressed 3 content areas:
AlgebraGeometry(2006-07Focus)
OverallScore
& &
35
Results of MKT Assessment
MKT scores can be interpreted like z-scores
Results were aggregated within schools
Number & Operations Algebra Geometry Overall
High 0.19 -0.33 0.75 0.57 Low -1.05 -1.27 -0.70 -0.93 Mean -0.55 -0.92 -0.14 -0.30 Median -0.66 -1.03 -0.34 -0.44 SD 0.35 0.33 0.50 0.43
There is tremendous variability in the results
Geometry was the MMP PD focus in 2006-07
36
Social Network Analysis
Teachers and administrators in each school were asked to name individuals with whom they communicated about mathematics
This is a key indicator of distributed leadership
37
Mathematics Distributed Leadership Continuum
High Low
Tight NetworkMTL CentralMany Links to MTLMTS InsideMany Links to MTS
Loose NetworkMTL Not Central
Few Links to MTLMTS Outside
Few Links to MTS
38
Low
School n Total Named Network density
Density in school
MTL Role--In Degree
MTS Role--In Degree
G 11 42 6.1% 7.5% 9.52 1.19 Sample Average 21.9 57.1 6.3% 12.2% 18.84 2.69 SD 8.0 16.7 2.6% 5.0% 6.9 3.7 Median 22 51 5.7% 11.4% 17.56 0.92
No single factor—e.g., distributed leadership, teacher MKT, learning team performance—is sufficient for success, but all may be necessary
Schools that are performing well do many of the things MMP promotes well, andrealize synergy between manyof these activities and principles
42
Overall Conclusions
There is support for the argument that schools that have more fully adopted MMP principles are demonstrating stronger outcomes—though there is still a lot of work to do.
MMP Impact, though, is not being felt in all schools—thereis tremendous variability in MMPadoption and progress across the district
43
Overall Conclusions
Important considerations for sustaining MMP work
Creating Distributed Leadership in a school takes time—and communication is critical
Last year the Learning Team was perceived as the most important actor for improving mathematics teaching and learning.
This year, in schools that report high levels of math focus, that responsibilityseems to be dispersed throughout the school.
44
Overall Conclusions
Important considerations for sustaining MMP work
MTL role may be shifting from focal point to facilitator—we see a shift in the perception of who is responsible for helping the school focus on improving mathematics teaching and learning
MTS role may more importantthan ever—schools using the MTSappear further down the path
45
Focus Question
What message will you be taking back about…
Your ongoing work to improve math in your school?
Specific areas where your school canimprove its math education?
