1 Milwaukee Mathematics Partnership Program Evaluation Year 4 Results Carl Hanssen Hanssen...

transcript

Milwaukee Mathematics Partnership

Program EvaluationYear 4 Results

Carl HanssenHanssen Consulting, LLC

Cindy WalkerUniversity of Wisconsin-Milwaukee

November 2007 MTL Meeting

This material is based upon work supported by the National Science Foundation under Grant No. 0314898. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Evaluation Goals

Help the MMP better serve its constituents and improve its effectiveness

Serve the broader mathematics education community through documentation and dissemination of MMP activities

MMP Evaluation Logic ModelStudent

Achievement

Teacher Content& Pedagogical

Knowledge

Math FacultyInvolvement

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

NewCourses

DistrictBuy-in

MPA Ownership

MATCBuy-In

UWMBuy-In

ClassroomPractice

MMPActivities

ProximalOutcomes

DistalOutcomes

Presentation Overview

1. 2007 MMP Online Survey Results & Trends

2. MMP Impact on Student Achievement Gains

3. Detailed Work in 10 MPSSchools

1. 2007 MMP Online Survey Results & Trends

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

Online Survey ResponsesAcademic Year

2004-05 2005-06 2006-07

Math Teacher Leader 124 140 143

Learning Team Member &

Mathematics Teacher167 284 335

LT Member (Administrative, other)

127 165 225

Math Teacher Only 676 1,340 1,388

Total 1,094 2,029 2,091

Responses are aggregated within a school so that each schoolcounts as 1 case in the analysis

Online Survey Variables

75+ Survey items 17 Composite Variables

Example Composite Variable:

Alignment How aligned a school’s curriculum is to standards and learning targets

Items I feel the mathematics program my school uses aligns with:

MPS learning targets.

Wisconsin state standards.

Goals of the Comprehensive Math Framework.

State/district assessments (WKCE/Terra Nova).

Context for these results

nStage 1.Learning Targets

Stage 2.Align

Targets

Stage 3.Designing

Stage 4.Examine

Stage 5.DescriptiveFeedback

Year 1, 2003-04

101 38% 53% 9% 0% 1%

Year 2, 2004-05

97 18% 34% 38% 5% 4%

Year 3, 2005-06

89 13% 26% 41% 18% 2%

Year 4, 2006-07

89 1% 9% 25% 43% 23%

These data show that there has beenprogress toward embracing MMP principles in schools

Statistically Significant Improvements

Quantity of PD

Consistency in math instruction

Engaging in activities to align curriculum to learning targets

Engaging in activities using

CABS and student work samples

Engaging in activities to

gauge student progress

Talking about teaching & learningMathematics with others

2.84 3.01

3.172.88

Spring 2006 Spring 2007

School Math Focus

Consistent curriculum

Teachers working together

PD perceived as valuable

PredictsStrongMathFocus

Supportive Learning Teams

Predicts

Perceptionof a supportiveLearningTeam

MTL perceived as supportive +Curriculum alignedto targets +Learning Team focuseson math +Teachers working together +PD perceived as valuable

Supportive MTL

Predicts

Perceptionof a supportiveMTL

PD perceived as valuable +MTS perceived as supportive +Teachers working together +Learning Team focuseson math -Curriculum aligned to targets

Schools with a supportive MTL likely aligned curriculum to targets last year

1. Conclusion

Across the district, schools are reporting higher levels of involvement with MMP

Similarly, schools report more frequently engaging in activities that the MMP encourages and promotes

2. MMP Impact on Student Achievement

StudentAchievement

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

MMP Impact on 2006 Student Achievement

Are student achievement gains greater in schools that have more fully embraced MMP principles?

Sep 04

Sep 05

Sep 06

Sep 07

Sep 08

State TestFall 2004

MMP Online SurveySpring 2005

State TestFall 2005

State TestFall 2006

State TestFall 2007

Data Collection Timeline

Analytical Approach

Use Student Achievement Data from 2005

MMP Online Survey Results from 2006

to explain variability in

Student Achievement Gains from 2005 to 2006

Sources of Variability in Student Achievement Scores

VariabilityIn Student

AchievementIn 2006

81%Student

19%School

12%MMP

Alignment

79%Other

52%Student

AchievementIn 2005

48%Other

4th Grade

Quality

AchievementIn 2006

78%Student

22%School

Alignment

79%Other

56%Student

Achievement in 2005

44%Other

5th Grade

Quality

Achievement in 2006

78%Student

22%School

Alignment

96%Other

50%Student

AchievementIn 2005

50%Other

6th Grade

AchievementIn 2006

76%Student

24%School

10%MMP

Alignment

90%Other

58%Student

Achievement in 2005

42%Other

7th Grade

AchievementIn 2006

79%Student

21%School

Alignment

93%Other

56%Student

Achievement in 2005

44%Other

8th Grade

2. Conclusion

Schools that more fully embrace MMP principles are more likely to show gains in student achievement

MMP influence is perhaps felt most strongly by students in the lower grades

3. Detailed Work in 10 MPS Schools

StudentAchievement

Teacher Content& Pedagogical

Knowledge

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

Collaboration

Ten Case Study Schools

Diverse set of schools School Type

5 K-5 3 K-8 2 6-8

Geography 7 North 3 South

Median students = 430

Case Study Data Collection

20 learning team observations—2 in each school

40 classroom observations—4 in each school; 2 teachers observed 2 times each

MKT Assessment for math teachers

SNA Survey for mathteachers and administrators

Results of Learning Team Observations

Team Functioning

LeadershipParticipation

Organization/StructureResults

Overall Functioning

MMP Issues

Math Vision Consistency

Math LeadershipMMP Work

Overall MMP

StrengthsParticipation

Organization/Structure

Areas to ImproveMeeting Results

StrengthsMath Leadership

Vision, Consistency

Areas to ImproveMMP Work

Authoritarian

Directive leaderLittle discussionReporting out

Two Learning Team Models Emerging

Participatory

Active discussionConsensus building

Planning

Key Observation: to what degree are LTmeetings about learning versus school

administration?

Characteristics of High &Low Rated Learning Teams—Team Functioning

Focus on learning Distributed leadership Positional authority is

less important Multiple views are

represented and heard Multiple segments of the

school are represented Written agenda, note

taker, facilitator Explicit action items Participants have hi

knowledge and skill levels

Focus on administration Principal does all the

talking A few individuals

dominate the discussion No agenda or team is

easily distracted from the agenda

Little follow-through on assignments

No clear action items

High Low

Characteristics of High & Low RatedLearning Teams—MMP Issues

Consistent curriculum Math is addressed

alongside and in combination with other subjects

Coherent within grades and across grades

MTL clearly in charge with respect to math

Attention to CABS; reference to MMP courses; reviewing student work

Variation in curriculum

Math not addressed at the meeting

No clear math leader—i.e., hard to tell who the MTL is

Confusion about the MMP and CMF

High Low

Results of Classroom Observations

General Practice

Identify the Math TaskIs the Math Correct?

Formative Assessment

ComprehensiveMath Framework

UnderstandingComputingApplicationReasoning

Engagement

StrengthsIdentify the math taskCorrect Mathematics

Areas to ImproveFormative assessment*

StrengthsUnderstanding

Reasoning

Areas to ImproveApplication

Engagement

Characteristics of Strong & Weak Rated Classroom—General Practice

Math is correct Math task within the

lesson was easy to identify

Math task was discrete and level-appropriate

Encouraging self-assessment and peer-assessment

Establish criteria for proficiency

Promoting problem solving and independent thinking

Incorrect Math Math task was too

complex or obscure Only feedback provided

was if answer was correct

Little teacher involvement in the lesson

Feedback focuses on student behavior

Strong Weak

Characteristics of Strong & Weak Rated Classroom Performance—CMF

Student explanations sought

Computation is presented as a means to an end

Problem solving was emphasized

Students had to justify solutions

Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives

Close ended questions are emphasized

Only one way to solve problems presented

Minimal time allowed to share solutions

Students not accountable for responding to questions

Problems not presentedin context

Strong Weak

Results of MKT Assessment

Number &Operations

43 item assessment addressed 3 content areas:

AlgebraGeometry(2006-07Focus)

OverallScore

Results of MKT Assessment

MKT scores can be interpreted like z-scores

Results were aggregated within schools

Number & Operations Algebra Geometry Overall

High 0.19 -0.33 0.75 0.57 Low -1.05 -1.27 -0.70 -0.93 Mean -0.55 -0.92 -0.14 -0.30 Median -0.66 -1.03 -0.34 -0.44 SD 0.35 0.33 0.50 0.43

There is tremendous variability in the results

Geometry was the MMP PD focus in 2006-07

Social Network Analysis

Teachers and administrators in each school were asked to name individuals with whom they communicated about mathematics

This is a key indicator of distributed leadership

Mathematics Distributed Leadership Continuum

High Low

Tight NetworkMTL CentralMany Links to MTLMTS InsideMany Links to MTS

Loose NetworkMTL Not Central

Few Links to MTLMTS Outside

Few Links to MTS

School n Total Named Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

G 11 42 6.1% 7.5% 9.52 1.19 Sample Average 21.9 57.1 6.3% 12.2% 18.84 2.69 SD 8.0 16.7 2.6% 5.0% 6.9 3.7 Median 22 51 5.7% 11.4% 17.56 0.92

Student Achievement:2006: 20% Proficient4-year trend: -4%

Medium

Density in school

MTL Role--In Degree

MTS Role--In Degree

I 28.0 75.0 4.0% 12.2% 23.31 0.33

Average 21.9 57.1 6.3% 12.2% 18.84 2.69 SD 8.0 16.7 2.6% 5.0% 6.90 3.70 Median 22.0 51.0 5.7% 11.4% 17.56 0.92

Student Achievement:2006: 21% Proficient4-year trend: -19%

Density in school

MTL Role--In Degree

MTS Role--In Degree

A 22 43 11.7% 20.1% 30.61 4.40 Sample Average 21.9 57.1 6.3% 12.2% 18.84 2.69 SD 8.0 16.7 2.6% 5.0% 6.9 3.7 Median 22 51 5.7% 11.4% 17.56 0.92

Student Achievement:2006: 50% Proficient4-year trend: +7%

3. Conclusion

No single factor—e.g., distributed leadership, teacher MKT, learning team performance—is sufficient for success, but all may be necessary

Schools that are performing well do many of the things MMP promotes well, andrealize synergy between manyof these activities and principles

Overall Conclusions

There is support for the argument that schools that have more fully adopted MMP principles are demonstrating stronger outcomes—though there is still a lot of work to do.

MMP Impact, though, is not being felt in all schools—thereis tremendous variability in MMPadoption and progress across the district

Overall Conclusions

Important considerations for sustaining MMP work

Creating Distributed Leadership in a school takes time—and communication is critical

Last year the Learning Team was perceived as the most important actor for improving mathematics teaching and learning.

This year, in schools that report high levels of math focus, that responsibilityseems to be dispersed throughout the school.

Overall Conclusions

Important considerations for sustaining MMP work

MTL role may be shifting from focal point to facilitator—we see a shift in the perception of who is responsible for helping the school focus on improving mathematics teaching and learning

MTS role may more importantthan ever—schools using the MTSappear further down the path

Focus Question

What message will you be taking back about…

Your ongoing work to improve math in your school?

Specific areas where your school canimprove its math education?

1 Milwaukee Mathematics Partnership Program Evaluation Year 4 Results Carl Hanssen Hanssen...

Documents