1

Educator Evaluation:Challenges and Opportunities

Massachusetts Association of School Superintendents

May 19, 2011updated

2

Context

The three-year Superintendent Induction Program

An ESE – M.A.S.S. Partnership

Four Focus Areas

1. Focused Instructional Leadership2. Collaborative Relationships and Effective

Leadership Teams3. Strategic Management of Human and

other Resources4. Robust System of Supervision and

Evaluation

3

Agenda

Goals of the Proposed Regulations

Key Features of the Proposed Regulations

Components of the Model System Challenge: Assessing Educator Impact on

Student Growth using Multiple Measures

Challenge: Self-Assessment and Goal Setting

Challenge: Timeline for Implementation

4

Goals of Improved Educator Evaluation

• Promote leaders’ and teachers’ growth and development

• Place student learning at the center using multiple measures of student learning, growth and achievement

• Recognize excellence in teaching and leading

• Set a high bar for professional status (tenure)

• Shorten timelines for improvement

5

Key Features

• 5-step Evaluation Cycle, starting with educator self-reflection and goal setting

• 3 Categories of Evidence:

– Multiple measures of student learning, growth and achievement

– Products of practice – Educator’s collection of other evidence

6

Features, cont’d• 4 Standards with “core” indicators for

administrators and teachers

• 4 Ratings on performance: exemplary, proficient, needs improvement, unsatisfactory

• 3 Ratings of impact on student learning, with focus on learning gains: high, moderate, low

• Different Paths & Plans depending on career stage and performance

7

A 5-Step Evaluation Cycle

8

Rubrics for4 Statewide Standards and Indicators

Administrators Teachers

Curriculum, Instruction & Assessment

Management & Operations

Family & Community Partnerships

Professional Culture

Curriculum, Planning & Assessment

Teaching All Students

Family & Community Engagement

Professional Culture

9

3 Categories of Evidence

• Multiple measures of student learning, growth and achievement

• Products of practice, including observation of practice (announced and unannounced)

• Educator’s collection of other evidence,

including analysis of feedback from: – Students– Parents– Staff (for administrators)

10

The summative performance rating(exemplary, proficient, needs improvement,

unsatisfactory)

Based on: • Performance against each of four

standardsand • Progress toward meeting student learning

and professional practice goals

Educator’s impact on student learning, growth and achievement “counts” in standards on curriculum, instruction and assessment

11

Paths & Plans: Differentiated by Career Stage and

Performance• Educators in their first three years:

Development Plan (one year)• Performance rated as proficient or

exemplary: Self-directed Growth Plan (one or two years)

• Performance rated as in need of improvement:Directed Growth Plan (one year)

• Performance rated as unsatisfactory: Improvement Plan (up to one year)

12

Decision Flow for Experienced Educators

13

Rating of Educator Practice

Exemplary

Proficient

Needs Improvement

Unsatisfactory

Low Moderate High

Impact on Student Learning(multiple measures of student learning, including MCAS student growth percentiles where available, with a focus

on learning gains)

Linking Student Learning and Educator Practice

14

Low Rating of Educator Practice

BUT

Moderate or HighImpact on Student Learning• Evaluator

reviews discrepancy with educator.

• Evaluator’s supervisor considers discrepancy trends in evaluating evaluator.

What Happens When There’s a Discrepancy?

15

What Happens When There’s a Discrepancy?

High Rating of Educator Practice

BUT

Low Impact on Student Learning

• Educator has 1-year growth plan focused on discrepancy.

• Evaluator’s supervisor MUST review rating.

• Superintendent has final authority to determine summative rating.

16

Agenda

Goals of the Proposed Regulations

Key Features of the Proposed Regulations

Components of the Model System Challenge: Assessing Educator Impact on

Student Growth using Multiple Measures

Challenge: Self-Assessment and Goal Setting

Challenge: Timeline for Implementation

17

Key components of the Model planned

• Contract language describing:– Process– Timelines– Collection of evidence

• A rubric for each standard and indicator that:– describes professional practice vividly and

clearly at four levels of performance– Is differentiated for different roles, e.g.,

classroom teacher, caseload teacher, counselor, principal

– Includes core and supplementary indicators

18

19

Other components of the Model• Templates for:

– Self-assessments– Goals– Plans

• Developing educator plan• Self-directed growth plan• Directed growth plan• Improvement plan

20

Other components of the Model, con’t

• Guidelines for developing and using multiple measures of student learning, growth and achievement

• Guidelines for determining low, moderate and high impact on student learning

• Examples and Resources on:– Multiple measures of student learning– Determining educator impact

• Examples of ways to collect and use feedback from:– Students– Staff (for administrators)– Parent feedback (initially for administrators)

21

Stakeholder Feedbackfrom Students, Staff and Parents

• Focus on school-wide feedback (initially)

• Students, starting in grade 6 (?)

• A possibility: ESE-supported on-line data collection

22

Stakeholder Feedback Examples New York City Learning Environment Survey

•Garners annual feedback from parents, students and teachers.•Results factor into school progress report rating and help schools

better understand their own strengths and target areas for improvement. 

•http://schools.nyc.gov/Accountability/tools/survey/default.htm

Massachusetts Teaching, Learning and Leading Survey (Mass TeLLS)•Taken by 40,000 teachers and administrators in 2008.•Educators provided views about teaching and learning conditions,

including leadership, empowerment, facilities and resources, PD, and time, in their schools.

•http://www.masstells.org/index

Boston Student Advisory Council (BSAC) Student to Teacher Constructive Feedback•Students provide annual, anonymous feedback about individual

teachers •http://www.youtube.com/user/BSACbuzz#p/a/f/0/CMB_8DjAAgM

23

Supports planned for the Model

Annual updatesOrientation tools and resources for a

variety of audiencesOn-line and hybrid professional

development on observation, goal setting, etc.

Networks of Practice- - - - - - - - - - - - - - - - - - - Web-based rubric “library”

24

Supports for the Model System

Outreach to state associations, e.g., Principals (MESPA and MSSAA)Department Heads and Supervisors

(MASCD)Counselors (MASCA)ESL (MATSOL)Art (MAEA)

Training and support for regional collaboratives to develop and share expertise and resources among member districts

25

Agenda

Goals of the Proposed Regulations

Key Features of the Proposed Regulations

Components of the Model System Challenge: Assessing Educator Impact on

Student Growth using Multiple Measures

Challenge: Self-Assessment and Goal Setting

Challenge: Timeline for Implementation

26

What are “multiple measures”?

MCAS growth percentile data, when applicable

MEPA growth scores, when applicableOther assessments comparable district-

wide across grade or subject, including approved commercial assessments and district-developed pre/post unit and course assessments

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Teacher-developed assessments (individual

and/or team, school)

27

MCAS Growth to grade 7: Three students

230 230

200

220

240

260

280

Grade 5 Grade 6 Grade 7

MCAS

ELA

sca

led s

core

Advanced

Proficient

Needs Improvement

Warning/Failing

80 to 99

60 to 79

40 to 59

20 to 39

1 to 19

SGP

Gina

230

35%

65%

SGPs between 40 to 59 are typical

2006 2007 2008

source: www.doe.mass.edu/mcas/growth/

28

248 248

200

220

240

260

280

Grade 5 Grade 6 Grade 7

MCAS

ELA

sca

led s

core

Advanced

Proficient

Needs Improvement

Warning/Failing

Growth to grade 7: Three students

Harry

244

25%

75%

2006 2007 2008

source: www.doe.mass.edu/mcas/growth/

29

214 214

200

220

240

260

280

Grade 5 Grade 6 Grade 7

MCAS

ELA

sca

led s

core

Advanced

Proficient

Needs Improvement

Warning/Failing

Growth to grade 7: Three students

Ivy

2268%

92%

2006 2007 2008

30

248 248

230230

214214

200

220

240

260

280

Grade 5 Grade 6 Grade 7

MCAS

ELA

sca

led s

core

Advanced

Proficient

Needs Improvement

Warning/Failing

Growth to grade 7: Three students

Gina, Harry, and Ivy

Harry

Gina

Ivy

2006 2007 2008

source: www.doe.mass.edu/mcas/growth/

31

Growth to grade 7: Three students

92226214214Ivy

25244248248Harry

35230230230Gina

SGP2008

Grade 72008

Grade 62007

Grade 52006

English language arts

source: www.doe.mass.edu/mcas/growth/

32

Median student growth percentile

Last name SGP

Lennon 6

McCartney 12

Starr 21

Harrison 32

Jagger 34

Richards 47

Crosby 55

Stills 61

Nash 63

Young 74

Joplin 81

Hendrix 88

Jones 95

Imagine that the list of students to the left are all the students in your 6th grade class. Note that they are sorted from lowest to highest SGP.

The point where 50% of students have a higher SGP and 50% have a lower SGP is the median.Median SGP for the 6th grade

class

source: www.doe.mass.edu/mcas/growth/

33

Challenging a Level 4 School

Median Student Growth PercentileEnglish Language Arts

200820092010

Murkland 26.5 22.5 22.0Lincoln 55.0 66.5 68.0Sokolovsky 67.0 71.074.0

source: www.doe.mass.edu/sda/dart

34

A 5-Step Evaluation Cycle

35

Grade 5

Grade 4

Grade 7

Grade 8

Grade 10

Grade 6

DISTRICT

STATE

0% 20% 40% 60% 80% 100%

23%

22%

15%

17%

18%

13%

18%

20%

18%

19%

18%

19%

20%

14%

18%

20%

20%

20%

22%

20%

18%

18%

20%

20%

18%

18%

22%

20%

19%

21%

20%

20%

21%

21%

22%

23%

24%

34%

24%

20%

Lowell Public SchoolsDistrict Mathematics • 2009 Growth

SOURCE: Massachusetts Department of Elementary and Secondary Educa-tion

very low low moderate high very high

Research Testing and Assessment - Lowell Public Schools

35Lowell Public Schools

A Case Study

36

Bartlett

Robinson

Rogers

Butler

Sullivan

Stoklosa

Daley

Wang

Pyne

DISTRICT

STATE

0% 20% 40% 60% 80% 100%

30%

22%

24%

16%

7%

8%

8%

3%

2%

12%

20%

32%

26%

18%

24%

12%

9%

7%

7%

4%

14%

20%

14%

21%

21%

16%

21%

11%

20%

16%

13%

18%

20%

16%

15%

20%

26%

22%

27%

19%

21%

21%

21%

20%

8%

15%

17%

18%

39%

44%

47%

53%

60%

34%

20%

Lowell Public SchoolsGrade 6 Mathematics • 2009 Growth

SOURCE: Massachusetts Department of Elementary and Secondary Educa-tion

very low low moderate high very high

Research Testing and Assessment - Lowell Public Schools

36Lowell Public Schools

37

Lowell Public SchoolsGrade 6 Mathematics • 2008 Growth

SOURCE: Massachusetts Department of Elementary and Secondary Education

30%

29%

24%

4%

9%

7%

5%

4%

14%

20%

26%

27%

26%

13%

15%

13%

14%

17%

2%

18%

20%

22%

22%

21%

28%

21%

22%

19%

17%

11%

21%

20%

13%

16%

19%

22%

27%

25%

30%

21%

13%

21%

20%

9%

6%

10%

33%

28%

33%

32%

41%

74%

26%

20%

0% 20% 40% 60% 80% 100%

RogersRobinson

ButlerBartlett

WangSullivan

StoklosaDaleyPyne

DISTRICTSTATE

Research Testing and Assessment - Lowel l Publ ic Schools

very low low moderate high very high

38

Stoklosa - ELL

Other LPS - ELL

Stoklosa - Reg. Ed.

Other LPS - Reg. Ed

Stoklosa - ALL

Other LPS - ALL

0% 20% 40% 60% 80% 100%

12%

9%

5%

9%

8%

12%

6%

15%

13%

14%

9%

15%

12%

18%

8%

21%

11%

19%

20%

21%

26%

21%

25%

20%

51%

38%

47%

35%

46%

34%

Lowell Public SchoolsGrade 6 Mathematics • 2009 Growth

SOURCE: Massachusetts Department of Elementary and Secondary Educa-tion

very low low moderate high very high

Research Testing and Assessment - Lowell Public Schools

38Lowell Public Schools

39

Grade 8 - ELL

Grade 7 - ELL

Grade 6 - ELL

Grade 5 - ELL

Grade 8 - Regular

Grade 7 - Regular

Grade 6 - Regular

Grade 5 - Regular

0% 20% 40% 60% 80% 100%

6%

13%

12%

6%

15%

16%

5%

18%

26%

6%

3%

8%

24%

13%

3%

16%

24%

12%

12%

22%

30%

8%

19%

27%

21%

20%

21%

18%

19%

26%

32%

33%

16%

51%

59%

37%

10%

47%

46%

Lowell Public SchoolsStoklosa Middle School • 2009 Mathematics Growth

SOURCE: Massachusetts Department of Elementary and Secondary Educa-tion

very low low moderate high very high

Research Testing and Assessment - Lowell Public Schools

39Lowell Public Schools

40

Other measures that can help Stoklosa staff assess impact on

student learning and growth

Massachusetts English Proficiency Assessment (MEPA)

District math benchmark assessmentDistrict-approved commercial assessment

tied to district curriculumDistrict-adopted curriculum-embedded

performance assessment

41

Agenda

Goals of the Proposed Regulations

Key Features of the Proposed Regulations

Components of the Model System Challenge: Assessing Educator Impact on

Student Growth using Multiple Measures

Challenge: Self-Assessment and Goal Setting

Challenge: Timeline for Implementation

42

A 5-Step Evaluation Cycle

43

Self-assessment and Goal Setting

Based on: Standards & indicators (rubric) + district & school priorities An analysis of multiple measures of learning and growth of

our students in the past An analysis of the students we have now

At least: One goal for professional practice One goal for student learning, growth and achievement

Attributes of a Useful Goal

Specific Measurable Attainable

Relevant Time-bound

44

SMART goals of Stoklosa’s 7th grade team

• Professional Practice goal: I/We will…

• Student Learning goal: My/Our students will…

45

SMART goals of a middle school music teacher

• Professional Practice goal: I will collaborate with my colleagues in the music department to develop, pilot, analyze, revise and share 2 performance-based assessments

• Student Learning goal: My students will be able to identify and apply music terms, symbols and definitions in the curriculum guide for 6th, 7th and 8th grade. Using a department-developed assessment, 75% of my students will score 85% or above on the third quarter assessment.

46

SMART Goals of an 8th grade social studies teacher

• Professional practice goal: To strengthen expository writing, I will study the “workshop process” for writing, observe it in practice, and introduce it in at least two of my classes by the start of second term.

• Student learning goal: At the end of the third quarter unit on the constitution, students will demonstrate proficiency by writing a pamphlet for new citizens about their constitutional rights. Using a department-developed rubric, a majority of my students will have moved one level on the writing component of the rubric since the start of the second term

47

SMART Goals of a Middle School Principal

• Professional practice goal: I will complete 100% of goal setting conferences with my fifth and sixth grade teams by October 15th, seek anonymous feedback about staff perceptions of their usefulness in improving their practice, research effective goal setting with my colleagues, and identify steps I will take in mid-year formative assessment conferences to improve the likelihood that their practice and student growth goals will be achieved.

• Student learning goal: The proportion of fifth and sixth grade students with high or very high SGP growth will increase by 5% points in both ELA and Math.

48

SMART Goals of a 10th grade geometry teacher

• Professional practice goal: To engage students more, starting second term, we will incorporate at least one real-world application of geometry into 2 of every 5 homework assignments.

• Student learning goal: 85% of our students will score 80% or above on the district-developed third quarter exam.

49

Agenda

Goals of the Proposed Regulations

Key Features of the Proposed Regulations

Components of the Model System Challenge: Assessing Educator Impact on

Student Growth using Multiple Measures

Challenge: Self-Assessment and Goal Setting

Challenge: Timeline for Implementation

50

Anticipated timeline for Implementation

For 2011-2012• Level 4 schools + volunteer “early

adopters”

For 2012-13• All Race to the Top Districts

For 2013-2014• All other Districts

51

Educator Impact on Student Learning

603 CMR 35.10 (4)

By September 2013, each district shall adopt a district-wide set of student performance measures for each grade and subject that permits a comparison of student learning gains.– MCAS Student Growth Percentile shall be

employed where it is available.– At least two measures of student learning

gains shall be employed at each grade and subject in determining impact on student learning.

52

Priorities for ESE action and support?

1. Model contract language and rubrics for teacher and principal

2. Strategies for “making time” to do evaluation well3. Orientation materials for many audiences4. Strategies for using the rubric to develop a shared,

specific picture of practice at four levels of proficiency5. Access to low-cost PD for evaluators to use the rubric

effectively6. Guidelines and examples: Self-Assessment and Goals7. Guidelines and examples: Educator Plans8. Developing District-wide Measures of Student Learning*9. Determining Educator Impact on Student Learning*10. Student, Staff and Parent Feedback*

* Will take more than one year to develop, pilot and validate

53

This summer, I’d: Begin (or deepen) work on building educator’s capacity to

analyze data about student learning and set SMART goals. Work with my principals on their SMART goals and establish

clear expectations for what I want to be seeing when I do my first visits to their schools this fall and observe practice with them.

Begin to line up potential partners and supports; for example, I’d ask my collaborative if it will work with my district and other member districts on implementation.

Introduce the first draft of ESE’s model rubric (available mid-July, hopefully) to see how well it might match my district’s needs.

Attend the M.A.S.S. Summer Institute to get the latest information from ESE on the status of the regulations and the model; and to confer with colleagues

If I were in your shoes….

54

• I would not start bargaining now. There isn’t enough information to go on yet.

• I would, however, let my committee and union know that we will have to open the contract to bargain this.

If I were in your shoes….(con’t)

55

Questions? Suggestions?

Priorities?Please complete the feedback

sheetKarla Brooks Baehr

Deputy Commissionerkbaehr@doe.mass.edu

781-338-3101