Teacher Evaluation in Newark:Evaluator Training
August 21 or 22, 2013
1
/ 2
The Irreplaceables explores retention through the experience of the nation’s best teachers, who urban schools desperately need to keep.
Estimates of Irreplaceables percentage based on teachers with value-added or growth data; District A high performers: 21%; District B high performers: 20%; District C high performers: 20%; District D high performers: 18%; Student impact estimates calculated following the methodology of Hahnel and Jackson (2012). Source: District data from SY 2009-10 and SY 2010-11.
The “Irreplaceables” are teachers so successful that they are nearly impossible to replace.
Who Are the Irreplaceables?
/ 3
When an Irreplaceable leaves a low-performing school, the school is almost guaranteed to hire a less-effective replacement.
Estimates based on teachers with value-added or growth data; Low performing schools include schools in the lowest quintile of proficiency by school level; Percentage of high-performing potential replacements in all schools - District A: 12%; District B: 17%; District C: 15%; District D: 15%; Low-performing schools - District A: 12% ; District B: 10%; District C: 3%; District D: 9%. Source: District data from SY 2008-09 and SY 2009-10.
Likelihood of Replacing a High Performer with a Teacher of Similar Quality
When a great teacher leaves a school, it can take 11 hires to find one teacher of comparable quality.
/ 4
Yet most schools retain Irreplaceables and low performers at fairly similar rates.
Struggling teachers remain for too long: Most have more than 9 years
of experience and plan to stay for at least another 10 years.
School Retention Rates by Teacher Performance, 2009-10
Chart: School retention defined as teachers remaining at their school from one year to the next. Bottom statement: Median years of experience 9-10 years across districts; Percentage planning to stay more than 10 years 48-62% across districts. Source: District data from SY 2008-09 through SY 2010-11
/ 5
Low performers rarely improve significantly. Even three years later, most perform worse than the average first-year teacher.
Chart: Median percentile ranks by population scores in District C; Populations defined in 2007-08. Bottom statement: District A: 44% veterans less effective; District C: 39% veterans less effective. Source: District data from SY 2008-09 through SY 2010-11
Performance Comparison of New Teachers and Low Performers over Three Years
40 percent of teachers with 7+ years of experience are less effective than the average first-year teacher
/ 6
Low-Cost Retention Strategies for Irreplaceables
Low-cost retention strategies defined as those that influence planned school retention of Irreplaceables. Source: District and survey data.
There are simple, low-cost steps principals can take that double the time Irreplaceables plan to remain at their schools.
Top teachers who experience two or more of these retention strategies plan to keep teaching at their schools for nearly
twice as long (2-6 more years).
/ 7
However, Irreplaceables report receiving little recognition or attention at the school level – often on par with the lowest performers.
Source: District B data and survey data. Trends confirmed across districts.
Principals used 7 of 8 top retention strategies at similar rates
for high and low performers.
Teachers Reporting Recognition at School
The Framework for Effective Teaching is at the core of the evaluation system
8
Our goal today: Ensure all are ready to implement the teacher evaluation system as a tool for differentiated management
Today, we will: Reflect on SY12-13 and discuss what is changing
for SY13-14 View instruction and norm on the framework Conduct a deep dive into the student goal setting
process Discuss details of evaluation requirements and
prepare to implement as the school year begins
9
REFLECTIONS ON 2012-13 TEACHER EVALUATION PROCESS
10
Activity: Think-Pair-Share
Reflecting on implementation of teacher evaluation this past year, write down:
• 1 to 2 successes from this year
• 1 to 2 challenges from this year
Turn to your neighbor and share what you wrote down
11
Quick Quiz
What percentage of our teachers received an annual evaluation by the end of the year?
12
Completion rates for observations, mid-years, and annuals evaluations were strong
13 Confidential - Do Not Distribute
At least 1observation
(4/2012)
At least 1 observation
(7/1/13)
Met observation
requirements (7/1/13)
Mid-year Review (2/22/13)
Annual Evaluations
(7/2012)
Annual Evaluations (8/13/2013)
76%
93% 91%
72%77%
94%
Completion rates for observations and evaluations are consistently high across networks
14
Notes: Observation completion rates measured as percent of teachers with required number of observations;Central includes Office of Special Education, Early Childhood, Title I Office, Master Teachers, and anyone else evaluated by central office staff except staff on long term leave
Data current as of 7/1/2013
Brad (n = 961)
Mitch (n = 580)
Peter (n = 535)
Roger (n = 478)
Tiffany (n = 484)
Central (n = 108)
93%99% 97% 94% 95%
65%
92%98%
90% 93% 92%
56%
Annual EvaluationsObservations
Confidential - Do Not Distribute
Completion rates for annual evaluations and observations less consistent across schools
Number of schools and departments
Completion rate Annual Evaluations Observations
100% 35 28
95 – 99% 17 22
90 – 94% 11 9
80-89% 8 6
Less than 80% 6* 12*
15
52 schools have annual evaluation or completion rates above 95%
Note: Only 1 school has an annual evaluation completion rate below 80% and only 6 schools have observation completion rates below 80%
Data current as of 7/1/2013
Quick Quiz
What percentage of teachers in the EWPS pool did not receive a rating in 2012-13?
16
Teachers in the EWPS pool receive lower ratings compared to the district distribution
17
• 29% of teachers in the 2012-13 EWPS pool (and 21% of teachers projected 2013-14 EWPS) did not receive annual evaluations
12-13 EWPS Pool (n=153)
13-14 EWPS Pool (n=339, projected)
District
20% 13%4%
30%32%
16%
48%50%
69%
2% 5% 11% Highly EffectiveEffectivePartially EffectiveIneffective
Teachers on long-term leave have lower ratings compared to other teachers
18
• 11% of teachers on long-term leave have completed annual evaluations
• 23% have the required number of observations and 26% have been observed at least once
Long term leave (n = 57)
District (n = 4832)
Long term leave (n = 57)
District (n = 2969)
Observations Annual Evaluations
30%
4%17%
11%
15%
21%
50%69%
45%
68%
33%16%
10% 7% 4%
Highly EffectiveEffectivePartially EffectiveIneffective
Quick Quiz
What percentage of observations had a partially effective rating?
What percentage of annual reviews had a partially effective rating?
19
20
Highly Effective Effective Partially Effective Ineffective
7%
68%
21%
4%
11%
69%
16%
4%
ObservationsAnnual Evaluations
Observations and Annual Ratings Distribution, 2012-13
Quick Quiz
What was the percentage of teachers rated highly effective last year (2011-12)?
What was the percentage of teachers rated highly effective this year (2012-13)?
21
22
Compared to last year, % HE annuals decreased and % PE annuals increased
Highly Effective Effective Partially Effective Ineffective
17%
69%
10%
4%
11%
69%
16%
4%
2011-20122012-2013
Annual ratings vary across networks
23
Brad(n = 890)
Mitch(n = 574)
Peter(n = 521)
Roger(n = 447)
Tiffany(n = 461)
Central(n = 70)
3% 6% 4% 2% 3% 7%17%
20% 23%
9% 14% 1%
72% 62% 63%
75%75%
70%
9% 13% 10% 14% 8%21%
Highly EffectiveEffectivePartially EffectiveIneffective
Ratings vary even more across schools: Some schools still rated few teachers as partially effective or ineffective
24
Observations:> 27 schools had no ineffective observation ratings.> 3 schools had no partially effective or ineffective observation
ratings.> 34 schools assigned more than 80% of observation ratings
in the top two categories.
Annual Evaluations:> 28 schools had no ineffective annual ratings.> 4 schools had no partially effective or ineffective annual
ratings.> 37 schools assigned more than 80% of evaluation ratings in
the top two categories.
Quick Quiz
What competencies receive the greatest proportion of partially effective and ineffective ratings?
25
On Annuals, Competencies 2 and 4 have the highest share of PE and IE ratings
26
Compe
tenc
y 1
Compe
tenc
y 2
Compe
tenc
y 3
Compe
tenc
y 4
Compe
tenc
y 5
2% 3% 2% 3% 2%13%
20%13% 16% 11%
66%67%
65%70%
66%
19%10%
20%11%
21%
Highly EffectiveEffectivePartially Effective Ineffective
Ratings distribution is fairly consistent across different evaluator types (slightly higher for DCs)
27
Pri
nci
pa
l
Vic
e P
rin
cip
al
De
pa
rtm
en
t C
ha
ir
Pri
nci
pa
l
Vic
e P
rin
cip
al
De
pa
rtm
en
t C
ha
irObservations
(n = 4832)Annual Evaluations
(n = 2969)
5% 4% 3% 5% 3% 3%
24% 21% 19% 17% 17% 14%
65% 67% 71% 65% 69% 73%
6% 8% 7% 13% 10% 10%Highly Effective
Effective
Partially Effective
Ineffective
Ratings are slightly higher both from observations and mid-years to annual evaluations
28
Annual Evaluation
Highly Effective EffectivePartially Effective
Ineffective
Mid-Year
Highly Effective 71% 29% 0% 0%
Effective 3% 90% 8% 0%
Partially Effective 0% 21% 71% 8%
Ineffective 0% 3% 36% 62%
Note: Percentages calculated as percent of row total; red numbers indicate inflation, blue numbers indicate deflation
TRANSITION TO NETWORKS
29
The five competencies set the standard for teacher practice in NPS
30
Students sustain focus on a specific objective that moves them toward mastery.
Instructional strategies challenge all students and provide multiple pathways to mastery.
A learning-focused environment of shared high expectations promotes mastery.
Students show evidence of, and teacher monitors, growth.
The teacher demonstrates commitment to excellence and to the professional growth of his/her school and peers.
Framework language changes to increase clarity and user-friendliness
12-13 version• Some indicators used the
phrase “and/or”, particularly in PE or I ratings
• The distinction between performance levels in some indicators wasn’t very clear
• Indicator 3e did not include reference to the teacher modeling high expectations
13-14 version• Simplified to use either “and”
or “or” to make rating these indicators easier and more consistent
• Language was changed to make distinctions clearer
• Indicator 3e revised to include the expectation that the teacher is a model of high expectations
31
Framework language changes to increase clarity and user-friendliness
12-13 version• Highly Effective in indicator 3b
asked for students to demand persistence of each other.
13-14 version• This phrase in Highly Effective
was removed
• Competency 4 over-time indicators were revised to better align with the new student learning goals in the IPDP.
32
Most remaining content of the Framework did not change. But does require additional training to ensure consistency across evaluators.
• All/nearly all, most, some, few
• Calling out a teachers’ physical classroom space
• Explicitly listing professional standards
• Making an explicit description of attendance metrics in Competency 5
33
Let’s reflect on the changes to the framework and evaluation system this year
• Consider each competency and the changes (both in language and evidence collection) between 2012 and 2013• On your handout
• Note what these changes signify about the intent of the competency
• Note what these changes signify about the implementation of the competency
• Share your thoughts with your table mates• Be prepared to share your group’s thoughts with the larger group
about one competency
34
This year, there are many ways to collect evidence to evaluate and support teachers
35
Lesson Design & Focus
Rigor & Inclusiveness
Culture of Achievement
Student Progress Towards Mastery
Commitment to Personal &
Collective Excellence
Wha
t Can
Be
Obs
erve
d
Wha
t Can
Be
Seen
in
Artif
acts
Wha
t Can
Be
Seen
in
Qua
ntita
tive
Dat
a
BREAK
36
VIEWING AND RATING INSTRUCTION
37
Viewing instruction:
Using an observation template or your own method, transcribe what happens in the following teaching clip.
After you view the video, categorize your evidence and assign ratings on the NPS framework. Then, note your ratings on the flip charts at the front of the room.
38
Video linked here
Let’s review our ratings and discuss
39
Where are we aligned in ratings?
Where are there outliers? What evidence can you share for these outlier ratings?
What do we need to do to ensure we and our school leaders are aligned?
What feedback would you give this teacher? What is the highest-leverage thing she could do to improve?
LUNCH
40
ASSESSING STUDENT LEARNING IN TEACHER
EVALUATION
41
Student learning is already a core part of our framework
42
Main Takeaways1. Competency 4 defines how we approach the use the assessment of
student learning in teacher evaluation.
2. The Individual Professional Development Plan (IPDP) is the way we will document how a teacher addresses Competency 4.
3. The IPDP requires setting student learning goals and teacher goals related to those student learning goals.
4. The goal setting process in the IPDP documents what effective teachers already do.
5. The IPDP should serve as a communication tool for teachers and their administrators on goals.
6. The specificity of student goals should be determined by the strength of available tools and resources (e.g., curriculum, assessments).
7. The Corrective Action Plan (CAP) has the same process as the IPDP for setting student learning goals and a more rigorous process for setting and tracking teacher goals.
8. All IPDPs and CAPs will be entered into an online system.
9.
43
This year’s Individual Professional Development Plan (IPDP):
• Provides a tool for goal setting for teachers and administrators around both student learning goals and the teacher’s own development goals
• Leverages over-time indicators and Common Core planning in the goals teachers set for students and themselves
• Is a tool for teachers and administrators to communicate about goals and growth areas throughout the year
44
The IPDP content includes:
Student Learning Plan:• Areas of Focus• Student End Points• Student Starting Points• Instructional Tools and Resources
Professional Growth Plan:• Growth Areas• Action Steps for Teachers
45
Let’s review a sample IPDP form together
The IPDP form is created in the beginning of the year and examined at conferences throughout the year
46
Goal-Setting Conference
Annual Conference
Mid-Year Conference
Observations and Conferences
Observations and Conferences
The final assessment of progress towards goals happens at the annual conference
Specifically, evaluators reflect their assessment of whether or not the teacher met his or her learning and professional development goals in the over-time indicators in Competency 4:
• 4d. Using Data: Teacher tracks assessment data to understand each student’s progress toward mastery and uses results to guide planning and instruction
• 4e. Understanding of Growth: Teacher can articulate specifically (and with evidence) whether or not each student has internalized grade-level standards and, if not, what s/he still needs to learn.
• 4f. Progress Toward Goals: Data reflect that students are mastering the objectives of the focus areas, leading toward mastery of grade-level standards.
47
Teachers rated PE or I at the end of SY12-13 will have a Corrective Action Plan (CAP), which replaces their IPDP
48
How is the CAP different from the IPDP?
CAP is more robust to ensure that:- Struggling teachers are getting the support they need and
-The district is collecting sufficient evidence to support tenure charges if necessary
The CAP includes 2 extra steps in setting professional development goals:1) Establishing metrics & processes to monitor progress
2) Articulating the steps administrators will take to support teachers’ development
There are several follow-up steps required by state lawAdditional observations
Mid-year conference
Observations by multiple observers
TRANSITION
49
IMPLEMENTING TEACHER EVALUATION: 2013-14
50
51
Goal-Setting Conference
Annual Conference
Mid-Year Conference
Observations and Conferences
Observations and Conferences
GOAL-SETTING CONFERENCE
52
Goal-Setting Conference at Beginning of Year
• Set student learning and professional development goals
• Finalize IPDPs or CAPs for all teachers
Goal-Setting Conference
53
By 9/15 for teachers on a CAP; by 9/30 for all others
Evaluators’ first focus should be on creating a CAP for teachers rated PE or I last year
CAPs must be on file for all relevant teachers by September 15.
54
In addition to the requirements mentioned earlier, a strong CAP is:
Very explicit in indicators and competencies that are areas of focus for the teacher
Very clear on the plan for development, including the role of both the teacher and the administrator
Is wholly co-developed by evaluator and teacher.
Some evaluators submitted CAP as part of the Annual in the Zoho system
55
25% of PE teachers and 50% of IE teachers have CAP already
These evaluators and teachers should re-visit their CAP when they input into the new BloomBoard system.
Partially Effective(n = 488)
Ineffective(n = 108)
121 54
367
54
Did not receive a CAPReceived a CAP
CLASSROOM OBSERVATIONS
56
Observations requirements have changed but reinforce best practices
Factor SY12-13 SY13-14
Length of Observations
Formal: Full classPPO: No guidance
Long: 40 minutes Short: 20 minutes
# of Observations 3 formal for non-tenured1 formal for tenuredUndefined PPOs
3 for non-tenured (length depends on years of experience)3 for tenured teachers (all short)
# of Observers One observer required per teacher Each non-tenured teacher and teacher with CAP must have at least two observers
Timing Just happen at some point in the school year
At least one observation must occur in each semester
Pre-Obs Conferences
Recommended for an announced observation
One observation each year must be announced with a pre-conf within 7 days before observation
Post-Obs Conferences
Formal post-observation required after formal observation within 10 days
Post-observation required for all observations within 15 days (for tenured Ts not on CAP, can be “informal”)
57
Quick Quiz
What percentage of teachers were observed by more than one evaluator?
58
Consideration: The use of multiple observers
59
1 2 3
90%
9%1%
Number of unique observers in 2012-13(excludes Peer Validators)
13-14SY Requirement: All non-tenured teachers and a teacher with a CAP must be observed by more than evaluator Note: A co-observation counts toward this requirement
However, only 10% of teachers were observed by more than one evaluator in 2012-13
Bottom line – This is a shift from evaluator practice last year.
Quick Quiz
How many observations were announced this year?
60
Consideration: Pre-observation conferences
61
13-14SY Requirement: Each year, every teacher must have at least one announced visit with a pre-observation conference within 7 days before the observation
In 12-13SY:70% of observations were announced and77% of those announced observations included a pre-observation
conferenceBottom Line – Not a big shift from previous evaluator
practice.
Consideration: Post-conference timing
62
Less than 10 days
10 to 15 days 15 to 20 days More than 20 days
72%
13%5% 10%
Percent of observations that take place within a given time-frame
13-14SY Requirement: Post-Conferences must occur within 15 days of any observation Note: But for tenured teachers not on a CAP, these can be “informal” post-
conference through the BloomBoard system
In 12-13SY, 88% of post-obs conference occurred within 15 days
Bottom line – Not a big shift from last past practice.
Consideration: Timing of observations
63
41166
41178
41190
41199
41209
41225
41234
41246
41256
41271
41285
41298
41310
41320
41332
41342
41353
41372
41382
41394
41403
41415
41426
414370
10
20
30
40
50
60
70
80
90
100
Nu
mb
er
of
ob
se
rva
tio
ns
Start of the 3rd marking period
13-14SY Requirement: At least one observation must occur in each semester
In 12-13SY, the majority of observations took place in the first semester
SCORING
64
Scoring remains the same in SY2013-2014
65
4 points 4 points 4 points 4 points 1, 0, -2 or -6
The evaluation rating is determined based on the teacher’s total score on all 5 competencies out of 17 points
Based on a preponderance of evidence, evaluators: • Assign a rating of Highly Effective, Effective, Partially Effective or
Ineffective on Competencies 1-4• Assign a rating of Exceeds Expectations, Meets Expectations, Slightly
Below Expectations, or Significantly Below Expectations on Competency 5
However, scoring will be slightly different for short vs. long observations
Short ObservationsEvaluators will report:• Ratings at the indicator and
competency level for which they have sufficient evidence
• An overall rating for the observation based on preponderance of evidence No formula to determine full rating
Long ObservationsEvaluators will report:• Ratings at the indicator level
for which they have sufficient evidence
• Ratings required for each competency based on preponderance of evidence
• Rating is determined by adding up competency ratings
66
MID-YEAR REVIEWS
67
The Mid-Year Conference is a chance to review evidence and assess progress towards goals
• Set student learning and professional development goals
• Finalize forms, including CAPs for applicable teachers
Goal-Setting Conference
• Review evidence of all indicators and assess progress towards goals
• Adjust approach to goals to move toward meeting goals
• Provide a rating based on evidence collected so far
Mid-Year Conference
68
By 9/15 for teachers on a CAP; by 9/30 for all others
By 2/15 for all teachers
ANNUAL REVIEWS
69
At the Annual Conference, evaluators review evidence and assign ratings to Competencies
• Set student learning and professional development goals
• Finalize forms, including CAPs for applicable teachers
Goal-Setting Conference
• Review evidence of all indicators and assess progress towards goals
• Adjust approach to goals to move toward meeting goals
• Provide a rating based on evidence collected so far
Mid-Year Conference
• Assess all evidence, including whether or not student learning goals were met
• Rate each indicator and competency based on all evidence to determine final evaluation score
Annual Conference and Evaluation
70
By 9/15 for teachers on a CAP; by 9/30 for all others
By 2/15 for all teachers
By 4/15 for non-tenured; 5/15 for tenured teachers on CAPs; and 6/15 for all others
Quick Quiz
How many annual evaluations were completed on time?
71
Consideration: Deadlines for Annuals
72
Deadlines will stay the same (with the new addition of the May 15th deadline for tenured teachers on CAPS)
74% of annual evaluations are completed on time (for April 15th and June 15th deadlines) 100% on time: American History Program, Fourteenth Ave., Ridge St.,
Roberto Clemente, Samuel L. Berliner, Science High
However, evaluations completed on time received higher ratings:
Highly Effective Effective Partially Effective Ineffective
12%
70%
15%4%
8%
67%
21%
3%
On time
Not on time
Summary of Deadlines
Item Deadline
CAPs finalized with student learning goals at beginning of year conference for all applicable teachers
September 15, 2013
IPDPs finalized with student learning goals at beginning of year conferences for all other teachers
September 30, 2013
Observations At least one in the first semester
Mid-Year Reviews complete for all teachers February 15, 2014
Observations At least one in the second semester
Annual Reviews complete for non-tenured teachers (whether or not on CAPs)
April 15, 2014
Annual Reviews complete for tenured teachers on CAPs May 15, 2014
Annual Reviews complete for all other tenured teachers June 15, 2014
73
Supports for implementing the evaluation system
B
l
o
o
m
B
o
a
r
d
:
S
o
p
h
i
s
ti
c
a
t
e
d
d
a
t
a
c
o
ll
e
c
ti
o
n
E
a
rl
y
S
e
p
t
e
m
b
e
r
Teacher Evaluation Guidebook: Clear policies and proceduresEarly September
Instructional Resource Center and BloomBoard: Easy access to supplemental resourcesNow
Peer Validation: Additional support for evaluators and teachersOctober
Successful Implementation
74
School Improvement Panels (SIP)
75
Membership: Each SIP must include the school’s principal, vice principal, and a
teacher. SIPs may have more members, but at least 1/3 of the SIP’s members
must be teachers.
The deadline for finalizing SIP membership is August 31, 2013.
Function: Oversee the mentoring and evaluations of teachers, conducting a mid-
year evaluation of a teacher who may receive an IE or PE rating at the Annual
Identify professional development opportunities for instructional staff members that are tailored to meet the unique needs of the students and staff of the school.
Request Peer Validators
In summary, while there are some changes, the core elements of evaluation remain the same,
What stays the same• Framework at the core of
every teacher’s observation
• Multiple observations for each teacher allow ample opportunities to collect evidence
• Mid-year and end of year review conferences to check in on goals
• Online data entry for real-time reporting and monitoring
What changes• A beginning-of-year conference to
start the year focused on growth and development
• Better, more detailed tools with some new tools for structured goal setting of both adult and student learning goals
• Requirements for the # of observations captured in the online system
• A more sophisticated data system providing better tools and resources
76
REFLECTION AND WRAP UP
77
Who are the Irreplaceable teachers in your building?
• How do you know?
• What strategies could you employ to support those teachers?
• How do you help your other teachers become Irreplaceable?
• How can implementing the Framework for Effective Teaching support you in identifying, developing and retaining Irreplaceable teachers?
78
Next Steps: Your critical role in ensuring your teachers understand the evaluation system this year.
79
Date Activity
First week of September Teacher Evaluation Guidebook distributed to all teachers and staff
Early September Online trainings on using the BloomBoard data system
By mid-September Present training information to teachers in a one-hour meeting (potentially with A Supt, SATQ)
SURVEY
80
Please complete the exit survey and drop it on the table in the back of the room before you leave.