Linking a Comprehensive Professional Development

Literacy Program to Student Achievement

Edmonds School District

WERA December 4, 2008

Presenters

• Nancy Katims, Director of Assessment, Research, and Evaluation

• N. Lynn Caulkins, Assessment and Data Specialist

• Maggie Conners, District K-6 Literacy Coach

• Lara Drew, Director of Elementary Education

QuickTime™ and aGraphics decompressorare needed to see this picture.

Purpose of Study

• To determine whether the set of professional development activities provided through the Collaborative Literacy Project (CLP) has a measurable impact on student performance in literacy.

Purpose of this Presentation

• To describe an evaluation design that we found helpful in investigating the effectiveness of a comprehensive professional development program.

Agenda

• Introduction• Purpose• Description of the Collaborative

Literacy Project (CLP)• Description of the Evaluation Design• Findings of Two Studies• Evaluating the Design• Next Steps

What is the Collaborative Literacy Project?

Professional development model designed to improve student learning in reading and writing in grades K-6.

Provides in-depth training in literacy for principals and teachers.

Develops demonstration classrooms and learning labs across the district.

Develops a literacy coaching model.

What is the Collaborative Literacy Project?

Learning is focused on effective instructional practices in literacy.

Teaching for Understanding

Explicit Instruction of Metacognitive Strategies

Using the Gradual Release of Responsibility

Using a Workshop Structure

What is the Collaborative Literacy Project?

Developed literacy coach model to support school-based professional learning in literacy.

Provide one-on-one coaching Support small group learning opportunities Facilitate whole staff learning opportunities Facilitate classroom observations Support leadership of principals and teacher

leaders Collect and share resources

Difficulties of Study

• CLP encompasses many types of professional development (PD).• Teachers participate in a variety

of activities that overlap.• How do we isolate the effects of

the CLP?

Study 1

Student Achievement in the 06-07 School Year

Evaluation Design

• Listed the major CLP PD activities.• Gave a value weight to each activity (see

page 1 of report).• Tallied the number of activities in which

each K-6 teacher participated in 05-06 and 06-07.

• Assigned “CLP PD Points” to each elementary teacher.

Evaluation Design

• Our K-6 teachers had CLP PD points that ranged from zero to 53 points.

• We formed two groups of teachers:High CLP PD -- all 06-07 teachers with 22 or

more points Low CLP PD -- a sample of those teachers

with 3 or fewer points matched to the high CLP PD group as closely as possible on grade level and student demographics (see page 2 of report)

Evaluation Design

• Our goal was to then examine the achievement of students in each of these two teacher groups.

Think/Pair/Share

• Think about a comprehensive professional development program with which you are familiar.

• Would this type of design be useful in evaluating that program?

• Why or why not?

Student Demographics

Despite our best efforts to match the student groups. . .

• Compared to students in the Low CLP PD group, a higher percentage of the students in the High CLP PD group were:On free/reduced meal statusELLHispanic

• Therefore, on average students in the High CLP PD group would be predicted to perform less well than those in the Low CLP PD group.

Teacher Characteristics

How did the two groups of teachers compare on characteristics other than the amount of CLP professional development?

• Compared to the High CLP PD teachers, the Low CLP PD teachers on average:Were more experiencedHad more overall graduate credits / clock

hours

Findings— Reading

• On K-6 measures of reading performance, students in the High CLP PD group performed higher than students in the Low CLP PD group on nine of the ten measures available (see page 3 of report).

Findings— Math

• If the teachers in the High CLP PD group were just “better teachers,” then we would see similar results in math.

• On measures of math performance in grades 2-6, students in the High CLP PD group performed on average higher than students in the Low CLP PD group on three of the six measures available (see page 3 of report).

Findings— Writing

• On the Grade 4 Writing WASL, students in the High CLP PD group performed slightly lower than students in the Low CLP PD group.

Findings— Science

• The Science WASL requires a great deal of reading to be successful.• On the Grade 5 Science WASL, a

much higher percentage of students in the High CLP PD group met standard than students in the Low CLP PD group.

Possible Sources of Error

• Comparing demographics grade-by-grade showed that the High CLP PD group did not have a higher percentage of low income students at every grade.

• Scoring of the district assessments might not be applied consistently across all classrooms.

Combining WASL Data

Combining WASL data avoids some issues:

Overall student demographics for grades 3-6:-- 41% of High CLP PD students were low income. -- 38% of Low CLP PD students were low income.

WASL scoring is consistent across all classrooms.

Combining WASL Data

• Combining Reading WASL Grades 3-6:73% of High CLP PD students met standard66% of Low CLP PD students met standard

• Combining Math WASL Grades 3-6:58% of High CLP PD students met standard57% of Low CLP PD students met standard

Summary of Study 1

• High CLP PD students performed better on practically every reading measure compared to Low CLP PD students.

• This was true despite the fact more High CLP PD students were from low income homes, and High CLP PD teachers were less experienced than the other group.

• The High CLP PD students did not perform better in math, showing that these students and teachers were not simply “more competent” than the Low CLP PD group.

Study 2

Student Achievement in the 07-08 School Year

Goal of Study 2

• To try to replicate the results of Study 1, using the same model.

The CLP PD Continuum

• Our teachers in Study 2 had CLP PD points that ranged from zero to 78 points.

• We formed two new groups of teachers:High CLP PD -- with 26 or more points Low CLP PD -- a sample of those teachers

with 6 or fewer points matched to the high CLP PD group as closely as possible on grade level and student demographics (see page 5 of report)

Student Demographics

• Again, the High CLP group had more students from traditionally low achieving groups (low income, Hispanic, ELL).

• The two groups at each of the grade levels were closer in demographics than in Study 1.

Teacher Demographics

• The two groups of teachers were closer in experience and graduate credits/clock hours than those in Study 1.

• A slight edge was in the direction of the High CLP teachers in Study 2.

• Some teachers were in both studies: 25 teachers in the High CLP group 13 teachers in the Low CLP group

Study 2 Findings

• Findings for Study 2 were more inconsistent than in Study 1 (see page 6 of report).

• Again, students in the High CLP group showed more strength in reading and science than in math or writing compared to the Low CLP group.

Summary of Findings

• A summary of the findings from both studies is on page 7 of the report.

Evaluating the Model

• “High CLP” teachers were identified only by the number of activities in which they participated, not by actual observation of their classroom practices. The inference is that they use effective literacy practices as a result of the CLP PD activities.

• The activities tallied as CLP PD points included only those provided through district coaching staff. Some schools provided CLP training through their school staff.

Evaluating the Model

Were the “High CLP” teachers better at teaching reading before participating in CLP activities?

• Data from the 2005 WASL (“pre” CLP) indicates that students of teachers who later became “High CLP” actually did less well in reading than those of teachers who later became “Low CLP.”

Next Steps

• Narrow the PD activities• Conduct classroom observations• Focus on intermediate grades• Focus on schools with high ELL• Use expertise of “High CLP” teachers

to help us refine the model.

Group Sharing

• What designs have you used to evaluate comprehensive professional development programs?

• What have you found to be the strengths and weaknesses of designs you have used?