Improving Teacher Quality Grants, Cycle 4:

External Evaluation ReportDecember 7th, 2007

University of Missouri-Columbia Evaluation Team

Evaluation Team

Principal Investigators

Sandra Abell

Fran Arbaugh

Mark Ehlert

John Lannin

Rose Marra

Graduate Research Assistants

Kristen Hutchins

Ya-Wen Cheng

Zeynep Kaymaz

Michele Lee

Dominike Merle

Chia-Yu Wang

Context of the Evaluation

• Improving Teacher Quality Grant program, Cycle 4, 2006-2007– Focus on high-need schools– 14 funded professional development projects– Science and mathematics, grades 4-8

Participant Summary• 383 participants;

• 84.0% white and 84.2% female;

• Most held regular certification in elementary/early childhood (n=236) and/or middle level mathematics (n=122) and/or science (n=203);

• 29% had been teaching for 1-5 years, 22.7% for 6-10 years, 34.5% for 11-15 years, and 9.5% for 26 or more years;

• 64.7% had participated in some form of PD the previous year; however, few participated in mathematics or science-specific, district-sponsored PD;

• 62.6% were new to the ITQG program;

• Taught in 124 different Missouri school districts, 3 private schools, and 3 charter schools;

• Directly impacted 33,580 students in the 2006-2007 school year.

Percentage of Participants from High-Need Districts

44.2%

55.8%

% participants from high-need districts % participants from non high-need districts

Were Cycle 4 ITQG Objectives Met?Objective Data Responding to Each

•Improve student achievement in math/science

•Students learned new content (MAP scores, student content tests)

•Increase teachers’ content knowledge and improve teachers’ pedagogical knowledge and skills

•Teachers learned new content and instructional strategies, and some began to use more inquiry-based instructional practices (evaluation instruments, teacher content knowledge tests)

•Impact on pre-service teachers

•Higher ed institutions improved their preparation of future teachers (evaluation instruments)

•Enhance use of assessment •Teacher knowledge/use of assessment was not improved (evaluation instruments, teacher content knowledge tests)

Four Effective PD Practices

• PD content

• PD delivery

• PD sustainability

• PD as part of a system

PD Content--Effective PracticesWhat the research says:Effective PD provides opportunities for teachers to build their content and pedagogical content knowledge and examine practice. (Loucks-Horsley et al., 2003)

What the standards say:Professional development for teachers of science requires integrating knowledge of science, learning, pedagogy, and students; it also requires applying that knowledge to science teaching. (NRC, 1996)

What the Cycle 4 RFP said:•Provide content knowledge related to standards and classroom instruction.•Provide instructional strategies related to content being taught in the classroom.•Provide research-based instructional strategies to the participating teachers.

PD Content—What We Found1. Science and mathematics subject matter content

--Relevant to teachers--Less is more

2. Integration of science and mathematics content--Need for explicit connections

3. Integration of subject matter and pedagogy--PIs must make their expectations for teacher learning clear,

make their pedagogical models explicit, and integrate theory/practice as well as content/pedagogy.

PD Content--Participant Preferences

Teachers preferred when PD content:•Related to their grade level and subject

•Aligned with local curriculum and GLEs

•Reflected teacher knowledge needs

•Related to student knowledge needs

•Included both science/mathematics content and pedagogical content

Representative comments:“I liked the different project ideas that relate to the GLEs.”“Working with professors that could help us combine the GLEs with our lesson design was probably the most valuable part of the experience for me.”“The most valuable part was working in groups to create inquiry-based lessons that are aligned to the GLEs.”

PD Content—Relevant Learning

0=No gain in content knowledge1=None of the gain was relevant to my teaching assignment2=Very little was relevant3=A Moderate amount was relevant4=Most of the gain was relevant5=All of the gain was relevant

FrequenciesDescriptive Statistics

Subject area 0 1 2 3 4 5 n/a n M SD

Mathematics 7 1 19 47 59 39 4 172 3.6 1.2

Science 3 3 7 46 75 84 6 218 4.0 1.0

End of Summer Institute

PD Content—Science/Math Subject Matter Amount

• Projects delivered PD to teachers across a range of grade levels. – largest range: 6 grade levels (grades 3-8) – smallest range: 2 grade levels (grades 6-7)

• Projects listed 2-8 subject matter foci

• Teachers were more satisfied when fewer content areas were covered

How to Design for Effectiveness in Science/Math Subject Matter

• UCM—Grade-level appropriateness

• Lincoln—Less is more

PD Content—Science/Math Integration

• 14 Cycle 4 projects– 5 targeted science – 2 targeted mathematics– 7 seven targeted both science and

mathematics

PD Content—Science/Math Integration

What participants saw as “least valuable”• The higher level math since I am lower level

science. • I am an elementary teacher and many of the

lessons were designed for high school math. I did not find this helpful to me.

• This experience wasn't related to teaching math. I am a math teacher and this experience was hard to incorporate into my classroom.

• I am a math teacher exclusively, so the time spent on science was not well spent.

How to Design for Effective Science/Math Integration

• UMR—Science and mathematics integration

PD Content—Subject Matter and Pedagogy

Contribution of Professional Development Emphasis Areas to Practice Reported by Participants at End of Professional Development Projects

UEmphasis AreasU UnU UMU USDU

Pedagogy 220 7.6 1.9 Content knowledge 221 7.1 2.2

0=none to 10=very much

How to Achieve Effective Balance of Subject Matter and Pedagogy

• SEMO—Integrating pedagogy into math content week

• NMSU—The learning cycle and mathematics

• Rockhurst—The learning cycle and science

PD Delivery

What the research says:Effective PD is driven by a well-defined image of effective classroom learning and teaching (Loucks-Horsley et al., 2003)

PD Delivery

How to use a framework to guide effective PD:

• Lincoln

• Rockhurst

PD Delivery

Challenges:

• Diversity of project teams

• Shared vision develops over time

• Involvement of local school district personnel

PD Delivery What the research says:•Effective PD engages teachers as adult learners in the learning approaches they will use with their students. (Loucks-Horsley et al., 2003)Create a high level of cognitive dissonance, and Provide sufficient time, structure, and support for teachers to think through the dissonance experienced (Thompson and Zeuli, 1999)

What the Cycle 4 RFP said:Provide research-based instructional strategies to

participating teachers

PD Delivery

Table VI.4: About How Often Do STUDENTS in Your Class Take Part in Each of the Following Types of Activities as Part of their Mathematics/Science Instruction?

Frequencies Descriptive Statistics

0 1 2 3 4 n M SD

Participate in student-led discussions 1 6 11 17 6 41 2.5 1.0 Participate in discussions with the teacher to further math/science understanding 0 3 5 24 10 42 3.0 0.8

PIs and Project Staff, Cycle 4 (p. 38)

PD Delivery

Table VI.5: How Often Do You Assess Student Progress in Mathematics/Science in Each of the Following Ways?

Frequencies Descriptive Statistics

0 1 2 3 4 n M SD Conduct a pre-assessment to determine what students already know 0 10 14 14 4 42 2.3 0.9

PIs and Project Staff, Cycle 4 (p. 38)

0=Never; 1=Rarely; 2= Sometimes; 3=Often; 4=All/almost all lessons.

PD Delivery

Table VI.2: About How Often Do YOU Do Each of the Following in Your Mathematics/Science Instruction?

Frequencies

Descriptive Statistics

0 1 2 3 4 n M SD

Introduce content through formal presentations 2 3 17 15 5 42 2.4 1.0 Engage the whole class in discussions 0 0 10 16 16 42 3.1 0.8 Pose open-ended questions 0 1 3 23 15 42 3.2 0.7 Require students to explain their reasoning when giving an answer 0 1 8 12 21 42 3.3 0.9 Ask students to explain concepts to one another 0 5 2 22 13 42 3.0 0.9 Encourage students to explore alternative methods for solutions 0 1 9 15 17 42 3.1 0.8

PIs and Project Staff, Cycle 4 (p. 35)

0=Never; 1=Rarely; 2= Sometimes; 3=Often; 4=All/almost all lessons.

PD DeliveryTeacher comments (Cycle 4)

I learn really well with this hands-on approach, and until these workshops, I have had limited knowledge of physics, especially how to formally teach energy transfers, etc.

We were able to work through the activity in the role of the student and then received the content knowledge that went along with it. By doing the activity myself I was able to completely understand how to recreate the activity for my students during the regular classroom.

Learning in the mode we expect our students to learn in was a very valuable tool in my teaching. It allowed me to feel some of the frustration my students would later feel when I changed from being the "giver" of information to the facilitator of students "finding" the information during the inquiry part of each lesson. (p. 48)

PD Delivery

How to engage teachers in approaches they will use with their students:

• UMSL - literacy strategies

• MU - notebooking

PD Delivery What the research says:Effective PD provides opportunities for teachers to collaborate with colleagues and other experts to improve their practice. (Loucks-Horsley et al., 2003)

What the Cycle 4 RFP said:–Contain grade-level and/or content-area collaboration

and work–Provide teachers with the opportunity to reflect on their

practices and to give the district feedback on the effectiveness of participation in this activity/experience

PD DeliveryTeacher comments (Cycle 4)

•To me, as a fairly new teacher the chance to collaborate with other professionals for that amount of time on our unit was very valuable. I loved having the chance to get to know others and build a team situation rather than feeling like I was on my own.

• I think talking and visiting with other teachers was invaluable! Being able to collaborate with other classroom teachers that also taught my grade level! I am the only 4th grade teacher within my district and I enjoyed meeting and planning some lessons with other teachers who also taught 4th grade.

•Time to collaborate with other teachers, share teacher techniques, classroom management strategies, plan and share ideas for lessons -- all of these things allowed me to expand my practices and methods in the classroom

•Networking with other teachers on the projects. I think I gained more knowledge from them as we worked through the activities. It was great to talk to teachers and realize you had similar problems in your classes. (p. 46)

PD Delivery

A vehicle for teacher collaboration:

• MSSU--E-conferencing

PD Sustainability

What the research says:Effective PD builds capacity for sustainability

(Loucks-Horsley et al., 2003)

What the Cycle 4 RFP said:Proposal narrative should:…• discuss how teachers and other participants will

be actively engaged over the life of the project and the potential for the project’s sustainability beyond the end date of the grant

PD Sustainability

Sustainability in ITQG projects:

• Year-long sustainability

• Multi-year sustainability

• Post-project sustainability

Year-long Sustainability

What did you do in your Cycle 4 project to promote teacher engagement throughout the year?

Teachers’ Views

PD Delivery Mechanisms n M SD max/min

Summer institute 219 8.3 2.0 1/10

School-year call backs 221 7.1 2.5 0/10

Other school-based activities 213 6.6 3.1 0/10

0=no contribution to practice, 10=very much contribution.

Year-long Sustainability: Call-back Sessions

• The Saturday call-back sessions. Much of the time we were not engaged in a meaningful task.

• I came to the first follow-up session and I felt that I wasn't getting anything out of it. I guess this is why I chose not to come to most of the sessions.

• The call back sessions because of the stress created trying to juggle my extra-curricular school responsibilities to allow me time to come.

Year-long Sustainability:Classroom-based Support

• Focus on PD content

• Clarity about the purpose(s) of classroom-based support

• How to sustain classroom-based support:– UCM– UMKC1

PD Sustainability

• Multi-year sustainability– PD improvement over time– Repeat participants

• Post-project sustainability– Small number of districts to concentrate

impact– Multiple teachers from the same school

PD Sustainability

Challenges:• Participant attrition (within a year; from year to

year)• “Mile-wide, inch deep” approach to participant

recruitment (issues of scale)• District priorities• Continued funding

PD as Part of a System What the research says:• School culture, leadership and support, and time for teacher learning

influence the quality of PD (Garet et al., 1999; Sparks, 2002)• Effective PD links with other parts of the education system (Loucks-

Horsley et al., 2003)

What the Cycle 4 RFP said:• Directly link to district and building school improvement plans;• Be developed with extensive participation of teachers, principals, and

other administrators;• Provide time and other resources for learning, practice, and follow-

up;• Be supported by district and building leadership.

Sample Teacher Comments“Although I enjoyed meeting and discussing content and objectives with other teachers I found that planning and writing lessons … with teachers outside of my own district was not helpful. Each district has their own focus so it was hard to work it out.”

“Some of the lessons really did not apply to my district’s alignment.”

“This project was wonderful. I only regret that I was not able to use it to the fullest extent based upon decisions made by the district I worked for. “

PD as Part of a System

How to achieve various partnerships:• UMKC2—Partnership with KC schools at

planning phase• UMSL—Partnership of science and literacy• 3 Rivers—Key role of school-based personnel in

planning and instructional team (Also links to Math Academy)

• MSSU—ITQG project as part of larger PD effort• MSU—networking with rural schools

PD as Part of a System

Challenges for future cycles:• Improving participation of high-need districts• Forming partnerships with K-12

– During design phase– Around assessment issues– For sustainability

• Linking higher education components

Four Effective PD Practices

• PD content

• PD delivery

• PD sustainability

• PD as part of a system

Future of Missouri METS

Importance of connecting ITQG with other METS initiatives:– DESE-funded MSP projects– METS Alliance– Teacher Leadership Academy

For example:– CMU—Competition for middle school students

Questions

Copies of the report and Executive Summary available at:

www.pdeval.missouri.edu