Center for Research on the Context of Teaching Stanford University
Evaluation of the New Teacher Center-Ravenswood City School District
Instructional Improvement Initiative
Final Report
Joan E. Talbert Jane L. David
with Pai-rou Chen Ann Jaquith M. Ken Cor
January 2010
Table of Contents
Executive Summary
Introduction 1
Theory of Action Underlying NTC’s Work in RCSD 2
Evaluation Approach and Data 4
Organization of the Report 6
Progress Over Three Years 7
Providing Teacher Mentoring Across the District 7
Developing Teacher Leaders and Learning Teams 14
Developing School Leadership 22
Providing, Brokering, and Collaborating on Professional Development 25
Trends on Intended Outcomes 31
Teacher Knowledge and Skills for Instruction and Leadership 32
School and District Conditions of Teaching 36
Teacher Retention 43
Student Performance on California Standards Tests 45
Lessons and Issues for Capacity Building Initiatives 49
Appendices
Appendix A. RCSD Student Demographics and Teacher Characteristics
Appendix B. Teacher Survey Scale Definitions
Appendix C. RCSD Teacher Retention by School, 2005-06 to 2009-10
Appendix D. RCSD Student Performance: Trends on California Standards Tests (CST)
Tables
Table 1. CRC Field Research by Year, 2006-7 through 2008-09 5
Table 2. All Teachers and Survey Respondents With and Without Mentors 6
Table 3 RCSD Classroom Teacher Attrition, 2006-07 to 2009-10 44
Table 4. Grade-level Trends on English and Math CST Scores by School and Years Working with NTC: Average Percent Students Scoring Proficient and Above from the Pre-NTC years to 2009 47
Table 5. School Trends on English and Math CST Scores by All Students and English Learners (ELs): Percent of Students Scoring At or Above Proficient from 2006 to 2009 48
Figures
Figure 1. NTC Logic Model 3
Figure 2. Teacher Ratings of Mentor Activities in 2007 and 2009 8
Figure 3A. How Teachers Rate Mentors’ Help with Instructional Skills in 2007 and 2009 11
Figure 3B. How Teachers Rate Mentors’ Help with Formative Assessment in 2007 and 2009 11
Figure 3C. How Teachers Rate Mentors’ Help with Developing Collaboration and Leadership Skills in 2007 and 2009 12
Figure 4. How Grade-Level Learning Teams Worked Together: K-5 vs. 6-8 17
Figure 5. Teacher Value Ratings of Professional Development, 2009 26
Figure 6. Change in Teachers’ Self-ratings on Instructional Skills 33
Figure 7. Change in Teachers’ Self-ratings on Skills in Using Formative Assessment and Individualizing Instruction 34
Figure 8. Teachers Self-ratings on Ability to Take a School Leadership Role: Change and Breakout for Teachers Who Where in Leadership Positions 36
Figure 9. Teacher Ratings of Selected Indicators of School Working Conditions 38
Figure 10. Teacher Ratings of School Professional Culture 39
Figure 11. Teacher Ratings of Principal Leadership, 2006 to 2009 41
Figure 12. Teacher Ratings on District Professionalism Indicators 42
i
Executive Summary
In the fall of 2006, the New Teacher Center (NTC) undertook an ambitious multi-year effort to expand their work in the Ravenswood City School District (RCSD) to strengthen teaching and learning throughout the district. Building on prior work with three of the district’s seven schools, the William and Flora Hewlett Foundation funded NTC to work towards system-wide instructional improvement and, at the same time, funded our three-year independent evaluation to document the effort and develop lessons for the field about partnerships to develop district capacity for continual improvement – districts, support providers, funders, and researchers.
The NTC-RCSD initiative was designed on the premise that adult learning must be at the center of efforts to increase student learning. NTC brought a long track-record and national reputation for excellence in mentoring new teachers, who are heavily represented in RCSD and most at risk of floundering. Mentoring new teachers formed the core of NTC’s work in RCSD and extended to enriching professional learning and leadership opportunities for all teachers. In addition, this initiative took NTC a qualitative move beyond its core work into a new frontier of developing district and community leadership to strengthen the system’s infrastructure to support teaching and learning. It also went beyond the district’s experience of working with external partners that provide specialized services, expecting district leaders to collaborate with NTC on a broad agenda to develop system capacity at all levels.
The fundamental idea was that investing in building the capacity of teachers—especially
new teachers—and helping to develop a district infrastructure to support ongoing professional learning would increase the retention of new teachers with solid skills and, consequently, increase student learning.
Our evaluation followed this logic, focusing on the main classes of activities proposed by
NTC and expected outcomes. Through repeated interviews with all stakeholders, observations of key events, and annual surveys of teachers, we documented progress on the major facets of NTC’s work: mentoring, professional development, learning teams, and leadership development. We also documented intermediate outcomes, including growth in teacher knowledge and changes in school and district support for instructional improvement. To assess ultimate goals, we analyzed data on teacher retention and student performance on California’s annual assessments in English and mathematics.
Finally, we consider the initiative’s theory of action and experience to draw lessons for
the field and to frame problems for practice and future research on district reform initiatives that aim to develop system capacity for continual improvement.
Progress over three years Our assessments of progress should be interpreted in the context of the challenges confronting the district during this time period. In addition to the expected challenges of a high-poverty, increasingly non-English speaking community, RCSD faced major annual budget cuts from the state plus increasing charter school enrollment and high turnover of leaders at all levels of the
ii
system. It also continued to grapple with a long-standing court decree for full inclusion of special education students in all classrooms and, in the second year, was named a Program Improvement district by the state resulting in additional state mandates. Providing teacher mentoring across the district. Each year NTC mentors worked with roughly half the teachers in the district, primarily those in their first or second year of teaching. Mentors provided a variety of types of assistance, tailoring the specifics of their work to each teacher. Both new and veteran teachers highly valued this support, as did principals, and their enthusiasm for mentors’ help with skill development increased from 2007 to 2009. Moreover, based on teacher reports and observations, teachers were putting into practice the instructional ideas and strategies introduced by their mentors and professional development. Developing teacher leaders and learning teams. In the fall of 2006, RCSD launched grade-level learning teams with strong support from NTC to focus professional development more on teacher collaboration—a significant departure from past practices. The goal was that teacher teams would conduct multiple cycles of inquiry throughout the school year. NTC offered monthly training to each grade-level team’s facilitator—about one-fifth of the district’s teachers. The process for selecting and training facilitators improved over the three years as did clarity about the purpose of the learning teams and tools to support their work. Teachers reported working together well, although teams varied enormously in their capacity to collaborate and carry out a cycle of inquiry. The efforts represented a strong start in a system with no history of collaboration, particularly given lack of time and uneven support from school and district administrators. The original design for four annual inquiry cycles of several meetings each did not survive competing demands on teachers’ and administrators’ time and by 2008-09 had been scaled back to two cycles, each of which was interrupted by last-minute changes in the meeting schedule. Developing and broadening school leadership. NTC administrator coaches and lead mentors in each school provided support to site administrators. Most principals and assistant principals, particularly those new to the job, highly valued their support. Initiated in the first year by NTC administrator coaches, principals met regularly for walk throughs in each others’ schools and valued time together, but they struggled to develop and sustain their own learning community in the absence of district validation. Moreover, it was difficult for them to maintain momentum and mutual trust with annual turnover among the seven principals; having new faces each year meant starting anew. Four of the six schools that remained intact throughout the three years of our evaluation saw at least one change in leadership during that time. Providing, brokering, and collaborating on professional development. Ensuring that teachers have access to high-quality formal professional development is central in NTC’s strategy to build teacher capacity. In addition to mentoring individual teacher and leading the development of grade-level learning teams, NTC brokered external workshops and extended training for teachers. Mentors also designed and led in-house professional development sessions with district staff that aimed to arm teachers with a range of effective instructional strategies focused on literacy and on English learners. Across all three years, teachers gave high ratings to these learning experiences, particularly those that provided opportunities to observe strong practice and to link training directly to their curriculum.
iii
Despite the high quality of NTC-provided professional development and enthusiastic teacher response, the instructional principles and practices promoted did not gain traction with district administrators. Tensions between the different philosophies of instruction espoused by NTC and district administrators were evident in 2006-07, and some district leaders perceived that NTC did not care about students’ scores on the CST. Initially, NTC leaders tried to address attacks and bridge underlying knowledge gaps by framing the required district curriculum as the “what” and the instructional strategies they offered as the “how.” But this distinction was not as clear to many teachers, particularly those not working with NTC mentors, as it was to NTC and district staff. Further, NTC’s ambitions for teacher learning and professional judgments far exceeded those of the district. In the third year, when the district was subject to intensified pressure from the state-mandated district assistance and intervention team [DAIT] to teach the core curriculum “with fidelity,” the NTC team took steps to redesign its approach to professional development, tying instructional strategies directly to the district’s curriculum. This new approach to professional development provided a teaching-learning framework of “gradual release” that applies across subject areas and featured teacher expertise through videotapes of RCSD classrooms. It embodies NTC’s principle of teaching to develop independent student learning in the context of the district curriculum, and it features district students and their teachers as learning context and evidence of success. The evolution of NTC’s professional development efforts in RCSD is testimony to the organization’s learning from its practice in a district under extreme pressure to improve student outcomes and with limited professional capacity to do so. It also provides evidence of the challenges a high-quality intermediary organization faces in working with a struggling district – fundamental challenges that center on how to maintain its quality standards in an inhospitable context. NTC struggled to navigate challenges to its core values that were presented by district leaders’ and the DAIT’s approaches to improving instruction. Trends on Intended Outcomes The intended outcomes of NTC’s work in RCSD include growth in teachers’ knowledge and skills and improved conditions of teaching sufficient to retain effective teachers and steadily improve student achievement. Teacher knowledge and skills for instruction and for school leadership. NTC’s mentoring of teachers, as well as professional development brokered or designed by NTC, should result in teachers’ growth in instructional skills and leadership. Teachers’ self-ratings on our annual survey registered change over time on these outcomes. District teachers’ sense of how prepared they are on a variety of instructional skills show a consistent pattern of small gains over the three years, which could reflect mentoring and professional development or experience and interactions with colleagues. Most salient are strong gains in the third year in teachers’ self-ratings on using formative assessments to inform instruction and individualizing instruction for English learners, reflecting NTC’s focus on preparing teachers to assess student learning in the gradual release framework. Individualzing instruction for special education students did not show gains—a finding consistent with the high demands on teachers from the court-mandated full-inclusion model coupled with minimal support. Survey evidence of teachers’ preparedness to
iv
take on a leadership role in their school remained high over the three years. Evidence from teachers’ actions point to increasing leadership in several arenas: becoming active members of school leadership teams, taking on the role of grade level facilitators of learning teams, mobilizing to present a petition to call the school board’s attention to failed curriculum leadership, and volunteering to be videotaped for professional development sessions. School and district conditions of teaching. Given the dismal condition of school physical plants and yards, dearth of instructional resources, and uncoordinated after-school programs that characterized RCSD in 2003-04, NTC’s work with the district included efforts to establish basic conditions of teaching and learning. NTC worked with community partners to improve school physical plants and yards and set up school resource rooms; and they partnered with the district to develop instructionally rich after-school programs. In these areas, teacher ratings of school conditions showed significant progress from 2006 to 2009. Progress was evident also in measures of school culture over the same time period: shared leadership, collaboration on instruction, colleague support, and teacher knowledge development. The data offer a picture of gradual overall improvement until the last year, coinciding with a significant decline in teachers’ perceptions of district support. With the exception of a spike in ratings of district support for teacher professional development at the end of the first year, during which learning teams were introduced, teacher ratings on indicators of district leadership for instructional improvement remained low on all indicators. In the third year, coincident with pressure to implement “corrective actions” through pacing guides and fidelity to the curriculum, teacher ratings of district support plummeted, most dramatically around their support for professional development of teachers—the area which earlier had seen the largest increase. Teacher retention. A primary goal of the initiative is to increase teacher retention, since investments in building their capacity does not hold long-term benefit for the district if they leave. Based on annual reviews of teacher rosters, we calculated overall retention and then collected data on individuals’ reasons for leaving the district in order to create an adjusted retention rate that excludes leavers whose contract was not renewed, who took non-teaching jobs in the district, and who retired or left teaching for reasons such as going to graduate school. Over the three years, RCSD’s annual teacher retention rate ranged from 73 percent to 77 percent, slightly up from the 68 percent we computed for the baseline year. Adjusted rates from 2006-07 through 2009-10 were 89 percent for each year, indicating that retention of skilled teachers was unchanged over the three years. Long-term retention of classroom teachers is considerably lower than the annual rates might suggest. Of the 131 teachers in 2006-07, only about half (51%) were still teaching in the district three years later (2009-10). The three-year retention rate for classroom teachers who received NTC mentoring during 2006-07 (70 of the 131 teachers) is 36 percent. Retention of the subset of 32 mentored teachers who were in their first or second year of teaching in 2006-07 and not in the Teach for America program is 34 percent still in the classroom. Special education teachers had greater turnover and extremely low three-year retention rates. Of the 11 specialists who received NTC mentoring in 2006-07, only 4 (36 percent) remained the following year and just 1 (9 percent) was still teaching in the district in 2009-10.
v
Given the court decree to implement a full-inclusion model where all special education students are placed in regular classrooms, this turnover rate has significant repercussions for classroom teachers who are faced with challenging students for whom little and inconsistent support exists Student performance on California Standards Tests (CST). To investigate whether NTC’s efforts made a difference for students’ academic performance as measured by the CST, our evaluation compared trends in English and mathematics for students in schools that worked with NTC for different periods of time. As a very rough gauge of impact, we expected that schools with more years of mentoring (those that began working with NTC prior to the district-wide initiative) would show greater gains than the other three district schools with three years of mentoring. Grade-level trends in percent Proficient or Advanced in English and mathematics by school show an overall picture of variation in gains across grades, subjects and schools, with a slight advantage for schools with more than three years of mentoring experience (primarily in that many fewer grade levels showed declines in proportions of students scoring proficient on English and math tests during their time of working with NTC mentors). Three-year trends at the school level for all students and English Learners show progress in percent reaching proficiency, especially in mathematics. Our estimate of NTC’s effect on student learning in RCSD is hampered by lack of data for early grades (K-1), where mentors worked intensively, since the CST system begins testing in the 2nd grade. . It also is hampered by teacher turnover. NTC mentoring may have had a strong effect on teachers’ success with their students in any given year, but the fact that only 36 percent of teachers mentored in 2006-07 were still teaching in the district three years later dampens its potential long-term effect on student achievement. Our evaluation of NTC’s work with RCSD spanned a three-year period during a most challenging time. From the school board to the state-provided assistance team to the central office and schools, few leaders brought prior experience to their jobs. In that light, it is not surprising that outcomes are not as strongly positive as might have been desired by everyone involved. Without a doubt, clear benefits to the growth of teachers’ instructional skills and their leadership in district schools accrued as a result of the Hewlett Foundation’s significant investment in the partnership. So far, there is little evidence of gains on the ultimate goals of increasing new teacher retention and improving student achievement. Lessons and Issues for Capacity Building Initiatives
The particular experiences of the Hewlett Foundation, NTC, and RCSD in this initiative help pinpoint a series of questions and dilemmas that may be useful for foundations, external partners, and struggling districts, as well as for future research on district reform. They include the quandary of investing in building the capacity of individuals who are unlikely to stay in the system and the related question of how guidance for instructional improvement provided by an external partner can be sustained after external funding ends. The experience also calls attention to potential tension between the philosophy, agenda, and culture of a district and external partners, particularly in struggling districts during times of heightened accountability pressures
vi
from the state. Both the challenges and accomplishments of NTC’s work with RCSD inform these fundamental problems of district capacity building initiatives.
What does capacity building look like in a high-turnover district? How the problem of district capacity building is conceived makes all the difference for progress. An approach that features developing individual human capital is bound to disappoint in a system that loses professionals at high rates annually. A complementary approach centers on developing organizational supports for teaching and learning. Capacity in this view includes structures and tools that scaffold teacher and administrator learning and that address challenges presented by high turnover. Among them are designs to leverage and support professional learning communities of teachers that use data to focus instruction and improve student learning. However, as was apparent in RCSD where large proportions of teachers are new to teaching, such designs must create teams that include a critical mass of experienced teacher leaders to advance the quality of their practice and support new teachers. Research typically identifies high turnover as a problem to be solved. We suggest a parallel line of inquiry that takes turnover as a given in many urban school systems and investigates the kinds of structures and resources that promote continued adult learning in the face of changing casts of characters. The most significant professional capacity developed in RCSD through this initiative was new norms and practices of teacher leadership and collaboration, yet the district lacks mechanisms and resources to support teams’ ongoing learning. What does “sustainability” mean in a capacity-building partnership? Sustainability could mean continuing with an external provider; however, this is unrealistic in tight money times. More fundamental is the question of whether and how school and district professional cultures embrace and carry the principles for practice that an external partner nurtured. The case of NTC-RCSD points to the difficulties of sustaining principles—such as “gradual release”—which are not embraced by the majority of district and school administrators. Is it possible to sustain practices without understanding and commitment from district leaders? Can external partners help build local policies, structures and tools to support practices in the absence of district leadership commitment to the underlying principles for practice? These questions make clear that learning at all system levels is essential if a district is to take on new ways of doing business. Unless district leaders share the same goals and are learning along side teachers and collaborating with the external partners to make the learning usable in their school contexts, the knowledge and practices engendered by an initiative are not likely to last. How these conditions for sustainable system improvement are established, especially in a struggling district under pressure to get short-term boosts in student test scores, is an important frontier for future research. How does a support organization adapt its “model” to meet district needs? One key to sustainable, capacity building district reform is the ability of external partners to be strategic in adapting their successful practices to context and readiness conditions of a particular district. An organization like NTC, with a well-developed model for mentoring beginning teachers and a national reputation for quality, faces a dilemma in partnering with a struggling urban school
vii
district. Can it adhere to its model and philosophy with fidelity and simultaneously adapt to the particulars of the district context of teaching? NTC struggled with this tension, especially in not wanting to compromise its stance on quality instruction by embracing the district’s Open Court literacy textbook and district pacing guide that put forth a lock-step curriculum and limited teachers’ discretion to adapt to their particular students. Yet the mentored teachers had no choice but to use the mandated text and those who did not faced sanctions from the district. As NTC’s work in RCSD evolved, the team adapted its work and created new tools that both adhered to its core principles for instruction and adult learning, such as ongoing use of formative assessments, and complemented district instructional mandates. Both support organizations and districts who partner with them to improve teaching and learning face the question of how to customize a program or model to the system in ways that maintain quality standards and meet local needs. Ironically, maintaining fidelity to surface structures of a support organization’s model may sacrifice core principles on which the model was developed. Customizing work for a district then entails explicitly defining core principles for quality and using them to design and continually adapt work in the district—and even to choose which districts are potentially viable partners.
Research designed to flesh out the meaning and processes of high-integrity adaptation of
an instructional improvement model would contribute to understanding when and how support organizations can effectively customize their philosophies and practices. Similarly, studying how districts make strategic choices about which external organizations to partner with and how to stage their work in the district would yield insights for developing productive partnerships. How can a foundation leverage the success of its capacity building initiatives? An important lesson from this Hewlett-funded initiative centers on the struggle between NTC and district leaders over authority and influence during the three years documented by our evaluation. Lacking a shared understanding and vision for “district capacity building,” district and NTC leaders sometimes held different views of their respective roles in the initiative. In significant ways, the “district reform” vision and agenda of the initiative was set by the Foundation – through proposal requirements, funding to NTC, and an external evaluation charged with documenting system change. Neither NTC project leaders and mentors nor district and school administrators regarded the initiative as aiming to develop the system’s capacity for continual improvement. Nor did they share a common view of how the “partnership” would work. A dilemma for foundations that support district reform initiatives centers on how heavy-handed, versus hands-off, the funder should be in establishing a shared vision of desired outcomes and promoting an effective working partnership between a district and external support provider. Key is the question of what the funder should constrain in order to leverage a productive partnering relationship and avoid predictable tensions. What could have made a difference for this initiative’s partnership? Research might explore how particular funding and organizational arrangements create incentives to forge shared understandings and commitments, including consideration of how evaluation can play a more proactive and constructive role.
viii
* * * * * Our evaluation of the NTC’s work in RCSD identified a broad range of positive effects on professional development, teachers’ skills and knowledge, and teacher leadership. At the same time, the NTC-RCSD partnership surfaced challenges and puzzles endemic to a system capacity building initiative. It illuminated where and how NTC efforts found traction, and it generated the issues sketched here. A program of research to capture how successful districts choose and develop partnerships and their partnering practices would contribute important new knowledge about the potential for external agents to help develop and sustain system capacity for continuous improvement.
1
Introduction
At the request of the William and Flora Hewlett Foundation, Stanford University’s Center for Research on the Context of Teaching (CRC) undertook a three-year evaluation of the New Teacher Center’s (NTC)’s work with the Ravenswood City School District (RCSD) to strengthen teaching and learning across district schools. The system initiative began in 2006-07, building upon a three–year history of NTC mentoring support for beginning teachers in three RCSD schools.1 This final report documents the evolution of NTC’s work with the district and accomplishments from the fall of 2006 through spring 2009, and frames issues about partnerships to develop district capacity for continual improvement for the field – districts, support providers, funders and researchers.
The NTC-RCSD initiative rests on the premise that adult learning must be at the center of
efforts to increase student learning and it leads with a focus on new teachers who are heavily represented in this district and most at risk of floundering. NTC brought a long track-record and national reputation for excellence in mentoring new teachers. The quality of its work is grounded in the organization’s first-hand experience and a growing body of research on adult learning: (1) it builds up from the identified needs of teachers, (2) it focuses on formative assessment at each level of the system,2 (3) it provides intensive and sustained opportunities for adult learning, and (4) it aims to create communities of adult learners within and across schools. Further, NTC aimed to apply the same principles for adult learning, formative assessment of professional practice, and learning communities to its own organization, investing heavily in the continuing development of high-quality mentoring practice among individuals that NTC employs.
The NTC-RCSD endeavor is ambitious because it is fundamentally about increasing the capacity of the district to continually improve teaching and learning. For NTC, the initiative took the organization a qualitative move beyond its established new teacher induction model. It took NTC into a new frontier of supporting the development of leadership for improvement throughout a system and collaborating with district administrators and the community to strengthen the system’s infrastructure to support teaching and learning. It also went beyond the district’s experience of working with external partners, who typically provide specialized services, expecting the district to collaborate with NTC on a broad agenda to develop system capacity at all levels.
This endeavor began as a major undertaking and increased in difficulty over the three years as the district struggled with filling central office positions and complying with an ongoing special education consent decree.3 It was intensified by state requirements and oversight
1 In 2003-2005 NTC worked with two schools and added a third in 2005-2006. 2 We use “formative assessment” in its broadest sense: using feedback on desired outcomes to improve one’s practice. For example, we include teachers’ uses of diagnostic data to adjust instruction, mentors’ feedback to teachers, as well as district administers’ use of information from teachers to improve their support for teaching and learning. 3 The consent decree mandates a “full inclusion” model for special education instruction in RCSD, a model adopted in no other district in the nation. In this model, special needs students are included in classes according to their age, and regular teachers are expected to meet their needs.
2
stemming from the district’s Program Improvement status. In the spring of 2008, the state assigned RCSD a District Assistance and Intervention Team (DAIT) that developed a Correction Action Plan to be implemented in 2008-09. Although the ultimate goals of all the players were similar—to increase student learning—the pressure to raise test scores quickly became the primary driver.
Our evaluation centered on the overarching question: What does it take to develop system capacity to create conditions that attract and retain new teachers as well as enable continuous improvement of practice in a district like Ravenswood? Such districts are characterized by: high proportions of students from poor families whose native language is not English, high community transience and student turnover, high teacher turnover, high administrator turnover at school and district levels, and inconsistent board and community leadership. California data on district demographic and teaching force characteristics show that RCSD has substantially higher rates of English language learners, poverty, and new teachers than the Bay Area and state averages (see Appendix A). 4 Although small by urban school system standards, RCSD is beset by many of the same problems of poverty and population churn, contentious board politics, and violent crime that threaten professionals’ and students’ sense of security. In sum, RCSD’s context, history of leadership turnover, teacher attrition, and poor student performance represent a syndrome of conditions typical of the poorest communities in California and the nation. The Hewlett investment and this evaluation address the question of whether and how this syndrome can be reversed through an intensive and broadly defined professional development strategy.
The evaluation documented NTC’s work with the district and tracked intended school and district culture changes, as well as ultimate outcomes of teacher retention and student achievement. How did the NTC mentors and coaches work to promote professional learning in district schools? How did professionals at all levels of the system perceive and respond to the external partner’s support? To what extent and how did this make a difference for teaching and learning across district schools? What follows is our characterization of the logic that underlies NTC’s partnership with the district and description of our research methods for tracking the work. Theory of Action Underlying NTC’s Work in RCSD RCSD and NTC share the dual goals of increasing teacher retention and student achievement. The logic model underlying the effort, represented in Figure 1, rests on the premise that multiple forms of high-quality professional development and support throughout the school system can create a culture that focuses on strengthening teaching and learning. It is grounded in a set of assumptions about the relationships between student learning, teacher learning, and system learning.
NTC’s core work is grounded in the premise that student learning is strongly affected by
what teachers know and know how to teach effectively, which is strongly influenced by their opportunities to continually refine their practice. The richest opportunities for teacher learning 4 Notably, in 2004-05 twenty-seven percent of RCSD teachers had less than two years of teaching experience, a rate that is 5 standard deviations above the state mean of 5 percent.
3
combine effective mentoring, professional development, and professional learning communities, all focused on using formative assessment to refine practice. In its work with RCSD, NTC has used the same logic to create learning opportunities for administrators designed to improve their support of teaching and learning.
Figure 1. NTC Logic Model
School leadership and support for teacher learning and instruction
Teacher mentoring, training, and support
Increased student achievement
Increased teacher knowledge of content and skills for instruction
District leadership and support for principals, teachers, and students
Higher teacher retention rates
Development of district PD and supports for teaching and learning
NTC and RCSD initiative to increase teacher retention and student
achievement
Principal coaching, training and PLC development
Teacher leadership and PLC development
Establishment of grade level learning teams to improve instruction
Improved school and district conditions that support teaching and learning
4
The model predicts that mentoring focused on new teachers, coupled with enhanced opportunities for high-quality professional development and support, will increase teachers’ knowledge of content and their repertoire of teaching strategies and skills. Similarly, opportunities that engage teachers in teams that meet regularly to discuss teaching and learning provide a practice-based forum for teachers to learn from each other and develop into inquiry-based learning teams or professional learning communities (PLCs).
As Figure 1 depicts, NTC’s work with RCSD also intends support for professional
learning at the administrator level. Coaching of principals as well as opportunities for professional development and support is expected to increase school leadership in support of teacher learning and improved instruction. In addition to individual coaching and support, developing a professional learning community of principals and assistant principals provides opportunities for learning and collaboration around issues of teaching, learning, and school leadership.
NTC aims to enhance communication and alignment between levels of the system. The
model predicts that the work of the district and NTC to design professional development and supports for teaching and learning will lead to more powerful learning opportunities and stronger supports for principal, teachers, and students. As a result, both school and district conditions that support teaching and learning will be established.
Ultimately, RCSD teachers, who work with students struggling with school and with the
language, will be more likely to experience success and to stay in teaching and in their current job. The crux of the logic model is that all these elements will result in higher teacher retention and increased achievement for students.
Implicit within and between each level is the notion that ongoing formative assessment—
both formal and informal feedback on practice—is essential to learning and change for district leaders, principals, teachers, and students as well as for NTC. Also implicit is that the relationship between RCSD and NTC is dynamic and over the three years would be continually redefined to ensure that the trajectory of improvement is sustained beyond the next three years. Evaluation Approach and Data Our evaluation design followed the logic of the model laid out in Figure 1. Our data collection and analysis focused on the main classes of activities of NTC and RCSD and the expected outcomes.5
To track progress, we looked for evidence that the components of the logic model were being implemented as intended at the school or the district level as well as evidence of unintended consequences. Each year we interviewed central office leaders, principals in all seven district schools, and a sample of teachers who were mentored, and we conducted focus groups with new and veteran teachers in each district school. We also interviewed teachers who became 5 Our data collection and analysis discussions were carried out in collaboration with Amy Gerstein, NTC’s formative evaluator, and her colleague Lisa Westrich-Taylor.
5
facilitators of grade-level learning teams and a sample of teachers from several learning teams. In addition we interviewed NTC mentors, coaches, as well as leadership and staff in the district’s central office. We designed our protocols to capture perceptions of each of the major components of NTC’s work with the district: mentoring, professional development, learning teams, and leadership development in the schools as well as school and district support for instructional improvement.
We observed a range of professional development workshops and institutes designed or
brokered by NTC or the district, as well as principal classroom visits and facilitator training. In the third year we also interviewed members of the District Assistance and Improvement Team. Table 1 summarizes our field research over the three years.
Table 1. CRC Field Research by Year, 2006-07 through 2008-09
2006-07* 2007-08 2008-09 Interviews
NTC lead staff and mentors 34 24 21 RCSD district administrators 5 9 5 RCSD principals and assistant principals 6 6 16 RCSD teachers 20 45 30 Others (court monitor, external partners) 1 1 1
Observations Mentor meetings and professional development sessions for mentors 30 18 19
RCSD profession development sessions** 14 12 19 Learning team meetings and districtwide share-out meetings 10 9
Teaching and mentoring sessions 8 Principal Walkthroughs/Quick Visits and principal meetings 2 3
Other district meetings 3
Note: Data collection was carried out in collaboration with Amy Gerstein, NTC’s formative evaluator, and her colleague Lisa Westrich-Taylor. *During Spring 2006 a baseline teacher survey was conducted, along with interviews with NTC lead staff, observations of seven mentor meetings and professional development sessions for mentors, and observations of two RCSD profession development sessions and two other meetings. **Training sessions included facilitator training/forum, Writing/Literacy Committee meetings, New Teacher Teaching and Learning Seminar, Foundation in Literacy in Language, CELL/ExLL training, Classroom Intervention training, after school training, and districtwide professional development sessions.
We conducted surveys of teachers each spring from 2006 through 2009. We used the
spring 2006 survey results as a baseline marking the spring prior to NTC’s expansion to all seven district schools. Response rates were over 80 percent for every year except 2008 which was 60
6
percent (see Table 2).6 For the 2008 sample, we compared those who responded to the survey to non-respondents on several dimensions and were not able to detect bias. For example, the ratio of teachers mentored to all teachers was the same for the sample as for the entire district. In this final report, however, the bulk of our survey analyses compare spring 2006 baseline data with spring 2009 year at the end of the three years. For questions specific to mentored teachers, we use 2007 as the baseline since that was the first year all schools had NTC mentors.
Table 2. All Teachers and Survey Respondents With and Without Mentors
2007 2008 2009 All
teachers Survey sample
All teachers
Survey sample
All teachers
Survey sample
Had NTC Mentor
81 (48.2%)
67 (45.9%)
82 (48.0%)
49 (47.6%)
75 (41.7%)
62 (42.8%)
Did not have NTC mentor
87 (51.8%)
79 (54.1%)
89 (52.0%)
54 (52.4%)
105 (58.3%)
83 (57.2%)
To evaluate the initiative's intended intermediate outcomes, we analyze change between 2006 and 2009 on survey measures of teacher knowledge and skills, school resources and culture, and district support of teaching and learning. Our analysis of change in teacher retention uses teacher rosters for each year to identify those who left after the previous year and information from mentors to classify each according to the reason for leaving; adjusted retention rates exclude teachers whose contract was not renewed or who left teaching. Trends in student achievement are assessed using California Standards Test results for English and mathematics and measured as percent students scoring at or above Proficient by subject, grade-level, and English Learner status for all district schools. For each school we compare student proficiency rates in 2009 with those for the two years prior to NTC's work with the school and consider whether or not the change is greater than the statewide trend for the same time period. Organization of the Report The next section of the report describes progress over three years on key facets of the initiative. The subsequent section presents the results of our analyses of intermediate and ultimate outcomes. The final section offers lessons and issues for those investing in and carrying out district system capacity building initiatives.
6 Response rates by year: 2006 – 86 percent; 2007 – 87 percent; 2008 – 60 percent; 2009 – 81 percent.
7
Progress Over Three Years
Providing Teacher Mentoring Across the District Teachers new to the profession typically face challenging classroom situations for which they are inadequately prepared. Such challenges are exacerbated in schools serving students who are poor, minority, or English learners. Teachers with no or limited teaching experience find themselves overwhelmed by the range of learning needs presented by their students, from those who begin school unable to hold a pencil or communicate in English to those with severe emotional and cognitive disabilities. Moreover, in RCSD as in other urban districts, many of their colleagues are also new, limiting their opportunities for guidance and help from their peers.
The core work of NTC is the provision of mentoring to new teachers. In RCSD, NTC mentoring initially focused on first and second year teachers. Over time, NTC mentors expanded their role to include informal work with more teachers and closer coordination with school leadership. They worked formally with roughly half the teachers in the district during each of the three years. The role of “lead mentor” at each school was introduced in the second year to facilitate both coordination of mentors working in a school and communication with school administrators. Teachers received and highly valued a range of supports from NTC mentors NTC mentors provided a variety of types of assistance and support to both new and veteran teachers. For many first and second year teachers especially, mentoring becomes a lifeline as these inexperienced teachers struggle to keep their heads above water. These teachers characterized their NTC mentors as “fantastic,” “immensely valuable,” “the one I always go to,” and “an incredible help to me.” Mentors find that inexperienced teachers need assistance in three general areas: establishing a classroom climate that is conducive to learning, developing instruction to meet the needs of a broad range of learners, and attending to the needs of students with particular challenges, such as limited knowledge of English or learning disabilities. In fact, these three goals represent a trajectory of learning for new teachers. Mentors encouraged teachers to attend professional development brokered by NTC or offered in conjunction with the district to expand their knowledge and repertoires of teaching strategies. As part of their work, mentors led teachers through the credentialing process, ensuring they met all the state requirements for certification. They also provided support for grade-level learning teams, described in the next section. Teachers reported that their mentor spent an average of nine hours a month working with them, ranging from one to 30 hours a month. In surveys conducted at the end of the first and third years, 90 percent of teachers who had mentors reported that their mentor visited their classroom, provided feedback on their teaching, and discussed professional and personal problems with them. Almost as many reported that their mentor conducted formal observations, gave them tools (e.g., for assessment), and helped them set goals for professional growth. In both years, the least frequent activity that mentors did was modeling lessons, although the frequency increased over the three years.
8
Teachers gave consistently high ratings to their mentors’ work with them (see Figure 2). Positive ratings increased by more than ten percentage points for three mentoring roles:
• Worked with me to develop my leadership skills (67 to 85 percent) • Visited my classroom during instruction time (75 to 89 percent) • Helped me assess student academic and emotional needs (83 to 94 percent)
By the end of the third year, all of the mentoring roles were rated valuable or extremely valuable by more than 80 percent of the teachers.
Figure 2. Teacher Ratings of Mentor Activities in 2007 and 2009
0% 20% 40% 60% 80% 100%
Helped me with long-term planning of instruction
Helped me set goals for professional growth
Discussed professional and personal problems with me
Worked with me to develop my leadership skills
Looked at student work with me
Gave me tools (e.g., for assessment)
Provided feedback on my teaching
Visited my classroom during instruction time
Helped me assess student academic and emotional needs
% Valuable and Extremely Valuable
2007 N ~= 522009 N ~= 47
NTC mentors tailored their work to the particular needs of individual teachers Although mentors carry out many of the same activities, each aims to address the needs and developmental paths of individual teachers and adjusts assistance as needed. Survey ratings on particular mentor roles, therefore, capture only part of the picture. In practice, how a mentor works with a teacher depends on the teacher’s strengths and needs and on demands and resources in the teaching context. Our case studies of individual mentoring relationships offer insight into what the work looks like.
Illustrating the developmental stages of the mentoring relationship, one teacher described how her mentor first focused on classroom management and then shifted to lesson planning and tracking student understanding:
9
[My mentor] pretty much influenced so much of what I do in the classroom. Last year I think we focused a lot more on kind of classroom management, how to teach in a day, to maybe have a schedule, how can you help with those really talkative students or those big issues. And, as you can see, she helped me make some lines on the carpet, because it was hard to set up the carpet in the beginning of the year. Just little things, little hints here and there that really helped. And this year it’s been more about developing...“Okay, now your practice as a teacher: let’s work on lesson plans together, let’s see how we can use that backwards model to make sure that students are really learning the material. Let’s go to these trainings that will help you to help these students understand the material at grade level.
In another case, the mentor used a combination of observation and focused discussion to figure out what sorts of support were needed. The teacher described the support she typically receives in this way:
I asked [my mentor] for her opinion about writing instruction…what would have [kindergartners] understand writing the best. So she came in and she saw what I was doing, and then we met afterwards and kind of debriefed and she did ‘what was working’ and ‘what we needed to work on’. And then she came back the next week and she watched again after giving me suggestions. And she gave me further suggestions. And then she said she was going to come in and model the lesson after break.
As teachers develop, some of what mentors do can lead teachers to re-think how they view their struggling students. For example, the teacher quoted above described how her mentor led her to reconceptualize her view of certain students as “lazy.” The NTC mentor addressed the teacher’s view head-on saying “maybe we won’t call her lazy or see that the parents are babying her but instead realize that something else is preventing her from doing her work.” Helping teachers to shift their perspective on who can learn and on the teacher’s role in engaging students in the process of learning does not happen overnight. Progress, when it appears, is gradual and can seem transient or incident-specific. Sometimes, however, changes are more apparent. For instance, this same mentor coached another teacher on establishing “positive discipline” in her classroom. This teacher said,
We talked about why certain students are acting a certain way, and how we can support them and how we can help them. Like, what do they really want? So, for example, if there’s a bully in your classroom, what do they really want? They really want power. And how can I help them?
For this teacher, the mentoring support resulted in changed classroom practices, like the introduction of classroom jobs. She remarked that “my one student who was kind of that bully, he got to be our president the first week and it was just like… that was the best week of the school year!” In describing her work with her mentor, she said that her whole perspective on how she sees discipline in the classroom had changed.
Before, it was just like, “You get your name on the board, you’re going to owe me time, you’re going to do this, and if you do it again you’re going to get this punishment.” And [my mentor has] really helped me to see like, “Okay, let’s see what other options we can do. Let’s take the student aside and see really why they’re acting this certain way. And let’s try to figure out how we can help them.”
10
Although not all mentor coaching efforts result in transformed practices and understandings, the efforts that mentors make, on a case-by-case basis, are designed to help teachers establish classroom environments and instruction conducive to learning for all students. Mentors reported that they introduced more systematic coaching around establishing classroom routines and behavior management strategies for new teachers, in response to context demands. At the middle grade level, mentors intensified their focus on formative assessment and pacing of curriculum to help teachers address diverse learning needs in classes that typically included several special education students. Teachers gave high ratings to mentor’s help in developing their skills NTC mentors worked with teachers on a wide variety of teaching skills. Because mentors focus on different skills with different teachers depending on their needs, not all teachers worked with their mentors on any given skill. Nonetheless, large proportions of mentored teachers across the district reported that their mentors were helpful or extremely helpful in developing the range of skills tapped in our surveys. Notably, positive ratings increased between 2007 and 2009. Figures 3A and 3B show a clear pattern of increase in percent of teachers rating their mentor as helpful for each of the instructional and formative assessment skills listed in the survey. A similar trend appears for teacher ratings of their mentor’s help in developing their skills in working as part of a learning team and taking a leadership role in the school (Figure 3C). Teachers gave much higher ratings in 2009 than in 2007 of their mentor’s help with several skills (listed here in order of increase in percent with “helpful” or “extremely helpful” ratings):
• Teach English Language Learners (ELLs) (54 to 79 percent) • Teach Special Education students (48 to 67 percent) • Use formative assessment to inform reading instruction (52 to 70 percent) • Use formative assessment to inform mathematics instruction (50 to 67 percent) • Take a leadership role in my school (45 to 61 percent) • Plan lessons effectively (62 to 77 percent) • Work effectively with others in my learning team (55 to 66 percent)
The largest increases were for mentors’ help in developing teachers’ skills in using formative assessments to guide reading and math instruction and meeting the needs of ELL and Special Education students (see Figure 3B). These trends reflect NTC mentors’ efforts over the three years to direct their professional development support to the most pressing teaching challenges in RCSD. As described later in the section on professional development, NTC introduced a program to support ELL instruction during the second year, and in the third year re-directed their literacy professional development to more closely align with the district curriculum and to carefully scaffold teachers’ use of formative assessments to guide instruction with individual students. Through these efforts, combined with individual mentoring, NTC increasingly helped new and veteran teachers to meet the needs of struggling students in their classrooms. As described below, the mentors’ collaboration and learning community were important vehicles for their progress in adapting their work to the district context.
11
Teachers’ increased ratings of mentor help in developing their leadership and collaboration skills (Figure 3C) most likely reflect improved training for facilitators of grade-level learning teams, as well as mentors’ support of teachers’ roles in school leadership teams. These facets of NTC’s work in the district are taken up in a later section of the report.
Figure 3A. How Teachers Rate Mentors’ Help with Instructional Skills in 2007 and 2009
0% 20% 40% 60% 80% 100%
Select and adapt curriculum and instructionalmaterials
Design study units or lessons
Handle a range of classroom management ordiscipline situations
Plan lessons effectively
Use a variety of instructional methods
Teach your subject matter
Engage all students in learning
% Helpful and Extremely Helpful2007 N = 70 2009 N = 56
Figure 3B. How Teachers Rate Mentors’ Help with Formative Assessment in 2007 and 2009
0% 20% 40% 60% 80% 100%
Use formative assessment to inform writinginstruction
Create specific goals for individual students andmodify instruction accordingly
Use formative assessment to informmathematics instruction
Teach Special Education students
Use formative assessment to inform readinginstruction
Teach English Language Learners
Use formative assessment to informmathematics instruction
% Helpful and Extremely Helpful2007 N = 70 2009 N = 56
12
Figure 3C. How Teachers Rate Mentors’ Help with Developing Collaboration and Leadership Skills
0% 20% 40% 60% 80% 100%
Take a leadershiprole in my school
Work effectivelywith others in my
learning team
% Helpful and Extremely Helpful2007
2009
N = 70
N = 56
c
Consistently high teacher ratings of mentor help in developing instructional skills (Figure 3A) capture core strengths of NTC mentoring. Teachers provided numerous examples of skills that their mentor helped them to develop, such as using pictorials to help ELL students and incorporating reciprocal teaching into their instructional repertoire. Mentors frequently acted as a sounding board and troubleshooter for teachers—especially new teachers—by providing them with the needed tools, other resources and coaching to make effective use of these resources to strengthen their instruction. Mentors’ depth of content knowledge was particularly helpful to some. One teacher valued her mentor’s “amazing… science background… She knows so much…and helped us to do a unit on magnetism….” Another teacher singled out her mentor’s developmental knowledge of kindergartners as particularly helpful and all teachers valued their mentors’ feedback and ability to facilitate a process of reflecting on their instructional practice by “debrief[ing] what was working [well] and what [the teacher] needed to work on” after observing a lesson.
Teachers also reported invaluable mentor assistance in lesson development and pacing. As one middle grades teacher said,
“He’s [mentor] is really good at just helping me with the timing of my class because I’m one that tries to cram a week’s worth of lessons into one lesson. He’s really, really helped me learn how to chunk things up and put it into smaller pieces and pace myself at a pace that the students would take to and that was extremely valuable.”
Mentors combined ideas that help teachers think in new ways about their students with practical advice. For example, one teacher described how her mentor helped her establish a “community” in her classroom. This teacher described how she was able to create such a community by reconfiguring her students’ desks into table groups and because “[my mentor] showed me how students could work together and answer each others’ questions.”
13
Although not all mentoring relationships were highly successful, teachers learned from their mentors in numerous ways. And by all accounts – according to teacher reports and observations – they were putting instructional techniques into use that their mentors had coached them to use.
NTC mentors continued to improve their practice Through biweekly mentor meetings, formal professional development, and informal communication, mentors continued to build their own skills and knowledge. Such ongoing learning may have contributed to the increase in ratings teachers gave mentors on helping with instruction described above.
Biweekly meetings of all the mentors provided opportunities to discuss progress and challenges in mentoring and collectively brainstorm solutions. Mentor meetings typically began with charting progress and challenges at each school, allowing the identification of themes for mentoring that cut across schools. Other agenda items varied depending on which topics were most salient at the time, typically decided in advance by the mentors. Such topics included the role of mentors in learning teams, facilitator training, gradual release model, and BTSA portfolios. Meetings combined small group and large group discussion, often structured by protocols. Through information sharing and discussions, mentors expanded their repertoires of skills and strategies. Beginning in the second year, these discussions were enhanced through the regular attendance of one or more NTC site administrator coaches. Mentors were on site frequently, primarily working with teachers although lead mentors increasingly worked closely with site administrators. Administrator coaches were on site less frequently and worked exclusively with administrators. Pooling both perspectives in the meetings benefited both mentors and coaches. Mentors described these meetings as helpful, especially for learning about mentors’ work in other schools. As one mentor said, “I find them . . . useful for hearing what else is going on and what other people are doing, and some of the logistics of our program and districtwide issues to be aware of.” Another mentor commented: “The Wednesday meetings are great. We problem pose and solve. It works really well. I’m glad those keep happening, otherwise we are very isolated in our own schools. It helps us align a little more.” Mentors provided examples of lessons gleaned from their shared experiences. For example, several pointed to seeing the need to have more systematic mentoring around classroom routines and behavior management. They also recognized a need for more systematic training for facilitators on inquiry and expectations for the learning teams and for more mentor support in the learning team meetings. Individually, mentors described in some detail ways in which they had changed over the three years. One mentor said:
“[I’ve become] less instructional and more facilitative which was really hard for me to let go of. Really listening to teachers and helping them prioritize their needs. . . . Shifting from filling in the gaps to see how they can best organize themselves. It has been really good learning for me.”
14
Another said she had changed “big time”:
First and foremost, my own reflection of how I work in a group. The spirit of NTC which I don’t see in a lot of places is a whole valuing of everyone—what they bring to the table. Rather than one perfect right answer everyone has something to contribute and there is no definitive, it keeps changing according to the need.
In addition, mentors attended biweekly mentoring seminars at NTC which kept them informed about the Center’s tools and strategies. They also attended BTSA meetings which kept them up to date on changes in state requirements for induction. Oftentimes, mentors attended professional development sessions with their teachers, both to learn along with them and provide common ground for later conversations. For example one mentor described “one of the most amazing trainings I have had in a while” which was given by two NTC staff members on differentiation. Finally, mentors devoted considerable time to committee work, much of which involved designing professional development, which also expanded their own knowledge and design skills. Developing Teacher Leaders and Learning Teams A major goal of NTC’s work has been to develop teacher leadership and learning teams, a form of “professional learning communities.” Consonant with the research literature, these efforts are viewed as keys to building teachers’ knowledge and skills and collaboration to improve teaching and learning, as well as creating a professional environment that will contribute to teacher retention. In the fall of 2006, RCSD launched grade-level learning teams at each school, following the recommendation of a districtwide committee that met throughout 2005-06. The underlying idea was to locate more of professional development in the school in the form of learning teams, each led by a teacher facilitator. In theory, teachers would collaborate with their learning team colleagues in conducting “inquiry cycles” in which the team would focus on an instructional problem, decide on a strategy or intervention to try in their classrooms, collect information on students’ responses, share results with colleagues, and refine the intervention or go on to a new instructional problem. In 2006-07, the district design called for four inquiry cycles. In each cycle grade-level teams were to meet three times for two hours during Wednesday professional development time and a fourth time with all teams at the same grade level for a districtwide “share.” In 2007-08, the design was the same but reduced by one cycle for a total of three planned inquiry cycles. In 2008-09, learning team meetings were more sporadic and most teams completed only two cycles. Teacher facilitators were offered monthly training from NTC to prepare and support them in their new role in leading their learning team. During the first year, confusion over the role of facilitator and how they should be selected led to attempts to clarify selection criteria and refine facilitator training.
15
Facilitator selection and training improved over time During the first year of the learning teams, many of the facilitators were new teachers, leading to awkwardness in their role and some resentment from veteran teachers. In 2007-08, principals were to select facilitators based on their potential to lead their peers. The response from principals was uneven, and the result still reflected a preponderance of new teachers. By the third year, some teachers volunteered to be facilitators which had not happened initially and others with facilitator experience were willing to continue when asked by their principal or mentor.
Those chosen as facilitators were expected to attend monthly training sessions held after school for which they received a stipend. During the first two years, attendance was a problem, with approximately half the facilitators at most sessions and half who rarely attended any. By the third year, attendance increased to roughly two-thirds of the facilitators. Of the survey respondents who reported that they were facilitators in 2008, half rated the training as valuable or extremely valuable, a proportion similar to that in 2007. In 2009 the ratings were slightly higher: 58 percent rated the sessions as valuable or extremely valuable. Facilitators who had attended training the first year described an increased focus during the second year and a shift from group process skills to inquiry. By the third year participants in facilitator training, renamed the Facilitator Forum, noted that their needs were better addressed than in prior years, citing a shift from discussing how to run a group to the purpose of the learning teams and how to formulate an inquiry question and assess data. Participants reported that the sessions gave them a sense of what to expect and a clearer sense of what should be happening at the learning team meetings. One facilitator commented: “This year the meetings are run better and they’re able to tell us exactly what should be happening.”
Those we interviewed in years two and three all pointed to valuable skills or knowledge they had gotten from the training. Some appreciated learning how to prepare an agenda, some benefited from learning about norms. Others found these ideas less necessary because their team consisted of 2 or 3 teachers who knew each other and were working well together. One facilitator said, “I wasn’t in it last year but I learned a lot from [facilitator training] this year and we brought that back to our inquiry cycles and I feel like it’s been an education.” Another explained:
When we’ve had our professional development before, we usually go to these different meetings. Now you make your PD. You come together, I get my co-teachers and I say, “Okay, so this is our agenda” and before I’d be like, “I can’t write it.” So they showed us how to write our agenda and let’s go ahead and think of our focus question for this month and it’s just great. Whatever I did there, I feel empowered to come back and do over here with my learning team so that’s good.
At the end of the second year facilitators pointed to their need to understand more about inquiry and expectations for the work of the learning teams. Some also struggled with how to figure out what to do next when they identified areas in which student performance was weak—where to find the expertise to guide them to next steps. As one facilitator said: “I wish we had more of a way to get things if we needed them. . . I almost wish that there was somebody who
16
could be responsible for helping connect teachers with the resources they need to work on their inquiry question [like] research or books or videos.” This was particularly true for new teachers. Facilitators still struggled with these needs in the third year but felt that they understood much more about creating inquiry questions. Although facilitators who regularly attended the Forums in the third year gleaned a stronger sense of the purpose of learning teams, many continued to face the challenge of peers who were not keen on reflecting on their own practice. As one facilitator said:
I think that there’s a big problem when a lot of teachers in the district are not as open to change, and we’re trying to be trained to facilitate meetings that are looking at ways to improve classroom practice, and there are teachers who don’t think there needs to be an improvement in classroom practice. And I don’t think that was ever quite fully addressed in the Facilitator Forum meetings.
At the same time, facilitators expressed a sense of empowerment and community through meeting together regularly. One said: “I think the Facilitator Forum is one of the few places where teachers have a say in what’s going on in the district. And I don’t mean just some committee. I mean it really is the leadership committee for the whole district.” This facilitator went on to describe how questions are raised that have “direct immediate results in teachers’ classrooms.” Another facilitator saw the Forum as a mechanism for building community:
I think that it sort of also builds a sense of trust in the larger community of the district. Teachers tend to start to trust each other a little bit more, knowing that we all do share a common goal. And the common goal is ‘success for our children.’
Grade-level learning teams reported working together well
Over the three years, teachers generally gave their grade level teams high ratings on survey indicators of how well they work together. Notably, 80 percent or more agreed or strongly agreed that their interactions are mutually respectful and that they have tried out new ideas for instruction. These ratings of team functioning have remained relatively constant over the three years. Given little change in teachers’ overall reports on how well their teams worked together across the three years, we compare 2009 data for teachers in K-5 with those in 6-8 because these groups met in different configurations. Teachers of K-5 grades typically met as a grade level in each school, whereas middle grade teachers met in subject groups across schools (math and science teachers were grouped and English and social studies teachers were grouped). We wondered if these different arrangements might influence how the teams functioned. Figure 4 indicates that K-5 teachers report stronger agreement than their 6-8 colleagues (differences greater than 10 percent) on four items:
• using assessment data to identify areas for improvement (81 versus 68 percent) • understanding the goals for learning teams (80 versus 64 percent) • sharing commitment to working together (77 versus 66 percent) • effective routines for doing this work (72 versus 61 percent)
17
Figure 4. How Grade-Level Learning Teams Worked Together: K-5 versus Gr. 6-8
0% 20% 40% 60% 80% 100%
We have identified areas for improving our instruction
We have learned from one another about effective teachingstrategies
We have developed effective routines for doing this work
We have discussed what happened when we tried out new ideasfor instruction
Our facilitator does a good job of guiding our work anddiscussions
Team members attend scheduled meetings regularly
We have developed good ideas to improve instruction
We share a commitment to working together
We have tried out new ideas for instruction
Our interactions are mutually respectful
We understand the goals for grade-level learning teams
We have used student assessment data to identify areas forimprovement
% Agree and Strongly Agree
Gr 6, 7, 8 N = 53K-5 N = 95
Differences in favor of middle grade teachers were few and less than 10 percent in three areas: mutually respectful interactions, identification of areas for improving instruction, and learning from one another about effective teaching strategies. These comparisons suggest that middle school teachers struggled more with developing common assessments to guide their work, not surprising given their subject and grade-level diversity. The bigger theme is that well over two-thirds of the teachers surveyed reported strong agreement with a series of statements designed to capture the purpose and desired functioning of learning teams. At the same time, interview data indicate that grade-level learning teams varied enormously in their ability to work together constructively, across grade levels and schools. This is consistent with the literature on professional learning communities which suggests that learning teams, like teachers, go through a series of developmental stages. In some cases, team members took great pride in being part of what they described as a “professional model”:
We’ve actually set goals, followed through with them, feel really professional…like this is a real meeting and we’re actually doing something for the kids and here is a product…this is the result we want at the end so how are we going to map it out and how is it going with our…as soon the next meetings come along…and it’s great.
Other grade-level learning teams did not function well. As one facilitator commented:
18
This has been a rough year because the chemistry hasn’t been good. Some of teachers, they’re like lone rangers. They want to do it on their own and . . . they would be doing work on their own while some of use were engaged. It’s great that we’ve had an opportunity to meet but it’s hard, especially as a facilitator.
Each year, some teams needed to reconstitute themselves by combining teachers across grade levels or schools; for example, in one case, high teacher turnover and reliance on substitute teachers in two grade levels led to a decision to combine grade levels. Grade-level learning teams tackled a wide variety of problems The extent to which grade-level teams created inquiry questions, and the nature of the questions they tackled, varied widely across teams. According to survey respondents, the three activities carried out most often by learning teams were: planning cycles, analysis of student assessment data, and looking at student work. In terms of NTC and RCSD goals for learning team activities, planning is the least desirable because it often is does not reflect an inquiry question towards which assessment data and student work are analyzed. Over time the teams decreased their use of meeting time for lesson planning, from 87 percent reporting this activity in 2007 to 78 percent in 2008 and 67 percent in 2009. Consistently high proportions of teachers (around 70 percent each year) reported that their learning team looked at student work and analyzed student assessment data. Roughly one-third each year reported that their learning team did lesson study as one of their collaborative activities. Facilitators provided examples of the work of their learning teams that illustrate both successes and ongoing struggles. For example, one said, “Our focus on writing was because we were looking at our Reading Lions assessment and realizing that our students were performing poorly on writing, and there was a huge group that we could potentially move from ‘below grade level’ to ‘grade level’.” Another described agreeing on a rubric and comparing results:
We look at the assessment, how the kids do, compare percentage points. Like, we agree on a rubric we’re going to grade, we all grade on the same rubric, and we see how our kids did and what the problems were, if there’s anything we need to go over again, what are common things among not just class but among the grade level.
Another facilitator described how their learning team used the released test items to guide their analysis:
We have an organized agenda, I think we stick to the agenda, at the end we come out with certain outcomes that are very helpful. . . This year we looked at the release questions. . . in each unit and said, “What are the kinds of questions that are being asked? What are the frames of these questions? And we have to teach to these frames, we have to give the students the basic procedural skills to attack them, the basic vocabulary to understand what the question is asking, and then of course the ability for kids to apply those things to different question frames.
19
The following example captures one facilitator’s explanation of the challenge their team faced in trying to determine exactly what students need to learn when they perform at very low levels—how to figure out where to start to build the stepping stones to the larger goal. She said:
We started to say, then what do we need to do so that we don’t have this big huge insignificant goal when they can’t even do some of the little steps. So how do we use the assessment to make the little steps and bring them, you know, closer and closer and closer to them so that there’s a direct relationship from what we’re assessing to what we’re teaching. . . So that’s the way it started this year and then after, we sort of agreed that we needed to know that, then [mentors] afforded us with opportunities to practice doing assessment that was significant to everybody in the group. So the question wasn’t specifically like, “How did you teach synonyms?” The question was, “How do we do assessment that’s significant for our instruction?” So from that, we got to be able to spend our Wednesday afternoons in a meaningful, constructive way from something that everybody in the group wanted and needed and then…so there was action and there was evidence and there were all the things that proved that from our own needs, we can make something.
Many learning teams reached the limits of their capacity to design good solutions to the problems they identified. Some were able to draw upon mentors’ expertise. Others did not have that resource. The example below highlights the challenges some teams face as they try to figure out next steps in the absence of specific expertise on the team, both around instruction and assessment.
The first cycle we worked on [was] vocabulary and we had a question…I don’t remember exactly how it was phrased…but like ‘where the cognitive breakdown is in tests in terms of vocabulary’. Like, they took a test and they did fairly poorly on it. It was one of the quarterly assessment tests. And, in general, they didn’t do very well on it. And I was looking at it thinking, “I have a feeling there are a lot of words in here my students don’t know. Like, is that why they didn’t get this right?” They know what similes are, but they got these ones wrong—and it has to do with a simile. Or whatever. So we picked out the words that were difficult and gave them a test with only those words—just like a vocabulary test. … but it was just a very simple like “this word most likely means…” and it had like three definitions. And they did really well on it. So we realized, “Okay, so it’s not those words. They know what those words mean in isolated form. So, like, at what point do we lose them?” So we did that for the first cycle. And it was informative but we kind of didn’t come up with like an answer really. It was really kind of left open. Like, “Okay, well we know it’s not the one word.” And then we’d put them in paragraphs and they still did well. … So we still didn’t really figure out at what point they’re getting lost, but it still was a valuable process. . .And just the part that we did figure out was like, “Okay, well they do know more words than I thought they do—at least when they’re isolated.” So I still think it was useful. But we did kind of struggle with this inquiry question—like, “How do you actually, like, answer it?”
In a similar vein, another facilitator said: “One thing I thought was really missing was a way to get resources if your group decided that you wanted to go into an areas where you needed more information. . . like having a professional library or just a place where people can start compiling ideas so . . . if you want to do Reciprocal Teaching and you know nothing about it, so you know where to go to get that sort of thing.”
20
Interestingly, we found little change in teachers’ value ratings of different learning team activities from 2007 to 2009. Survey responses reflected roughly the same percentage at the end of each of the three years, all hovering around 70 percent.
Learning teams were challenged by lack of time and uneven support from administrators The initial idea that grade-level learning teams would implement multiple cycles of inquiry each year ran up against a host of competing agendas. Site leadership positions, professional development opportunities, and district expectations as well as the daily demands of class preparation competed for teachers’ time and attention. This competition for time was intensified by federal and state requirements, particularly Categorical Program Monitoring in the second year and activities associated with Program Improvement status in the third year. In the second year teachers noted that the time for their inquiry cycles was impeded by scheduling problems and competing demands for their time. In at least one of the three inquiry cycles, the break between meetings was several weeks resulting in lost momentum. One of the three also lacked a district share. One teacher described numerous last minute changes, “. . . we don’t know exactly when we’re going to have the time to meet as a group. We may plan and then all of a sudden . . . that’s become a site day or that has become another training.” Another teacher noted a similar problem:
This year was also difficult because of how the meetings were split up. We’d have like two, and then there’d be two weeks of site stuff or district stuff, and then we’d have out third meeting, and then we’d have district share-out. . . How they were scheduled last year wasn’t necessarily a lot better but the fact that we met more was definitely a lot better. I think we got a lot more out of it because of that.
One facilitator noted, “Some of the meetings were so far between that you go to a facilitator training and it sounds great and maybe you’re curious, but then there’s this huge gap of time before you actually have to use anything.” In a few cases, learning teams or subsets of teachers chose to meet on their own time more frequently. In one case, two learning team members who taught the same courses met six days a week to build weekly and daily lesson plans together and discuss strategies for students at different levels of proficiency. Other teams met weekly. These teams found that such frequent meetings greatly enhanced what they were able to accomplish and learn from each other. These problems intensified in the third year as a result of directives from, and district responses, to the DAIT team. Meetings mandated for district teachers were scheduled at the last minute, often during the time set aside for learning team meetings. Only two cycles were completed and each was interrupted by competing demands on scheduled meeting time. Facilitators from different schools reported different levels of support from their administrators, ranging from those who valued the teams and honored the learning team time to those who kept their distance. In some cases principals sat in on learning team meetings—a signal that the work was valued—while others rarely or never did so. Most principals were
21
described as wanting the learning teams to be successful although some scheduled competing meetings when learning teams were scheduled. In 2009, only 42 percent of survey respondents reported that their principal often or always “supports the development of adult learning communities”—a decrease from 58 percent in 2007. The district was named as the primary culprit for scheduling meetings during learning team time, particularly as the demands of Program Improvement escalated. In our 2009 survey, only about one fourth of district teachers agreed that “there is adequate time for teachers to work together.” Principals generally characterized their schools’ learning teams as a mixture of strong and weak teams, with variation in collaboration among team members, the quality of their inquiry questions, and consistency of attendance. Most had an example of a team rendered dysfunctional by personality clashes. Some principals pointed to the importance of continued training, believing that “we’re not there yet to carry this on our own . . . we’re not done with the pieces that will help us continue.” Others worried about how to create well-functioning team with mixed messages from the district and NTC. Most perceived NTC’s focus to be on process and a long-term view of changing the culture of teaching, in contrast to the district’s focus on urgency and pressure for results. One consequence was district office guidance to principals to collect agendas in advance of meetings and minutes afterwards. This approach signaled more of a focus on compliance with requirements than on instructional support for teachers and was implemented unevenly by principals. In only one school did we hear from facilitators that they received feedback from their principal on the agendas and minutes, although others reported that their site administrators had attended at least one meeting of their learning team. Teachers with newly developed expertise assumed leadership roles In addition to teachers who took on leadership responsibilities as facilitators for their grade level learning team, teachers leaders also emerged in other areas as a result of expertise gained from professional development, mentoring, and teaching experience. Teachers who developed a body of knowledge and instructional strategies in a particular area, and had the skills and desire to work with others, took on both formal and informal leadership roles. For example, one teacher was mentored for three years, attended several literacy professional development series, and went through facilitator training twice. She was a natural candidate to become a literacy coach at her school. Even prior to taking on that official role, she was someone to whom her colleagues turned for help on literacy strategies. Another teacher who attended ExLL and GLAD training was interested in becoming a GLAD trainer. Both these teachers, and many others who exemplified strong instructional practices, served as role models for their peers as principals and mentors brought other teachers to observe their classes. A new form of teacher leadership emerged at the end of the 2008-09 school year. Unhappy with the direction of the DAIT team demands and perceived heavy-handedness of the district response, teachers organized districtwide and presented a petition to the RCSD school board detailing their concerns about the district staff person in charge of curriculum and instruction and requesting that person’s resignation. That almost all teachers in the district signed the petition—which delineated the gap they perceived between state standards for instructional leadership and the district leader’s actions—and that several spoke publicly to the board, demonstrated an unprecedented level of teacher leadership in RCSD.
22
Developing School Leadership Strengthening school leadership for improving teaching and learning has been a major goal of NTC’s work with RCSD. Building the capacity of school administrators to provide instructional leadership and to create conditions that support teacher learning is challenging in the face of multiple agendas and turnover in site leadership. Also, context conditions limit principals’ ability to take on new roles. Faced with issues of school safety and violence, facility needs, a high proportion of new staff, and students with a wide range of social and emotional needs, principals are hard pressed to find the time and energy to be instructional leaders. Each of the three years has seen turnover of school leaders and even turnover of schools. Of the seven schools we began studying in 2006-07, one was dismantled and a second was added when a charter school’s charter was not renewed. Four of the six schools that remained throughout the three years saw at least one change in leadership. NTC’s primary strategy for supporting site leadership development was to assign NTC coaches to work with all principals and assistant principals throughout the year. Most of the administrative coaches were former principals working part time with NTC; each was assigned to work with two RCSD schools for up to one day per week. They provided individual coaching and facilitated the creation of a learning team of principals, all targeted to creating the space and the tools needed to strengthen their instructional leadership. Principal coaches and mentors provided a range of supports to new and veteran principals. NTC coaches worked with principals individually to help set up systems intended to free them to focus on instruction. Coaches varied in the amount of time they spent in their assigned schools, because of both individuals’ availability and the principals’ interest in working with a coach. For some principals, the time with their coach was ample; others, especially those new to the job, wished for more. To some, their coach filled a critical need for moral support and a trusted confidant. One principal said, “It’s been wonderful to have someone I can bounce ideas off of.” Another said, “She is there for me; does what I need.” For others, time with their coach was not deemed productive either for lack of trust or perceived mismatch between needed skills and those provided by the coach. In general, principals highly valued the resources their coach brought, from articles and materials to videos to use in staff development with teachers. Principals new to the job particularly valued critical feedback about their work and help in getting their arms around all the structures they needed to set up, from their Leadership Team to procedures for looking at data. One said: “She’s so good at that [critical feedback]. . . and that’s how I can learn.” Mentors, especially the lead mentors, spent far more time in the schools than the administrative coaches so they were also able to provide ongoing support to administrators as needed. In fact, the shift from multiple, undifferentiated mentors to naming a lead mentor in each school during 2007-08 facilitated both communication with and support for the school leaders. In one school, for example, the lead mentor helped the principal and assistant principal learn more
23
about using formative assessments to guide instruction. In another school the principal described her relationship with the lead mentor as a collaboration, citing examples in which the mentor organized events, supported learning teams, and brought important issues to the principal’s attention.
Principals were consistently positive in their reviews of mentors’ work in their schools, both with teachers and at the school level where mentors took on tasks of training, organizing book rooms, and videotaping classes, among other activities. “It’s great work,” said one principal. “Magnificent,” another said. Another principal described regular meetings with the lead mentor and literacy coaches to discuss student assessment data and implications for instruction. Another praised the mentor’s willingness to help with anything, including designing professional development for specific topics and particular teachers where the principal saw a need.
The degree to which principals communicated with mentors varied considerably across schools. Similarly, we observed large variation across the schools in the degree to which the principal incorporated teacher mentoring into a larger school vision. One principal described plans to broaden mentors’ impact by incorporating them into the leadership team so their work could have a schoolwide impact. For example, the principal might notice that one grade level or course was weak on certain skills or knowledge but rather than asking a mentor to work with one or two teachers, the principal might ask the mentor to design a professional development that all the relevant teachers would learn from. Both principals and mentors expressed concerns about the eventual departure of NTC mentors and how that void could be filled. One principal explained: “The [NTC’s] individual teacher support is huge; my administrator and I just can’t get to all the teachers.” Another expressed worry about maintaining the amount of quality mentoring teachers would get in the future after NTC leaves and pointed to the importance of their independence: “NTC’s mentoring is not tied to any accountability system to always report back to someone in an evaluative position.” Principals struggled to develop and sustain their own learning community. The first year saw the beginning of a collegial community among principals, representing a major shift from relationships previously characterized as competitive and individualistic. NTC coaches launched a process called Quick Visits in which principals and coaches gathered at one school, divided into teams, and observed 10-12 classrooms over the period of an hour. The observations were guided by a topic proposed by the host principal, one tied to teaching standards such as student engagement, and followed by discussion with feedback to the principal and composition of supportive notes to each teacher visited. During the second year, Quick Visits focused more on specific instructional strategies, including English language development strategies and reciprocal teaching. Over the year the Quick Visits segued into a new district requirement that principal observations be structured by state standards that were embedded into a Palm Pilot recording system. The shift reflected increased pressure on the district central office to “accelerate” learning, driven by their Program Improvement status. Principals were asked to make a certain number of classroom visits each week and send the data at regular intervals to the district office. Principals had mixed reactions to
24
this shift to a broader range of topics for observations and feedback to the district. In any case, the earlier emphasis of Quick Visits on positive communication from principal to teachers around instructional practices disappeared. By the third year, the district redefined walkthroughs and dissolved the principals’ group meetings with NTC coaches, reportedly in response to some complaints that principals were off site too much. The new approach involved having a district team do the walkthroughs with the principal at each school, thus removing the collaboration element of the earlier designs for classroom visit. However, near the end of the year, the district re-instituted principal visits to each other’s schools. As one principal said:
It was great to see each other again in that content. . . I don’t necessarily want to return to something where every month I’m going and reviewing one of my colleague’s sites. But I also don’t want to have it where a whole year goes by and I don’t see someone’s site either.
This principal went on to describe the importance of trust and shared understanding of the purpose of the walkthroughs, pointing out that the half-day spent with NTC coaches helped with that. In contrast, the district visits were viewed by teachers with suspicion. Principals were not clear why the shift from the NTC facilitated meetings occurred but presumed that it was a district office decision.
During the third year, some principals continued to meet together on their own. The assistant principals, or administrative managers, also met as a group with an NTC coach. Several met monthly throughout the year focusing on special education requirements and ways to streamline the process. District efforts to promote instructional leadership through group professional development were hampered by high principal turnover, since those with more experience had different learning needs than those new to the district. For example, in the second year professional development on effective instructional practices for English learners presented by a district staff person and a coach was well-received except by those who had already had such training. In the absence of training that differentiates among principals more and less knowledgeable about the topic, some sought their own consultants who could better meet their needs. Yet when the group had met regularly, most principals felt they benefited from the discussions, particularly as they evolved to focus on data-based instructional leadership. The group tackled pressing issues facing the district under Program Improvement, including how to reconcile the need to follow the adopted programs yet supplement them with rich instructional strategies that can engage students who are falling behind. Given the coaches’ goal of creating a common vision of what good instruction looks like and the pressure from the state to focus on the adopted textbook, principals found themselves in the middle of conflicting philosophies. As we describe later, all the players continued to grapple with the need to reconcile these different viewpoints.
25
Although principals valued time together, it was difficult for them to maintain momentum and trust among themselves in the face of chronic annual turnover of principals. In a small district like RCSD, principal turnover poses a serious challenge to developing a professional learning community for school administrators. Not only must they continually form new relationships amongst themselves, but each new principal has priorities that pull away from time with administrative peers – developing relationships with their faculty and learning the ropes of their school, and of the district if hired from outside. These factors, as well as concerns among veteran principals over the politics of firing and hiring in the district, inhibited the development of trust and sustained learning within the RSCD principal community. Providing, Brokering, and Collaborating on Professional Development Formal professional development has been a key element of NTC’s work with RSCD. In addition to mentoring teachers, helping to establish grade level learning teams and training facilitators, and coaching site administrators, NTC leaders have brokered external workshops and extended training for teachers; and they have designed and led in-house professional development sessions with central office staff. Both the district office and NTC have been working toward the goal of arming teachers with a range of effective intervention strategies to strengthen literacy instruction and instruction across the board for English learners. NTC professional development continued to be highly valued Across all three years, teachers had options to attend a variety of professional development offerings, which ranged from those with multiple sessions throughout the year to one-time conferences. Some were brokered by NTC and lead by an outside professional developer; others were run by NTC mentors. In the first two years, NTC brokered training for grades 1-3 teachers in Comprehensive Early Literacy Learning (CELL) and for grades 3-6 teachers in Extended Literacy Learning (ExLL). Most district teachers participated during 2006-07 and/or 2007-08 and gave it very high ratings. In each year’s survey, roughly 80 percent of participants rated these series as valuable or extremely valuable. Literacy professional development in 2008-09 built upon these programs to align directly with the district literacy program, discussed below a “a new model” for NTC professional development in the district. New teachers seeking their credential obtained training on the BTSA standards and requirements through the New Teacher Teaching and Learning Seminar run by NTC mentors. This training was also attended by other (non-BTSA) first and second year teachers to involve roughly half of the survey respondents at some time during 2008-09. Three-fourths of the participants rated it as valuable or extremely valuable, an improvement over the prior year in which 60 percent gave such high ratings. Figure 5 shows teachers’ ratings for this training, as well as all of the kinds of professional development available to district teachers during 2008-09. New teachers’ comments in interviews confirmed the positive survey ratings. One teacher said, “All of the professional development that I’ve attended has been really beneficial. It’s been essential in these first couple of years, and I’m looking forward to taking more. I’m pretty impressed with the options available and the classes I’ve taken have been really great.” Another
26
commented: “I did a CELL professional development last year also and so this year, it was much easier for me to just start the year and incorporate a lot more best practices.” And another teacher noted that, “They just offer so many opportunities for everyone to take.” Veteran teachers were equally positive. As one said, “I can tell you the one thing that’s keeping me here is NTC and mentors and professional development.” Another veteran noted, in response, “And even though I would stay no matter what, I think, it makes my life a lot more enjoyable.” At the same time, a new teacher pointed out that in some ways such powerful professional development is a double-edged sword for the district. She said. “They offer all of this great PD like CELL and all of these things and then you basically are wrapping this up like a perfect package to go some where else . . .” Both NTC and RCSD worry about making large investments in teachers who then leave. At the same time, it is these investments that have the potential to influence them to stay beyond their first year or two.
Figure 5. Teacher Value Ratings of Professional Development, 2009
0%
20%
40%
60%
80%
100%
GLAD (GuidedLanguage
AcquisitionDesign)training(55% )
New TeacherTeaching and
LearningSeminars
(51% )
ClassroomIntervention
(Small GroupInstruction)
(52% )
Foundations inLiteracy and
LanguageSeminars
(Saturdays)(39% )
Facilitatortraining(51% )
Open Courttraining(58% )
ELA StrategicPlanning
Sessions (bythe DAIT team
on February26th, 27th,and March4th) (67% )
% Valuable and ExtremelyValuable
Extremely Valuable
Valuable
Note: Percents in parentheses indicate the percent of respondents who participated in each type of professional development.
Training in Guided Language Acquisition Design (GLAD), introduced in the second year and involving over half the teachers by the end of the third year, garnered the highest ratings by participating teachers. Eighty percent rated GLAD training as valuable or extremely valuable, most indicating that it was extremely valuable. In addition to the highly rated CELL and ExLL trainings, GLAD was particularly appreciated by teachers struggling to meet the needs of their EL students. The program provides curriculum and instruction in academic language and literacy
27
through science and social studies content.7 RCSD teachers were particularly enthusiastic about the GLAD trainers’ approach of providing a four-day demonstration training in local classrooms while teachers observe. We describe some of its features in more detail below. Efforts to link external professional development to curriculum gained traction With state pressure to implement the adopted curriculum, principals and teachers expressed confusion about the relationship between their Open Court curriculum and the NTC-sponsored training they received and valued. Tensions between the respective philosophies of instruction espoused by district office staff and NTC leaders were no secret. During the second year, NTC struggled to define the relationship between the mandated curricula and the kinds of instructional strategies embodied in CELL, ExLL, and GLAD, for example. From the NTC perspective, the professional development offered “best practices” – instructional strategies proven to be effective. From the district office perspective, the fact that training did not made explicit the link to the core curriculum undermined its implementation and led to some of the confusion expressed by teachers. This position solidified during the third year under intensified pressure from the state in the form of the DAIT team to teach the core curriculum with fidelity, following district-developed pacing guides. In the second year, NTC’s strategy to bring coherence to their professional development and the mandated curriculum was to frame the curriculum content as the “what” of instruction and the strategies that help students learn the content in the curriculum as the “how.” They argued that since the skills and knowledge of many students lag behind their grade level, additional supports are critical, both in how the material is presented and how interventions are defined and implemented. Moreover, no core curriculum is without weaknesses. Open Court, for example, has documented shortcomings in writing and comprehension and for English learners. Although this distinction was clear to NTC staff, it was not clear to everyone in the central office or in the schools. In spite of persisting confusions, however, GLAD appeared to exemplify a mutually beneficial relationship between RCSD and NTC, initiated when an NTC mentor identified and arranged for GLAD training in one district school. Enthusiasm spread to other schools via teachers and mentors, and to the district office. As a result of this groundswell of interest in GLAD, the district funded GLAD training in the spring of 2008 for 30 teachers. Since each needed a substitute teacher, this represented a substantial investment of resources. One district administrator noted that GLAD was “so well received” by the teachers since it “aligns with the district’s focus on English Language Learners,” they funded an additional GLAD training over the summer. In the district’s 2008 summer school, The ELD Academy offered a four-week session in which six district teachers who had been trained in GLAD trained used GLAD units to teach students science and social studies content. In essence, this provided a practice opportunity for teachers. By the end of the summer, 76 teachers in the district had received GLAD training and more training sessions were planned for the following year. During 2008-09 teachers were 7 GLAD was field tested for nine years and was the recommended K-8 project by the California State Superintendent of Schools for teachers of English learners (http://www.projectglad.com, 2008).
28
beginning to incorporate GLAD strategies into their instructional approach, a task made easier by GLAD’s focus on science and social studies, subjects not as salient to Program Improvement requirements which focus on English language arts and mathematics. Both the district and NTC made significant investments in GLAD. Mentors participated in GLAD training so they could help teachers incorporate the instructional strategies into their classrooms. One mentor started a support group for teachers; others worked with individual teachers to ensure they could implement the units or incorporate particular GLAD strategies such as the use of pictorials into their instructional repertoire.
The after-school programs represent another arena exemplifying constructive partnering between RCSD and NTC. An NTC mentor worked with district staff to strengthen the instructional component of the after-school programs, working toward the concept of an extended-day program with small group instruction. During 2007-08, an average of two teachers at each school site attended training in Classroom Intervention (CI)/ Small Group Instruction, based on Reading Recovery methods, and applied that training in the after-school sessions where they taught their own students. They also helped to train other teachers in their school on using the CI techniques with individual students in their classrooms, and many chose to continue sharpening their own skills in small group instruction during summer school. In our spring 2009 survey, roughly half of the respondents indicated that they had participated in CI professional development and nearly three-fourths rated it as valuable. A new model for literacy professional development was launched in the third year NTC’s literacy professional development evolved to better meet district and teacher needs in 2008-09. After the initiative’s second year of providing CELL/ExLL training, NTC created and implemented a new professional development model designed to be coherent with the district curriculum and to prepare teachers to meet developmental needs of a diverse student population. The new model for literacy training in RCSD went a long way toward resolving long-standing tensions between NTC’s work in the district and demands on teachers to implement the adopted literacy program.
Although teachers had been learning new instructional strategies and were enthusiastic about their CELL/ExLL, its links to the core curriculum were not explicit. This meant that the NTC-brokered literacy professional development did not equip teachers with the knowledge and skills they needed to use Open Court materials well. As one mentor explained:
One of the things that we found is that we were training people in strategies for CELL or ExLL but they weren’t making the connection on how to use those strategies in their core curriculum. And so our goal this year was to do that, to take the strategies and show teachers how to embed them into their core curriculum.
With this in mind, a group of NTC mentors designed Foundations in Literacy and Language (FILL), the overarching goal of which was to prepare teachers to use new instructional strategies with the Open Court curriculum they were using. At the same time, NTC emphasized a view of literacy that encompasses all subject areas, not only language arts: “The literacy lens is not a subject –you can use it to teach reading, writing, math, science.”
29
Contributing to this shift in emphasis was NTC’s attempt at the beginning of the third year
to create demonstration classrooms. The mentors’ search for teachers who could model CELL/ExLL literacy strategies revealed that use of the CELL literacy strategies by district teachers was uneven and incomplete. According to one mentor, they discovered that
. . . there wasn’t the capacity for one person to have control over all the elements of balanced literacy. There were pockets and pieces of really good work that teachers were doing in one area, but not any one that had it all. So that kind of framed where we went from there. This collective observation by the NTC Literacy Committee members led them to focus on
a few best practices and to emphasize the use of “gradual release” as a framework for lesson planning and for assessing student learning needs. The notion of gradual release captures the role a teacher plays in guiding the development of student learning until a skill can be performed independently. This notion capture’s NTC overarching goal: that students become independent learners.
To scaffold teachers’ use of a gradual release approach to instruction, NTC developed a
“Blueprint” that features “helpful habits of reflection in designing effective instruction.” This professional development tool outlines stages of gradual release: 1) activating learners’ prior knowledge; 2) modeling through direct instruction a “new teaching point” or strategy; 3) facilitating student practice and 4) releasing for independent practice. Each FILL session focused on all developmental stages of instruction.
The FILL model also expanded leadership for improved literacy instruction in the district.
NTC mentors led the lesson for each FILL session along side literacy expert Adria Klein, who had run the prior CELL/ExLL training, thus expanding professional development leadership and creating closer links to teacher mentoring. Further, each session showed videotapes of district teachers using a featured instructional strategy in their classrooms. Many teachers of varying grade-levels from the seven district schools were videotaped. The videotaping gave these teachers an elevated status among their peers as “good” and “knowledgeable” teachers, even though many were beginning teachers. In a district with so few experienced teachers and so many beginning teachers, NTC made teaching expertise in the district visible and therefore accessible to colleagues. For the roughly 40 percent of teachers who regularly participated in the Saturday FILL sessions, experimentation with the modeled literacy strategies and reflective practice began to become the norm.
Finally, this promising new model of professional development influenced how the NTC
mentors thought about mentoring. Although they had always customized their assistance to the particular needs of an individual teacher, the focus was not necessarily aligned with particular instructional strategies. NTC came to recognize that often “we teach classroom engagement and classroom routines and other parts of new teacher mentoring…as isolated practices. And we don’t get as much power out of them.” Instead, NTC began to guide mentors to connect effective teaching practices to literacy content: “Everyone realized that the coaching had to have a literacy lens.” For example, “you can talk about classroom management with a literacy lens
30
involved…. And so you can have kids…figure out how to do Turn-and-Talk [literacy development strategy] as an engagement strategy.” Over time, mentored teachers began to use classroom management routines “in connection with a literacy practice,” to make connections between instructional techniques for engaging students and designing instruction based on a gradual release model.
The evolution of NTC’s literacy professional development in RCSD is testimony to this
organization’s learning from its practice in a district. It also provides evidence of the challenges that a high-quality intermediary organization faces in working with a struggling district and the critical need for customizing support to develop coherent resources for teacher learning and instructional improvement.
31
Trends on Intended Outcomes
The intended outcomes of NTC’s work in RCSD are to develop teachers’ knowledge and skills and to improve conditions of teaching in the district in order to retain effective teachers beyond their first years of teaching and to steadily improve student achievement. Measures of change in these intermediate and ultimate outcomes from spring 2006 to spring 2009 are based on: annual teacher surveys, teacher and administrator interviews, faculty rosters for district schools, and grade-level data on students’ performance on California’s English and mathematics tests for 2006, 2007, 2008, and 2009.
• Teacher knowledge and skills for instruction and for school leadership. NTC’s mentoring of teachers, as well as professional development brokered or designed by NTC, should result in teachers’ growth in instructional skills and leadership. We used teachers’ self-ratings on our annual survey to gauge change over time on measures of these outcomes.
• Conditions of teaching. NTC and RCSD aimed to establish conditions in schools that
support continuous improvement of teaching and learning including: shared leadership, collaboration on instruction, and teacher knowledge development and sharing. We used teacher survey measures of these school culture outcomes, as well as district support of teacher learning and instructional improvement, to track change over time.
• Teacher retention. Teacher retention is an important leading indicator of district
improvement, since high teacher turnover in the past has constrained RCSD’s ability to improve teaching quality and build teacher leadership. In measuring retention, we excluded teachers whose contracts were not renewed (due to poor performance ratings), who moved into non-teaching positions in the district, and who retired or left the labor force for other reasons.
• Student achievement. The ultimate outcome of district improvement is to increase
rates of students performing at grade-level standards in English and mathematics. To track change, we assess trends in the percent of students who scored proficient on the CST at each grade, as measured against the trend for all California schools.8 We analyze grade-level trends for each school on percent scoring at proficient levels, the state’s criterion for judging AYP. We analyze trends for three cohorts of district schools: the two schools that had NTC mentoring beginning in 2003-04 (Chavez and Green Oaks), the school that began in 2005-06 (Willow Oaks), and other district schools that began in 2006-07.
8 We used this strategy because CST grade-level tests have been revised several times during this time period. Therefore, changes in scores at any grade level can be due to changes in test difficulty, and California grade-level trends are needed to adjust for this source of change. If RCSD students’ gain is greater than expected based on the state-wide gain, we conclude that achievement has improved.
32
The following sections address each of these intermediate and ultimate outcomes in turn. Teacher Knowledge and Skills for Instruction and Leadership To assess growth in teachers’ instructional skills through mentoring and related professional development provided or brokered by NTC, we examine change in teachers’ self-ratings on particular facets of instruction and use of formative assessments to identify students’ learning needs. These analyses assume that growth in the extent to which teachers report feeling “well prepared” or “very well prepared” to carry out teaching practices – such as engaging students or teaching ELL students – result largely from their learning through professional development. The trend data provide both an overall assessment of teacher learning outcomes and a look at areas of greatest and least growth. We also assess trends in district teachers’ self ratings on ability to take a leadership role in their school. NTC worked in multiple ways to develop leadership among new and veteran teachers through: mentoring, professional development to build teachers’ expertise and ability to share knowledge with peers, and learning team facilitator training, NTC coaches’ and lead mentors’ work with principals also promoted the development of a culture of shared leadership in district schools. These trend data capture individual teacher outcomes for the range of capacity-building efforts underway in the district. Teachers’ sense of instructional competence has grown Survey data on teachers’ feelings of being well prepared for classroom instruction show a consistent pattern of small gains over the four years (see Figure 6). Gains of more than 10 percent are shown for: handling a range of classroom management situations (from 64 percent in 2006 to 79 percent in 2009), using a variety of instructional methods (from 73 percent to 84 percent), 9 and teaching your subject matter (from 84 to 94 percent). These outcomes are consistent with the focus of professional development for new and veteran teachers that NTC provided or brokered as well as the focus of mentoring for new teachers. It is impossible to know how much of the gains in instructional competence and confidence among the teachers are due to the effects of mentoring and professional development, as opposed to learning that comes from time in the classroom and through interactions with colleagues in and outside the school. However, as we describe in earlier sections, teachers give very high ratings to professional development brokered or provided by NTC; and new teachers credited their mentors’ help, particularly in developing effective classroom management skills, planning lessons, and engaging their students in learning.
Since 2006, increasing teacher involvement in NTC mentoring and related professional development, coupled with the district’s move to grade-level learning teams and emphasis on
9 This growth pertains to teachers’ skills in filling gaps in the adopted curriculum and text, e.g., introducing writing instruction into Open Court, and adapting instruction to struggling learners, e.g., using GLAD materials for ELLs.
33
teacher collaboration, has made it possible for new RCSD teachers to have steep learning curves and greater success than in the past.
Figure 6. Change in Teachers’ Self Ratings on Instructional Skills
0
20
40
60
80
100
Design unitsor lessons of
study
Handle arange of
classroommanagementor disciplinesituations
Select andadapt
curriculum andinstructional
materials
Use a varietyof instructional
methods
Plan lessonseffectively
Engage allstudents in
learning
Teach yoursubject matter
% Well Prepared or Very Well Prepared
2006 N ~= 140 2007 N ~= 138 2008 N ~= 101 2009 N ~= 139
Teachers learned to use formative assessments to address student learning NTC mentoring and professional development in writing and reading instruction aimed to develop teachers’ use of formative assessments and their ability to use the data to identify student learning needs and focus instruction. The most striking gains in teachers’ self-reports on instructional skills are in the area of formative assessment, a central focus of NTC’s work in the district. These gains are directly parallel to increases in how teachers rated their mentors on their help with formative assessment (see Figure 3B in earlier section).
Survey data show strong gains in teachers’ self-ratings on these skills between 2008 and 2009, consistent with the introduction of FILL during 2008-09 (see Figure 7). In one year, self-ratings of being well-prepared to “use formative assessment to inform writing instruction” grew from 62 percent to 80 percent; for reading the growth was from 71 percent to 84 percent. Gains
34
shown by the survey data are also strong for “use formative assessment to inform mathematics instruction (from 66 percent to 85 percent), suggesting that teachers have carried formative assessment skills developed in literacy into their instruction in other subjects.10
Figure 7. Change in Teachers’ Self-Ratings on Skills in Using
Formative Assessments and Individualizing Instruction
0% 20% 40% 60% 80% 100%
Teach Special Education students
Teach English Language Learners
Create specific goals for individual students andmodify instruction accordingly
Use formative assessment to inform writinginstruction
Use formative assessment to inform readinginstruction
Use formative assessment to inform mathematicsinstruction
% Well Prepared or Very Well Prepared
2006 N ~= 1402007 N ~= 1382008 N ~= 1012009 N ~= 139
Growth in teachers’ sense of preparedness to teach English Language Learners (from 73 percent in 2006 to 84 percent in 2009) likely reflects teachers’ involvement in GLAD training during the last two years. As described earlier, GLAD supports instruction of English Learners, and teaching ELLs garnered the largest increase in teachers’ ratings of their mentor’s help in developing instructional skills from 2007 to 2009. Finally, trends in teachers’ self-ratings document their persistent struggle to teach the special education students in their classes: less than half of district teachers felt well-prepared in each of the four survey years. Teachers across the spectrum of experience, and especially beginning teachers, are stretched to meet challenges presented by the court-ordered district policy to mainstream all special education students. Many have one or more extremely disruptive students who tax their ability to create order and instruction for all students. Although one NTC mentor provides support to special education teachers across district schools, RCSD’s special 10 As described earlier, the Blueprint for “gradual release” in literacy instruction that NTC developed for FILL training is meant to generalize to any subject. This tool might well have supported teachers’ increased skill in using formative assessment in their mathematics instruction. The district also introduced a new formative mathematics assessment.
35
education department has been chronically understaffed and teachers lack adequate onsite support for addressing the challenges. Unless district policy for full inclusion of special education students changes, much more intensive system strategies and supports would be essential to shifting the burden away from individual classroom teachers and improving their success. Teacher leadership expanded NTC’s work in RCSD emphasized the importance of developing broad teacher leadership in all schools and in district professional development. NTC aimed to develop teachers’ instruction skills and leadership in sharing their expertise with colleagues, which they pursued through mentoring, facilitator training, and professional development. Further, district and school policies and structures introduced after 2006 set conditions for expanding and distributing teacher leadership in the system – in particular, through grade level learning teams with facilitators and involvement of teachers in school leadership teams. Overall, teachers’ self ratings on preparedness to take a leadership role in their school has remained quite high over the four years – ranging from a low of 55 percent in 2007 to 62 percent in 2009 (see Figure 8). Over the same period of time, new teachers’ ratings of mentors’ help to take a leadership role increased from 45 to 61 percent (shown earlier in Figure 3C), suggesting that mentors contributed to teachers’ growing comfort with positions of leadership. Despite the overall lack of growth in teachers’ reported leadership preparedness, the expansion of leadership positions meant that large proportions of the district’s teachers had the opportunity to use and further develop their leadership skills. Nearly 40 percent of surveyed teachers had been grade-level facilitators and/or members of their school leadership team at some point since 2006-07 (55 of 141 respondents to our 2009 survey). As the last bar in Figure 8 shows, a very high proportion (82 percent) of these teacher leaders felt well prepared to take a leadership role in their school. This no doubt reflects the selection of promising teacher leaders into the positions, but it also reflects leadership skill development through facilitator training, mentoring, and experience. Although not tapped directly by the survey items, evidence of teacher leadership emerged in three other arenas. As described earlier, teachers mobilized around a petition drive to call the school board’s attention to their views of failed district curriculum leadership, which resulted in the resignation of the head of curriculum and instruction. In another unanticipated situation, a principal was dismissed mid-way through the school year, leaving the school without a principal for a period of time. Teachers responded by taking on responsibilities and contributing to the management of the school. The third example of teacher leadership was evident in the FILL professional development sessions, for which several teachers volunteered to be videotaped to provide examples for their colleagues to discuss.
36
Figure 8. Teachers’ Self Ratings on Ability to Take a School Leadership Role: Change and Breakout for Teachers Who Were in Leadership Positions
* ratings for teachers who were members of leadership teams or were team failitators in any of 2007, 2008, or 2009 (N = 55)
0 20 40 60 80 100
2009*
2009
2008
2007
2006
% Well Prepared or Very Well Prepared
The district-wide effort to develop teacher leadership is reflected in this second-year teacher’s comment:
There is a lot of expectation that you will take a position and be responsible for making a decision and everyone from each grade level needs to have someone on the leadership team and on the PBS team and… everyone is asked to go into a position where they probably have no experience with anything and just go straight in the first day on campus. So… we’re put into leadership roles pretty quickly.
Schools vary in the extent to which they have developed conditions of shared leadership
and opportunities for teachers to develop skills in supporting their school’s improvement, as elaborated in the next section. Nevertheless, growth in teachers’ exercise of leadership in the district since 2006 is a notable outcome of NTC’s work with the district. School and District Conditions of Teaching The logic model underlying NTC’s work in RCSD posits that collaboration with the district to support professional learning and improved instruction will create a more favorable and productive professional culture for teachers. At the school level, this would translate into greater shared school leadership, teacher support of colleagues, collaboration on instruction, and
37
knowledge sharing and development. At the system level, it would mean enhanced principal instructional leadership and district support of teachers’ professional learning and instructional improvement. In turn, teachers would increasingly perceive their school and the central office as supportive and rewarding environments in which to work. Consequently, teacher retention and student achievement would be expected to improve. We describe our analysis of change in school and district conditions of teaching in RCSD between 2006 and 2009 along four dimensions:
• Basic working conditions in district schools • School professional culture • Principal leadership • District professional culture
To assess overall trends, we examine teacher survey data aggregated across all district schools, acknowledging that some outcomes vary widely across schools. This summary highlights trends consistent with the underlying logic model, as well as unanticipated declines on some of the desired outcomes. Physical conditions of teaching and after-school program quality improved Given the dismal condition of school physical plants and yards, dearth of instructional resources, and chaotic after-school programs that characterized RCSD in 2003-04, NTC’s work with the district included efforts to establish basic conditions of teaching and learning. NTC worked with community partners to fix up school physical plants and yards, set up resource rooms, and partnered with the district to develop an instructionally rich after-school program.
Basic working conditions in district schools improved substantially, particularly during the project’s early years. In the two schools with which NTC began working in 2003-04, improvements to the physical plants were made prior to our 2006 baseline survey of teachers in all district schools. Nevertheless, the survey data document a positive trend on indicators of teachers’ basic working conditions. Improvements were most notable for teachers’ access to materials (see Figure 9). In 2006, only about half of the teachers (54 percent) agreed that “necessary materials such as textbooks, supplies, and copy machines are available as needed by staff” and by 2009 the proportion had increased to 70 percent. We also see positive trends for teachers’ pride in the physical appearance of their school (from 60 percent to 70 percent) and agreement that they have a resource room to support their instruction (68 percent to 78 percent. Conversely, teachers’ dissatisfaction with working conditions in their schools declined over time and significantly because NTC leaders and mentors worked with the schools to improve them. For example, the proportion of teachers who strongly disagreed that necessary materials were available to staff declined from 17 per cent in 2006 to just 6 percent in 2009.
38
Figure 9: Teacher Ratings of Selected Indicators of School Working Conditions
0%
20%
40%
60%
80%
100%
This school has a high-quality after-school
program
Necessary materialssuch as textbooks,supplies, and copy
machines are availableas needed by the staff
I am proud of thephysical appearance of
my school
This school has aresource room orlibrary to support
instruction
% Agree or Strongly Agree
2006 N ~= 140 2007 N ~= 138 2008 N ~= 101 2009 N ~= 139
Further, teacher ratings of their school’s after-school program improved substantially – from 43 percent agreeing that their school has a high-quality after school program in 2006 to 60 percent in 2009. This shift reflects the sustained collaborative effort between an NTC mentor and the district coordinator of after-school programs to establish a curriculum and teacher training for the programs, a major accomplishment given the need to coordinate with multiple after school providers as well as teachers. The benefits of this work were felt especially during the past couple of years, as the programs became more focused on instructional strategies in which teachers had received training. Improvements in these basic school conditions supported teaching and learning through teachers’ access to essential instructional resources and higher morale and through students’ extended and enhanced learning opportunities after school. School professional cultures became more collaborative
We tracked change on several dimensions of school professional culture that research associates with strong educational outcomes and that NTC aimed to develop in district schools.
39
Survey measures that tap each dimension combine teacher responses to several individual survey items (see Appendix B for items and reliability coefficients for each “scale”). These indicators of school culture include:
• Shared leadership, or teachers’ active roles in decision making, trust in school administrators, and collective problem solving;
• Collaboration on instruction, or teachers’ responsibility to help one another do well, use of time together to discuss teaching and learning, and discussing lessons that didn’t go well;
• Colleague support, or teachers’ mutual support to try out new ideas and comfort in seeking advice from colleagues
• Teacher knowledge development, or the extent to which teachers spend time together examining student data and work, sharing teaching ideas and materials, and sharing their knowledge of evidence from research.
The bars in Figure 10 represent the district average of teacher ratings of their school conditions (mean of school means) on each measure for each of the four years. The highest school mean is indicated by a + in the graph and the lowest is indicated by – within the bar. The figure captures both overall trends across district schools and the range of school ratings across schools for each year.
Figure 10. Teacher Ratings of School Professional Culture
1.0
2.0
3.0
4.0
5.0
2006 2007 2008 2009 2006 2007 2008 2009 2006 2007 2008 2009 2006 2007 2008 2009
Shared Leadership Collaboration oninstruction
Colleague Support Teacher KnowledgeDevelopment
Mean of School Means Highest School Mean Lowest School Mean
40
Trends on each of the school culture outcomes show significant progress between 2006 and 2009.11 School scores on all indicators, except for Colleague Support which was relatively strong in 2006, increase substantially from low levels in 2006. In particular, average school scores on Shared Leadership move from a mean of 2.9 (below a neutral rating of 3) to 3.5, a significant move into positive teacher ratings during the first year. As significant were increases for Collaboration on Instruction during the first year and Teacher Knowledge Development during the second year, most likely reflecting the development of grade-level learning teams in the first year and subsequent efforts to deepen the work. The developmental trajectory for building grade-level teams that learn to improve student achievement is not well understood. However, a close look at trends for the eight items that comprise the Teacher Knowledge Development scale reveal the direction teachers moved to focus their collaborative work. Teachers’ reports on how frequently they do various things with colleagues reveal significant increases only for the items on analyzing student work and assessment data. The proportion of teachers who report “analyzing samples of work done by our students” at least once a month increases from 47 percent to 57 percent; the proportion of teachers who report rarely (never or once or twice a year) “discussing student assessment data to make decisions about instruction” declines from 28 percent to 14 percent. Changes in teacher collaboration in the direction of using student work and data to inform instruction are consistent with NTC’s emphasis on formative assessment and also with federal and state accountability pressure on the district to raise student test scores. These trend data paint a picture of gradual overall improvement in the professional culture of district schools. Early strides are shown for shared leadership, collaboration, and colleague support; change on each of these indicators occurred primarily during 2006-07. The flattening of trends on all indicators after 2007-08 indicates both that improved school professional cultures were sustained – and that progress was stymied – during 2008-09 when many teachers perceived a significant decline in district support for their professional growth and success with students (see below). Teachers’ experience of principal leadership varies widely across schools Teachers’ overall survey ratings of principal leadership across district schools have not changed much since 2006 (see Figure 11).12 Principal turnover in district schools has been pervasive over the four years of the evaluation, with teachers’ experiences of their leadership going up and down on an annual basis. As a result, teachers’ ratings of their principal’s leadership have varied widely between schools in any given year and widely within schools between years. Figure 11 shows the very wide range of high to low school scores on principal leadership for each year.
11 Our analysis of change examines the difference between the overall RCSD school mean for 2006 and 2009. We consider a change significant if it is greater than one standard deviation (sd) from the 2006 mean. This means that the 2009 average school score moved up by as much or more than the range of school scores in the middle two-thirds of the district distribution in 2006. For each of the school culture indicators, change in the district average was more than 1 sd of the 2006 average of school scores. 12 The Principal Leadership scale is composed of 11 survey items (see Appendix B). Statistically, there was a .5 sd increase between 2006 and 2007, and the current level is slightly below that.
41
Over the three years of NTC’s work with all district schools, we see positive overall trends on three of the eleven principal leadership survey items. Between 2006 and 2009 teachers rated their principals higher on: using data to inform decisions (from 66 percent to 81 percent agreeing), conveying high expectations for all students (64 percent to 77 percent), and cultivating a shared vision and common purpose among staff (50 percent to 61 percent). These specific trends in principal leadership focus are not surprising in the context of increased pressures they have faced to raise student test scores to meet accountability pressures on the district. Whether and how a principal supports instructional improvement is also a source of variability in teachers’ ratings of their principals on other leadership indicators, such as encouraging teachers to be learners and supporting the development of teacher learning communities. Finally, a key source of variability in teachers’ ratings of principal leadership across years is turnover in principals; four of the six schools included in the three-year analyses had changes in principals during this period.
Figure 11. Teacher Ratings of Principal Leadership, 2006 to 2009
1.0
2.0
3.0
4.0
5.0
2006 2007 2008 2009
Principal Leadership
Mean of School Means Highest School MeanL S h l M
Teachers’ ratings of the district rose and then declined dramatically in 2008-09 Teachers’ survey ratings of the district culture have been strikingly low over the course of the project (see Figure 12). Apart from a spike during 2006-07 in positive ratings for professional development support – from 34 percent in 2006 to 52 percent in 2007 when grade-level learning teams were adopted – ratings on district professionalism indicators have ranged from about 20 percent to 40 percent over the four years. Large and small gains on several
42
indicators during the first year of the project were promising, and a view that the district was becoming a better place to teach came through consistently in teacher interviews at that time. During the early years teachers expressed excitement over the changes in professional development that NTC brought to the district. A veteran teacher described the change in these terms:
I never saw professional development like this before. I didn’t learn what professional development was until we started really working with the NTC and with the inquiry cycles and as a facilitator. Because before you’d go to these meetings and I thought they were just, “Oh, yeah, the district wants us to go listen to some stuff because it’s required because we’re a PI school.” It wasn’t really, “What are you doing to enhance the kids’ education? Where are you meeting them?” And not until I started becoming a part of [inquiry] and was a facilitator for my group that I started thinking, “Hey, this is actually for me to develop my kids’ academic success further. What can I do to further it?”
Teachers’ judgments that the district culture was improving – through grade-level learning communities and quality professional development in literacy – are reflected in their 2007 and 2008 survey ratings on “promotes professional development.”
Figure 12. Teacher Ratings on District Professionalism Indicators
0.0 20.0 40.0 60.0 80.0 100.0
Helps my school focus onteaching and learning
Supports my school’s wholeschool change effort
Is improving its support forteaching and learning
Ensures that student learning isthe “bottom line” in this school
Promotes the professionaldevelopment of teachers
Is committed to high standardsfor every student
Holds high expectations for ourschool
% Agree and Strongly Agree
2006 N ~= 1402007 N ~= 1382008 N ~= 1012009 N ~= 139
43
When the district was designated “Program Improvement” in 2007-08 under the No Child Left Behind accountability provisions, state intervention in the form of a District Assistance and Intervention Team (DAIT) began in the spring of 2008. Required to implement “corrective action” plans in 2008-09, the district pressed teachers to implement the curriculum with fidelity, follow a rigid pacing plan, and drop the kinds of innovative instructional strategies for which most had received NTC-sponsored professional development. In interviews, some teachers expressed the view that communication from central office administrators conveyed distrust or disapproval rather than support for their efforts to improve their students’ achievement. Several teachers reported feeling threatened and demoralized by the attitudes and tactics employed by district staff. In response, almost all district teachers signed a petition and delivered it to the RCSD Board of Education documenting their dissatisfaction with the district leader of this approach, who resigned immediately.
This experience is clearly reflected in teacher survey data. Declines in teacher ratings of district professional development were dramatic – from 56 percent in 2007 to 27 percent in 2009, well below the 2006 rating. Ratings also declined on “district is improving its support for teaching and learning” (from 39 percent to 21 percent) and on “supports my school’s whole school change effort” (from 29 percent to 19 percent).
The trend data document growth and decline in teacher confidence in district leadership for instructional improvement. Growth was during early days of the project in which NTC had taken over some functions of the central administration. Decline occurred when the district in collaboration with the DAIT team took charge, largely rejecting advice and support from NTC responding instead to pressure from the state to raise test scores quickly. A high rate of principal turnover in the last two years undermined continuity at the school level, and teachers perceived that the district was not responsive to their school’s needs during the transitions. Teacher Retention A primary goal of the initiative was to increase teacher retention in the district. Investments in building the capacity of new teachers does not hold long-term benefit for the district if they leave for teaching positions elsewhere. A certain amount of teacher turnover is inevitable—some leave for personal reasons, others leave because their contract is not renewed. Many leave the Bay Area because they cannot afford the cost of living. To estimate teacher retention in RCSD, we tracked all teachers who left the classroom annually, based on each year’s schools rosters, and determined their reasons for leaving through interviews with lead mentors at each school. Although we lack baseline data, NTC reported that teacher retention rates had been as low as 25 percent for the two schools they began working with in 2003-04. Based on teacher rosters for all district schools in 2005-06, we observe a retention rate of 68 percent between 2005-06 and 2006-07, prior to NTC’s work with all district schools 13. Because mentors were our source of information on teachers’ reasons for leaving, we could not make that determination for those who left at the end of 2005-06. However, we can compare the overall rates: Over the three 13 This rate was considerably below the regional average for teacher retention for 2004-05. National norms from the most recent Schools and Staffing Survey indicate that 81 percent of teachers in West Coast districts stayed in the same school between 2004 and 2005, considerably higher than RCSD’s rate of 68 per cent between 2005 and 2006.
44
years, RCSD’s overall retention rate ranged from 73 percent to 77 percent, slightly up from the 68 percent we computed for the baseline year. To get a better reading of the district’s ability to retain classroom teachers as of 2006-07, we adjusted the retention rate to exclude teachers whose contract was not renewed because they did not meet quality standards. We also exclude those who left teaching for reasons of retirement, pregnancy, career change, or to take a non-teaching position in the district. We include those who left the district for other teaching jobs, either in the region or elsewhere. Results of this tabulation are shown in Table 3 (see Appendix C for detailed data by school). The adjusted retention rate for teachers over the past three years has been 89 percent annually. Nevertheless, when all reasons for attrition are taken into account, the retention rate is only about 75 percent. In other words, the district must replace nearly 25 percent of its teachers each year.
Table 3. RCSD Classroom Teacher Attrition, 2006-07 to 2009-10
Reasons for leaving teaching in RCSD
2006-07 to
2007-08
2007-08 to
2008-09
2008-09 to
2009-10
Left teaching, stayed in the district 1 3 0 Personal reasons, including retirement 10 5 3 Contract not renewed 9 8 13 Left to teach in a charter school in the district 0 1 2 Left to teach in another Bay Area district 3 6 3 Left the region 8 5 6 Don’t know 4 2 3 Total number of classroom teachers 131 130 131 Percent who left the classroom for any reason 27% 23% 23% Retention rate based on shaded rows 89% 89% 89%
Note: The table includes data for classroom teachers only. Not included are Special Education teachers and other non-classroom teaching staff, for whom turnover was considerably higher: 39% of 36 left the first year, 46% of 41 left the second year, and 40% of 42 left the third year. Teacher turnover at McNair is not included. Long-term retention of classroom teachers is considerably lower than the annual rates might suggest. Of the 131 teachers in 2006-07, only about half (51%) were still teaching in the district three years later (2009-10). For classroom teachers who received NTC mentoring (70 of the 131 teachers), retention was even lower: 36 percent remained in the classroom. The rate is nearly identical for the subset of 32 mentored teachers who were in their first and second year of teaching in 2006-07 and who were not in the Teach for America program, which stipulates a two-year commitment: 34 percent were still teaching in the district in 2009-10.
45
Special education teachers had greater turnover and extremely low three-year retention rates. Of the 11 specialists who received NTC mentoring in 2006-07, only 4 (36 percent) remained the following year and just 1 (9 percent) was still teaching in the district in 2009-10. Given the court decree to implement a full-inclusion model where all special education students are placed in regular classrooms, this turnover rate has significant repercussions for classroom teachers who are faced with challenging students for whom little and inconsistent support exists Although teaching conditions generally improved over the years, teachers who recently left the district for teaching jobs elsewhere still pointed to frustrations associated with student discipline problems—many of which result from the court-ordered full inclusion model for special education—and management conditions. In 2006 and2009 surveys, 64 percent of district teachers agreed with the statement “It is stressful to be a teacher at this school.” In 2009, almost half of the teachers (49 percent) agreed that “I think about transferring to another district.” In exit interviews conducted by NTC’s formative evaluator during the past four years, teachers complained of management conditions such as being reassigned to a different grade level without apparent reason or consultation, not getting essential support from the principal, not receiving recognition for their efforts, and feeling that some in the district office do not respect their teaching knowledge and skills. Relatively low pay and benefits also were cited as reasons for seeking a teaching job in another district. Teachers who left at the end of 2008-09 further expressed deep dissatisfaction with district administrative policies and practices that constrained their ability to teach effectively. Some said that their hands had been tied, that the district prohibited them from using professional judgment needed to succeed with their students. Notably, half of those who left but remained in teaching moved to charter or private schools. Nonetheless, some teachers who left the district because of family moves or to pursue another profession, as well as many who stayed, commented on improved teaching conditions and the positive role played by NTC. Several told us that without their NTC mentor they could not have survived the struggles of their first year of teaching. One teacher who ultimately left the district commented that NTC mentoring and professional development “…was definitely a factor in the last several years. That’s why I stayed.” Another commented on the critical role her mentor played in supporting her persistence and growth between the first and second year:
I think definitely the New Teacher Center is the only reason why I came back the second year or one of the main reasons. I would have collapsed and not made it… She was always pushing me [saying] ‘Try it again. See how a second year will be different.’ I mean it’s a tremendous, tremendous difference from last year to this year and all the things they offer to support their [teachers]…
Yet whatever effect NTC’s work with the district has had on teacher retention, trend analysis indicates that the outcome has not changed since 2006-07 when the project expanded to include all district schools. Student Performance on California Standards Tests To address the question of whether the NTC-RCSD partnership is making a difference for students’ academic performance as measured by the CST, our evaluation compared trends on the English and math CST for students in schools that worked with NTC for different periods of
46
time. We expected that schools with six years of mentoring (Cesar Chavez and Green Oaks) and with four years of mentoring (Willow Oaks) would show greater gains than the other three district schools with three years of mentoring. Prior research on instructional improvement efforts finds that student achievement outcomes often do not show up until the third year, so we should see improvement in all district schools by 2008-09. However, our expectations are tempered by the finding that teacher retention did not increase which, according to the logic model, was intended to be a major factor influencing student academic performance. Our analysis of student achievement outcomes since NTC began working with RCSD uses performance levels on the English and math California Standards Test (CST) from 2002 through 2009. We look at the percent of students scoring at or above Proficient, since this is the criterion the state uses to evaluate district performance for accountability purposes under NCLB (Appendix D includes data for percent scoring Basic and above). For each school, we compare grade-level performance over time, using two years prior to NTC mentoring as the baseline.14 Thus, we use different baseline years to assess trends: 2002 and 2003 for Cesar Chavez and Green Oaks, 2004 and 2005 for Willow Oaks, and 2005 and 2006 for Belle Haven, Costano, and Flood. Data for 2009 are used as end points for all schools. We compute California norms using the same baseline years as those used for each of the three NTC-mentoring cohorts of district schools.
For each of the NTC cohorts, Table 4 displays the baseline percent scoring at or above Proficient by school, grade-level, and subject and the percent in spring 2009. Percent increases equal to or greater than the average grade-level gain across California for the same time frames are indicated by an asterisk.
The overall picture is one of variation in gains in percent scoring at or above Proficient across grades, subjects, and schools with a slight advantage to those with more than three years of mentoring experience (which also have somewhat lower baseline scores). The two schools with six years of mentoring show increases that meet or exceed the statewide gain in 4 of 13 comparisons (31 percent). The school with four years of mentoring shows such increases in 8 of 13 comparisons (62 percent). The three schools with three years of mentoring show increases in 14 of 39 comparisons (36 percent). A more striking result is that only two of 26 comparisons for the veteran schools (those with more than three years of mentoring) show decreases in percent proficient (8 percent), while 12 of 39 comparisons for the schools with three years of mentoring show decreases (31 percent). Overall, trends for most grade levels in most schools fall below state average gains, and this pattern tends to increase across grade levels.
14 We use two pre-NTC years’ scores to increase the reliability of student performance measures and make it more likely that results capture stable trends.
47
Table 4. Grade-level Trends on English and Math CST Scores by School and Years Working with NTC: Average Percent Students Scoring Proficient and Above from
the Pre-NTC years to 2009 Grade Level School Subject 2 3 4 5 6 7 8 Mentoring began in 2003-04
English 10 6 3 19 4 19 4 14 Cesar Chavez Math 18 10 5 25* 7 36*
English 8 17 8 13 16 20 Green Oaks Math 18 41* 13 41* 18 19 Mentoring began in 2005-06
English 9 43* 9 25* 19 27 12 13 9 23 17 40* 15 34*Willow Oaks Math 15 69* 24 72* 20 50* 10 4 11 5 28 48* Mentoring began in 2006-07
English 16 29* 9 21* 19 30 18 16 12 20 26 23 17 27*Belle Haven Math 31 60* 26 36* 16 27 17 7 13 18 28 28
English 62 56 15 26* 23 18 28 15 23 21 22 47* 11 14 Costano Math 75 84* 26 34* 19 10 27 7 16 27* 9 59* English 42 53* 32 38 33 39 34 25 31 41 36 24 33 29 James
Flood Math 51 57* 52 58 26 36 23 11 18 38* 32 20 State 2009
English 53 44 61 54 52 54 48 Math 63 64 66 57 49 43
Note: Percentages are rounded to the nearest whole number. Grades 4 and 5 moved from Cesar Chavez to Green Oaks, hence pre and post percents are shown in different cells. * Grade-level gain was equal to or larger than statewide gain.
Three-year trends at the school level for all students and English learners (EL) look somewhat more positive. Table 5 shows that five of the six district schools had average math gains for all students and for ELs that were greater than state average gains between 2006 and 2009. Moreover, the 2009 percentages of ELs proficient in math approach the state average for ELs. The pattern is similar but not as dramatic for rates of proficiency in English, where three schools show gains greater than the state for all students, and two of these for ELs as well.
48
Table 5. School Trends on English and Math CST Scores by All Students and English Learners (ELs): Percent of Students Scoring At or Above Proficient from 2006 to 2009
Note: Bold 2009 numbers indicate that the increase from 2006 to 2009 is equal to or greater than the state increase for the same time period and grade spans.
Despite evidence of progress on student outcomes across district schools, the trend data
do not show dramatic rates of acceleration in student achievement. NTC mentoring may have had strong effects on teachers’ success with their students in any given year, but the fact that only 36 percent of teachers mentored in 2006-07 were still teaching in the district three years later dampens its long-term effect on student achievement. Given that high teacher turnover is a reality that RCSD faces, a strategy to build district capacity for improved student achievement through individual teacher development is inherently limited, as recognized by the initiative’s broader theory of change. Perhaps most important for the district’s long term success were NTC’s success in getting basic school resources for teaching in place, helping to expand teachers’ leadership roles in schools and the district, and charting a new course for literacy professional development in the district.
% EL English Math 2006 2007 2008 2009 2006 2007 2008 2009 Belle Haven All 18 18 18 25 19 18 22 29 Belle Haven EL 80 13 16 14 23 18 17 20 28 Costano All 29 27 25 30 26 25 25 35 Costano EL 65 25 22 24 24 25 24 26 33 Flood All 32 30 33 36 23 26 26 35 Flood EL 35 30 24 26 29 19 27 21 35 Willow Oaks All 17 19 24 30 20 26 34 41 Willow Oaks EL 81 13 16 23 26 20 25 32 40 Cesar Chavez All 17 17 14 18 25 21 36 41 Cesar Chavez EL 83 13 14 14 15 23 19 36 38 Green Oaks All 11 14 13 18 28 22 29 31 Green Oaks EL 93 12 13 13 18 29 21 29 31 State All 45 46 48 52 48 49 51 54 State EL 25 26 29 33 35 36 39 43
49
Lessons and Issues for Capacity Building Initiatives In the fall of 2006 the Hewlett Foundation, NTC undertook an ambitious multi-year effort to expand their work in RCSD to strengthen teaching and learning throughout the district. This initiative to build district system capacity was carried out during tough times and in a most challenging context. During the first year of the initiative, the district was rapidly losing money from a combination of state budget reductions, the creation of charter schools, and the costs of its court decree for full inclusion of special education students in all classrooms. The student population brought challenges rooted in a high-poverty community in which an increasing proportion of families do not speak English. Professional capacity was compromised by inexperienced leadership and staff at all levels of the system – from the school board to the classroom; moreover, high turnover continued at every level during the course of the initiative. Complicating these challenges, in the middle of the second year the state named RCSD a Program Improvement district and assigned a DAIT team that became another inexperienced player in a leadership role.
This was the playing field on which the NTC-RCSD initiative attempted to fundamentally shift how the system functions to build capacity to retain teachers, strengthen their teaching skills, and increase student learning. Administrator turnover at all levels of the system added to the challenges of developing effective change strategies and partnering relationships. NTC worked regularly with central office administrators to establish mutual goals but found limited traction for collaborative work; and their partnering with school administrators was undermined by the churn and uncertainties of district decisions around principalships. In fact, over the three years NTC mentors played important partnering roles in maintaining forward momentum in several schools.
Without a doubt, clear benefits accrued from the initiative. Prior to NTC’s involvement,
RCSD’s history was one of low performance, distrust among stakeholders, and poor conditions for learning in the schools. During the three years of our evaluation, we documented significant improvements in school conditions, including enhanced resources for teachers and refurbished buildings and grounds. We also documented substantial improvements in teachers’ access to high-quality professional development, including mentoring and formal sessions brokered or designed by NTC. In addition, we observed a major expansion of teacher leadership roles in district schools, including early stages of teacher learning team development. Still, we saw limited progress on the ultimate goals of increasing the retention of new teachers and improving student performance.
Nonetheless, the initiative provides a rich knowledge base for learning about the
conditions under which a capacity building strategy can work in urban districts. Knowledge about the formation and execution of effective partnerships between external providers and districts undertaking system reform is sparse. With intense pressure on districts to improve, charter schools as a strategy have taken the lion’s share of research attention. Less attention has focused on the question of how a district can develop its capacity to continually improve school quality through partnering with external organizations. How does a district partnership work effectively to change not only what happens in schools and classrooms but also what happens at
50
the system level? What conditions enhance the likelihood of a constructive partnership? How can an effective partnership be constructed in a struggling district like RCSD?
The particular experience of the Hewlett Foundation, NTC, and RCSD in this initiative
helps pinpoint a series of questions and dilemmas that may be useful for foundations, external partners, and struggling districts, as well as for future research on district reform. They include the quandary of investing in building the capacity of individuals in a system who are unlikely to stay in the system and the related question of how guidance for instructional improvement provided by an external partner can be sustained after external funding ends. The experience also calls attention to potential tensions between the philosophy, agenda, and even culture of the district and external partners, particularly in struggling districts during times of heightened accountability pressures from the state. Both the challenges and accomplishments of NTC’s work with RCSD inform these fundamental problems of district capacity building initiatives.
What does capacity building look like in a high-turnover district? An approach that features developing individual human capital is bound to disappoint in a system that loses professionals at high rates annually. The initiative’s hypothesis that intensive mentoring and support for new teachers would lead to higher rates of retention did not pan out over the three years we documented teacher turnover in RCSD schools. For example, six out of ten teachers mentored by NTC during 2006-07 had left the district by 2009-10 and all but one of the principals had been replaced. Human capital built was not wasted, since students undoubtedly benefited by having teachers who were learning to improve their instruction. Moreover, most of those who left remained in the profession. However, this approach is not enough to meet the goal of building and sustaining capacity needed in systems like RCSD. Another conception of capacity building centers on developing organizational supports for teaching and learning, a task made more challenging in the absence of strong district support. Capacity in this view includes structures and tools that scaffold teacher and administrator learning and that address challenges presented by high turnover. NTC addressed this through its efforts to institutionalize grade-level learning teams and to improve the quality of professional development in the district. RCSD’s policy for grade-level learning teams was a step in this direction, however the infrequency and unpredictability of meeting times and the small numbers of teachers in many teams was not sufficient to generate a critical mass of teacher expertise to support learning for improved instruction. Moreover, the design initially lacked the focus, structure, and tools essential to communicate the purpose for learning teams and to scaffold and sustain teacher learning. Because few teams have the opportunity to mature in a high-turnover district, tools and predictable structures become particularly important. Tools such as the Blueprint NTC developed to guide literacy instruction can be a powerful system resource for teacher teams as well as for classroom instruction, provided the concepts are adequately introduced to teachers. Such tools can also be vehicles for developing a district instructional culture focused on student learning.
51
Research typically identifies high turnover as a problem to be solved. We suggest a parallel line of inquiry that takes turnover as a given in many urban school systems and investigates the kinds of structures and resources that promote continued adult learning in the face of changing casts of characters. The most significant capacity developed in RCSD through this initiative was new norms and practices of teacher leadership and collaboration, yet the district lacks mechanisms and resources to support teacher teams’ ongoing learning. What does “sustainability” mean in a capacity-building partnership? During the third year of the NTC-RCSD initiative, mentors and project leaders grappled with the question of what would be sustained from their work in the district. At one level, sustainability might be represented by district or other sources of funding to continue to hire NTC to provide mentoring to new teachers. However, this is unrealistic in tight money times, and the initiative intended more—that it would leave ways of doing business, of teaching and preparing teachers, that could continue after NTC’s departure. How realistic is this ambition?
In our view, the question centers on whether school and district professional cultures embrace and carry the principles for practice that NTC nurtured through its mentoring and professional development. For example, have teachers and administrators developed a deep understanding of the idea of “gradual release” in instruction, and has this become a ‘taken-for-granted’ principle for instruction in RCSD? Will veteran teachers initiate new teachers to this key idea and be able to introduce the Blueprint tool as “the way we do things in RCSD”? Do school and district leaders understand, and will they promote, the key instructional principles and strategies that NTC introduced during the course of their work with teachers? Will they ensure that structures are in place and tools accessible to support the practices?
These questions make clear that learning at all system levels is essential if a district is to take on new ways of doing business. District and school administrators and leaders cannot support the effort unless they share the same goals and are learning along side teachers and collaborating with the external partners to make the learning usable in their school contexts. Without shared goals and openness to learning, there will be no organizational commitment or medium for spreading and sustaining the learning and principles for practice. Further, if district leaders do not understand the principles for instructional improvement that a partner promotes, they cannot be strategic in selecting an guiding additional other partners to complement and build on the effort.
Further research to capture the process of developing sustainable principles and guidelines for instructional improvement might focus on districts that have sustained an effective focus in their improvement efforts over several years. Extant case studies of such districts point to a multi-level system learning community—or nested learning communities—focused on key ideas for teaching and learning, as well as leaders’ strategic attention to the developmental arc of the district’s improvement effort. How these conditions for sustainable system improvement are established is an important direction for future research.
52
How does a support organization adapt its “model” to meet district needs? One key to sustainable, capacity building district reform is the ability of external partners to be strategic in adapting their successful practices to context and readiness conditions of a particular district. An organization like NTC, with a well-developed model for mentoring beginning teachers and a national reputation for quality, faces a dilemma in partnering with a struggling urban school district. Can it adhere to its model and philosophy with fidelity and simultaneously adapt to particulars of the district context of teaching? NTC struggled with this tension, especially in not wanting to compromise its stance on quality instruction by embracing the district’s Open Court literacy program and a district pacing guide that put forth a lock-step curriculum and limited teachers’ discretion to adapt to their particular students. Yet the mentored teachers had no choice but to use the mandated text and those who did not faced sanctions from the district.
Over time, NTC mentors in RCSD developed ways of adapting their work to the district context that maintained the integrity of the organization’s core principles and met teachers’ needs to implement the district instructional program. Building on NTC’s core principles of using formative assessment to adjust mentoring to individual teachers’ needs, the mentors extended the principles to their work with the system. Specifically, they introduced a program focused on ELL instruction, a critical resource for teachers’ work with district students. And NTC’s literacy professional development gradually moved to embrace and complement the district curriculum, to involve teachers and district coaches in the design process, to acknowledge and draw upon diverse expertise, to develop tools to support teachers’ formative assessment and adaptation to students, and to use district classrooms and students to demonstrate literacy instructional principles. This work was a stretch for an organization traditionally focused on one-on-one teacher mentoring, lacking a track record in literacy professional development, and philosophically opposed to the general instructional approach embedded in the district’s literacy program. By anchoring this evolving RCSD work in its core principles of teaching and learning, NTC was able to effectively adapt its professional development to the learning needs of district teachers and leaders.
Both support organizations and districts who partner with them to improve teaching and learning must address the question of how to customize a program or model to the system in ways that maintain quality standards and meet local needs. Ironically, maintaining fidelity to surface featuresof an organization’s model may sacrifice core principles on which the model was developed. Customizing work for a district then entails surfacing the core principles for quality and using them to design and continually adapt the support work.
Research to flesh out the meaning and processes of high-integrity adaptation of an
instructional improvement model in struggling districts would contribute to knowledge about how support organizations customize their philosophies and practices to achieve success in diverse districts. Similarly, studying how districts make strategic choices about which external organizations to partner with and how to stage their work in the district would yield insights for developing productive partnerships. This problem for practice and research stands in stark contrast to current calls for research to identify highly-specified programs that are expected to be effective when well implemented regardless of context.
53
How can foundations leverage the success of capacity building initiatives? An important lesson from this Hewlett-funded initiative derives from the tension we observed between NTC and district leaders’ views and exercise of authority and influence over the initiative during the three years documented by our evaluation. Lacking a shared understanding and vision for “district capacity building” at the onset, district and NTC leaders sometimes held different views of their respective roles in the initiative. In significant ways, the “district reform” vision and agenda of the initiative was set by the Foundation – through proposal requirements, funding to NTC, and an external evaluation. Neither NTC project leaders and mentors nor district and school administrators were sufficiently introduced to the Foundation’s goals for the initiative to allow them to comprehend and act strategically to advance them.15 Nor did they share a common view of how the “partnership” would work. A dilemma for foundations that support district reform initiatives centers on how heavy-handed, versus hands-off, the funder should be in establishing a shared vision of desired outcomes and promoting an effective working partnership between a district and external support provider. Key is the question of what a funder should constrain in order to leverage a productive partnering relationship and avoid predictable tensions. What could have made a difference for this initiative’s partnership? Based on this initiative’s experience and our prior evaluations of several other district reform partnerships, we frame several specific issues for further exploration and research. First, might funding arrangements that involve both parties create buy-in and incentives to a common vision for change? Second, how might initial convenings of prospective partners with the foundation forge shared understandings and commitments to action among leaders? What documents and tools could carry these commitments? Third, what guidelines for partnering practice might a foundation develop and enforce to promote intended collaboration, e.g., use of memoranda of understanding, convenings to review work progress and plan? Fourth, what guidelines for evaluation practice might help to ensure that documentation of an initiative’s progress and intermediate outcomes feed into partners’ decisions to refine their practice, e.g., annual briefings involving all partners, follow-up sessions to address warning indicators? Finally, how can any such foundation guidelines avoid coming across as regulations and the trap of eliciting compliance responses? Especially in an era of mounting pressures on struggling districts to comply with federal and state accountability demands, foundations walk a fine line as they try to create proactive strategies for system reform.
* * * * *
15 In effect, the evaluation carried the original intent for the initiative. The evaluation logic model was derived from NTC’s proposal to the Hewlett Foundation. Notably, in briefings with NTC mentors and with district administrators after the first and second year of the evaluation, we were questioned about the intent for school or district culture change through the initiative. Many actors saw NTC’s work in the district as more narrow and conventional and were therefore surprised by the scope of the evaluation.
54
Our evaluation of the NTC’s work in RCSD identified a broad range of positive effects on professional development, teachers’ skills and knowledge, and teacher leadership. At the same time, the NTC-RCSD partnership surfaced challenges and puzzles endemic to a system capacity building initiative. It illuminated where and how NTC efforts found traction, and it generated the issues sketched here. A program of research to capture how successful districts choose and usepartnerships and their partnering practices would contribute important new knowledge about the potential for external agents to help develop and sustain system capacity for continuous improvement.
Appendix A
RCSD Student Demographics and Teacher Characteristics
Table A.1. Student Demographics in RCSD, 2006-07
Table A.2. Teacher Characteristics in RCSD, 2005-06
A1
Table A.1. Student Demographics in RCSD, 2006-07
School Grade Span Charter
Total Enrollment
% Asian
% Filipino
% Hispanic
or Latino
% African
American%
White
% Multiple
or No Response % EL
% Free or Reduced-
price Meals
Belle Haven* K-8 587 1.53 0.68 78.02 11.58 0.00 0.17 72.91 86.21 Cesar Chavez* 4-8 412 0.73 0.73 79.13 8.01 0.73 0.24 63.83 97.57 Costano* K-8 371 1.35 0.81 59.57 16.44 0.81 1.08 59.03 94.37 Green Oaks* K-3 471 0.64 0.21 85.14 6.16 0.00 0.21 90.23 93.74 James Flood* K-8 283 1.06 0.00 32.86 56.89 1.41 1.77 26.15 76.06 Willow Oaks* K-8 565 0.88 0.88 84.07 7.43 0.35 0.35 73.10 88.05 Forty-Niners* 6-8 104 0.96 0.00 61.54 28.85 0.00 0.00 50.96 76.47 East Palo Alto Charter K-8 Y 415 0.48 0.00 82.65 13.98 0.00 1.69 51.57 74.82 Edison-Brentwood K-4 Y 523 0.96 0.38 78.20 10.13 0.76 0.57 78.20 76.19 Edison-Ronald McNair 5-8 Y 414 2.42 0.24 72.22 13.04 0.72 1.69 53.14 69.95 Stanford New School K-12 Y 438 1.37 0.00 71.46 19.63 0.68 0.23 55.48 -- District Overall Mean 2793 1.02 0.47 68.62 19.34 0.47 0.55 62.32 87.50 (*7 regular schools) sd (total) 0.32 0.39 18.80 18.32 0.54 0.64 20.24 8.58 Bay Area Schools Mean 628.94 17.04 5.17 32.22 11.14 30.00 5.53 26.19 41.20 (1280 schools) sd 485.43 18.36 7.27 25.28 15.97 25.35 5.26 20.95 29.80 All CA Schools Mean 735.61 8.42 2.85 45.54 7.93 33.09 3.77 27.28 53.11 (8246 schools) sd 611.56 12.77 4.59 29.71 11.90 27.25 4.80 21.30 29.71
* All data are for the school year 2006-07, except that the % Meals data are for 2005-06.
A2
Table A.2. Teacher Characteristics in RCSD, 2005-06
School
School Grade Span
# teachers at school
Average years of teaching
Average years of
teaching in the district
% 1st-year teachers
% New teachers (1-2
years of teaching)
% teachers with full
credentials
% teachers with
emergency credentials
Belle Haven* K-8 28 14.71 11.93 17.86 17.86 71.43 14.29 Cesar Chavez* 4-8 27 7.37 3.78 18.52 25.93 66.67 7.41 Costano* K-8 22 13.32 10.77 22.73 31.82 81.82 4.55 Green Oaks* K-3 26 7.50 6.15 15.38 34.62 76.92 3.85 James Flood* K-8 13 10.92 6.85 0.00 0.00 92.31 0.00 Willow Oaks* K-8 33 8.52 6.45 12.12 36.36 75.76 0.00 Forty-Niners* 6-8 6 7.17 4.33 16.67 50.00 66.67 0.00 East Palo Alto Charter K-8 20 3.40 2.85 15.00 30.00 85.00 5.00 Edison-Brentwood K-4 22 3.36 2.36 31.82 50.00 72.73 4.55 Edison-Ronald McNair 5-8 16 4.06 3.50 31.25 43.75 56.25 6.25 Stanford New School K-12 District Overall Mean 22.14 9.93 7.18 14.75 28.08 75.94 4.30 (*7 regular schools) sd 9.44 3.09 3.08 7.26 15.83 9.09 5.25 Bay Area Schools Mean 32.03 12.03 9.40 8.20 14.89 94.32 3.91 (1280 schools) sd 21.47 3.68 3.39 9.44 12.83 9.91 9.01 All CA Schools Mean 35.65 12.60 10.30 6.80 12.52 95.04 2.78 (8246 schools) sd 25.64 3.86 3.63 8.84 12.24 8.80 6.65
Appendix B
Teacher Survey Scale Definitions
B1
Ravenswood Teacher Survey Scale Definitions
These survey scales were developed with data from Ravenswood Teacher Survey administered in 7 schools in Ravenswood City School District from 2006 to 2009 (Ns = 144 in 2006, 143 in 2007, 102 in 2008, and 145 in 2009). Principal components analysis was used to identify survey items that loaded on a common factor; alpha coefficients indicate the internal consistency of the scale. I. SCHOOL CONDITIONS
Cooperative Staff (SASS16, 4-point Likert scale, 2 items. Alpha = .64, .78, .75 and .81 in 2006, 2007, 2008, and 2009 respectively)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009Most of my colleagues share my beliefs and values about what the central mission of the school should be
1i 1i 1i 1g
There is a great deal of cooperative effort among the staff members
1j 1j 1j 1h
School Administrative Support (SASS, 4-point Likert scale, 2 items. Alpha = .77, .75, .77 & .65)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009The school administration’s behavior toward the staff is supportive and encouraging
1a 1a 1a 1a
In this school, staff members are recognized for a job well done
1k 1k 1k 1i
Satisfaction with School (SASS, 4-point Likert scale, 2 items. Alpha = .72, .77, .81, & .70)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009I am generally satisfied with being a teacher at this school 1p 1p 1p 1n I think about transferring to another school (reverse-scored) 1q 1q 1q 1o
16 The items are from the Teacher Questionnaire of 2003-04 School and Staffing Survey (SASS).
B2
Student Disruption (SASS, 4-point Likert scale, 2 items. Alpha = .42, .48, & .57)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009The level of student misbehavior in this school (such as noise, horseplay or fighting in the halls, cafeteria or student lounge) interferes with my teaching
1b 1b 1b --
The amount of student tardiness and class cutting in this school interferes with my teaching
1o 1o 1o --
Parent Support (SASS, 4-point Likert scale, 1 item.)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009I receive a great deal of support from parents for the work I do
1c 1c 1c --
Staff Material Resources (SASS, 4-point Likert scale, 1 item.)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009Necessary materials such as textbooks, supplies, and copy machines are available as needed by the staff
1d 1d 1d 1b
Rule Enforcement (SASS, 4-point Likert scale, 1 item.)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009Rules for student behavior are consistently enforced by teachers in this school, even for students who are not in their classes
1g 1g 1g 1e
B3
Teacher Expectation (SASS, 4-point Likert scale, 1 item.)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009Most students in this school will not be able to meet grade level standards
1l 1l 1l 1k
Bureaucratic Constraints (SASS, 4-point Likert scale, 1 item.)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009Routine duties and paperwork interfere with my job of teaching
1f 1f 1f 1d
Coordination Effort (SASS, 4-point Likert scale, 1 item.)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009I make a conscious effort to coordinate the content of my courses with that of other teachers
1n 1n 1n 1m
Teacher Stress (SRI, 4-point Likert scale, 1 item.)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009It is stressful to be a teacher at this school 1h 1h 1h 1f
B4
Student Learning Resources (4-point Likert scale, 3 items. Alpha = .70, .47, .23, & .33)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009This school has a resource room or library to support
instruction 1e 1e 1e 1c
This school has a high-quality after-school program 1v 1v 1v 1t Quality substitute teachers are available when needed 1y 1y 1y 1x
Strong Professional Community (5-point Likert scale, 4 items. Alpha = .80, .81, .81, & .85)
To what extent do you agree or disagree with each of the following statements about your school?
2006 2007 2008 2009This school has well-defined plans for instructional
improvement 1t 1t 1t 1r
Teachers share a vision of good teaching 1u 1u 1u 1s
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009Teachers trust one another 2a 2a 2a 2a The faculty works together to achieve excellence 2k 2k 2k 2k
Shared Leadership (5-point Likert scale, 3 items. Alpha = .82, .84, .77, & .84)
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009Teachers take an active role in school-wide decision
making 2g 2g 2g 2g
The faculty has an effective process for making group decisions and solving problems
2h 2h 2h 2h
Teachers trust the school administration 2i 2i 2i 2i
B5
Instructional Collaboration (5-point Likert scale, 3 items. Alpha = .85, .83, .85, & .86)
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009Teachers feel responsible to help one another do their best 2b 2b 2b 2b Teachers use time together to discuss teaching and learning 2c 2c 2c 2c Teachers discuss particular lessons that were not very successful
2d 2d 2d 2d
Colleague Support (5-point Likert scale, 2 items. Alpha = .83, .76, .80, & .87)
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009I feel supported by colleagues to try out new ideas 2l 2l 2l 2l When addressing particular instructional challenges, I feel comfortable asking for advice or help from fellow teachers
2m 2m 2m 2m
Collective Efficacy (5-point Likert scale, 1 item.)
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009Teachers believe they can meet the learning needs of their students
2e 2e 2e 2e
Teacher Advocacy (5-point Likert scale, 1 item.)
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009Teachers advocate for improved conditions in school and district
2f 2f 2f 2f
B6
Teacher Learning Orientation (5-point Likert scale, 1 item.)
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009I see myself as a learner 2j 2j 2j 2j
Teacher Self Perception as Leader (5-point Likert scale, 1 item.)
How well does each of the following statements describe professional relationships in your school?
In this school… 2006 2007 2008 2009I see myself as a leader 2o 2o 2o 2o
Teacher Knowledge Development (5-point frequency scale, 8 items. Alpha = .97, .87, .90, & .93)
How frequently do you do each of the following with other teachers in your school?
With other teachers in this school, I … 2006 2007 2008 2009Share ideas on teaching 3a 3a 3a 3a Discuss what you/they learned at a workshop or conference 3b 3b 3b 3b Share and discuss research on effective teaching methods 3c 3c 3c 3c Share and discuss research on effective instructional practices for English language learners
3d 3d 3d 3d
Explore new teaching approaches for under-performing students
3e 3e 3e 3e
Analyze samples of work done by our students 3f 3f 3f 3f Develop teaching materials or activities for particular classes
3g 3g 3g 3g
Discuss student assessment data to make decisions about instruction
3j 3j 3j 3j
B7
Principal Leadership (5-point frequency scale, 11 items. Alpha = .91, .91, .96, & .97)
Please indicate the extent to which your school principal does each of the following.
My principal … 2006 2007 2008 2009Demonstrates high expectations for all students 4a 4a 4a 4a Uses data to inform decision making 4b 4b 4b 4b Works with individual teachers effectively to improve instruction
4c 4c 4c 4c
Cultivates a shared vision and common purpose among staff
4d 4d 4d 4d
Encourages teachers to be learners 4e 4e 4e 4e Regularly observes my classroom -- -- -- 4f Creates opportunities for teachers’ learning 4f 4f 4f 4g Promotes improvement of student outcomes 4g 4g 4g 4h Supports the development of adult learning communities 4h 4h 4h 4i Provides useful feedback on my teaching -- -- -- 4j Works effectively to develop parent involvement in the school
4i 4i 4i 4k
Encourages teachers to be leaders 4j 4j 4j 4l Works effectively to develop community involvement in the school
4k 4k 4k 4m
Work with Colleagues in Grade-level Learning Teams (5-point frequency scale, 5 items. Alpha = --, .93, .89, & .93) Now think about your work this year with colleagues in grade-level learning teams. To what extent does each of the following describe how your team works together?
This district … 2006 2007 2008 2009We understand the goals for grade-level learning teams -- 5a 5a 5a We share a commitment to working together -- 5b 5b 5b Team members attend scheduled meetings regularly -- 5c 5c 5c We have developed effective routines for doing this work -- 5d 5d 5d Our interactions are mutually respectful -- 5e 5e 5e
B8
Learning Team Facilitator Effectiveness (5-point frequency scale, 1 item.) Now think about your work this year with colleagues in grade-level learning teams. To what extent does each of the following describe how your team works together?
This district … 2006 2007 2008 2009Our facilitator does a good job of guiding our work and discussions
-- 5f 5f 5f
Work with Colleagues in Grade-level Learning Teams to Improve Instruction (5-point frequency scale, 5 items. Alpha = --, .96, .94, & .96)
Now think about your work this year with colleagues in grade-level learning teams. To what extent does each of the following describe how your team works together?
This district … 2006 2007 2008 2009We have identified areas for improving our instruction -- 5g 5g 5g We have used student assessment data to identify areas for improvement
-- -- -- 5h
We have learned from one another about effective teaching strategies
-- 5h 5h 5i
We have developed good ideas to improve instruction -- 5i 5i 5j We have tried out new ideas for instruction -- 5j 5j 5k We have discussed what happened when we tried out new ideas for instruction
-- 5k 5k 5l
B9
II. DISTRICT CONTEXT
District Professionalism (5-point Likert scale, 9 items. Alpha = .94, .94, .93, & .94)
This question concerns the professional climate of your district. Please indicate how strongly you agree or disagree with each of the below statements.
This district … 2006 2007 2008 2009Inspires the very best in the job performance of its teachers 5a 8a 8a 12a Supports local innovation 5b 8b 8b 12b Holds high expectations for our school 5c 8c 8c 12c Supports my school’s whole school change effort 5d 8d 8d 12d Promotes the professional development of teachers 5e 8e 8e 12e Ensures that student learning is the “bottom line” in this school
5f 8f 8f 12f
Helps my school focus on teaching and learning 5g 8g 8g 12g Is committed to high standards for every student 5h 8h 8h 12h Is improving its support for teaching and learning 5i 8i 8i 12i
District Instructional Support (5-point Likert scale, 8 items. Alpha = .92, .91, .83, & .91)
Please indicate the extent to which your district does each of the following.
My district … 2006 2007 2008 2009Provides useful student assessment data 6a 9a 9a 13a Supports schools in using student assessment data 6b 9b 9b 13b Provides professional development tied to our improvement efforts
6c 9c 9c 13c
Provides teachers with support for curriculum and instruction in literacy
6d 9d 9d 13d
Provides teachers with support for curriculum and instruction in mathematics
6e 9e 9e 13e
Provides teachers with up-to-date teaching materials 6f 9f 9f 13f Provides timely and accurate information 6h 9h 9h 13h Provides teachers with easy access to district support staff (e.g., phone number directory)
6i 9i 9i 13i
District Facility Maintenance (5-point Likert scale, 1 item.)
Please indicate the extent to which your district does each of the following.
My district … 2006 2007 2008 2009Maintains school facilities 6g 9g 9g 13g
B10
III. PROFESSIONAL DEVELOPMENT AND SUPPORT
Teacher Preparedness: Teaching and Instruction (5-point Likert scale, 7 items. Alpha = .91, .93, .85, & .88)
When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Handle a range of classroom management or discipline situations
8a 21a 22a 22a
Use a variety of instructional methods 8b 21b 22b 22b Teach your subject matter 8c 21c 22c 22c Plan lessons effectively 8e 21e 22e 22e Design units or lessons of study 8f 21f 22f 22f Select and adapt curriculum and instructional materials 8m 21m 22m 22m Engage all students in learning 8p 21p 22p 22p
Teacher Preparedness: Assessment Use (5-point Likert scale, 6 items. Alpha = .90, .90,
.87, & .86)
When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Use formative assessment to inform reading instruction 8h 21h 22h 22h Use formative assessment to inform writing instruction 8i 21i 22i 22i Use formative assessment to inform mathematics instruction
8j 21j 22j 22j
Use formative assessment to inform science instruction 8k 21k 22k 22k Create specific goals for individual students and modify instruction accordingly
8l 21l 22l 22l
Teacher Preparedness: Use of Computer (5-point Likert scale, 1 item.)
When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Use computers in classroom instruction 8d 21d 22d 22d
B11
Teacher Preparedness: Teach ELL (5-point Likert scale, 1 item.)
When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Teach English Language Learners 8n 21n 22n 22n
Teacher Preparedness: Teach Special Education Students (5-point Likert scale, 1 item.) When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Teach Special Education students 8o 21o 22o 22o
Teacher Preparedness: Assess Student Emotional & Social Needs (5-point Likert scale, 1 item.)
When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Assess student emotional and social needs 8q 21q 22q 22q
Teacher Preparedness: Work with Parents (5-point Likert scale, 1 item.)
When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Work with parents to improve student performance 8r 21r 22r 22r
Teacher Preparedness: Take a Leadership Role (5-point Likert scale, 1 item.)
When you think about this year, how well prepared did you feel to…?
2006 2007 2008 2009Take a leadership role in my school 8s 21s 22s 22s
B12
Usefulness of NTC Mentor: Teaching and Instruction (5-point Likert scale, 7 items.
Alpha = .69, .96, .93, & .96)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Handle a range of classroom management or discipline situations
11a 13a 14a 11a
Use a variety of instructional methods 11b 13b 14b 11b Teach your subject matter 11c 13c 14c 11c Plan lessons effectively 11e 13e 14e 11d Design units or lessons of study 11f 13f 14f 11e Select and adapt curriculum and instructional materials 11m 13m 14m 11k Engage all students in learning 11p 13p 14p 11n
Usefulness of NTC Mentor: Assessment Use (5-point Likert scale, 6 items. Alpha = .63, .91, .90, & .94)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Use formative assessment to inform reading instruction 11h 13h 14h 11g Use formative assessment to inform writing instruction 11i 13i 14i 11h Use formative assessment to inform mathematics instruction
11j 13j 14j 11i
Use formative assessment to inform science instruction 11k 13k 14k -- Create specific goals for individual students and modify instruction accordingly
11l 13l 14l 11j
Usefulness of NTC Mentor: Use of Computer (5-point Likert scale, 1 item.)
Since you began working with an NTC mentor, how helpful has this been in developing these skills?
2006 2007 2008 2009Use computers in classroom instruction 11d 13d 14d --
B13
Usefulness of NTC Mentor: Teach ELL (5-point Likert scale, 1 item.)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Teach English Language Learners 11n 13n 14d 11l
Usefulness of NTC Mentor: Teach Special Education Students (5-point Likert scale, 1 item.)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Teach Special Education students 11o 13o 14o 11m
Usefulness of NTC Mentor: Assess Student Emotional & Social Needs (5-point Likert scale, 1 item.)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Assess student emotional and social needs 11q 13q 14q 11o
Usefulness of NTC Mentor: Work with Parents (5-point Likert scale, 1 item.)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Work with parents to improve student performance 11r 13r 14r 11p
B14
Usefulness of NTC Mentor: Take a Leadership Role (5-point Likert scale, 1 item.)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Take a leadership role in my school 11s 13t 14t 11r
Usefulness of NTC Mentor: Work Effectively in Learning Team (5-point Likert scale, 1 item.)
To what extent has your mentoring experience helped you to develop each of the following skills?
2006 2007 2008 2009Work effectively with others in my learning team -- 13s 14s 11q
Appendix C
RCSD Classroom Teacher Retention Patterns by School, 2005-06 to 2009-10
C1
Appendix C. RCSD Classroom Teacher Retention Patterns by School,
2005-06 to 2009-10
School
2005-06 to
2006-07
2006-07 to
2007-08
2007-08 to
2008-09
2008-09 to
2009-10 Belle Haven 64% 72% 75% 72% Chavez 68% 73% 76% 75% Costano 71% 56% 80% 82% Green Oaks 65% 83% 77% 70% Flood 60% 93% 73% 79% Willow Oaks 74% 70% 80% 88% McNair N/A N/A N/A 45% Districtwide 68% 73% 77% 75% Total number of teachers who left the classroom*
44 35 30 36
Total number of classroom teachers
136 131 130 142
* Includes classroom teachers who taught in the school during the year; teachers who transferred to other schools in the district were not counted for teacher attrition. Not included in the table are Special Education teachers and other non-classroom teaching staff. Attrition for this group was 38% of 40 in year 1, 39% of 36 in year 2, 46% of 41 in year 3, and 41% of 46 in year 4 of the table.
C1
Appendix D
RCSD Student Performance: Trends in California Standards Tests (CST)
Table D.1. Grade-level Trends on English and Math CST Scores by School and Years Working with NTC: Average Percent Students Scoring Basic and Above from the Pre-NTC years to 2009
Table D.2. Student Performance: RCSD Schools and CA Statewide Norms Table D.3. California Standards Test: Statewide Results, 2002 to 2009 Table D.4. California Standards Test: RCSD School and Statewide Results in
English, % Proficient or Above Table D.5. California Standards Test: RCSD School and Statewide Results in Math,
% Proficient or Above Table D.6. California Standards Test: RCSD School and Statewide Results in
English, % Basic or Above Table D.7. California Standards Test: RCSD School and Statewide Results in Math,
% Basic or Above
D1
Table D.1. Grade-level Trends on English and Math CST Scores by School and
Years Working with NTC: Average Percent Students Scoring Basic and Above from the Pre-NTC years to 2009
Grade Level
School Subject 2 3 4 5 6 7 8 School Group A: 2002 & 2003 to 2009
English 41 37 27 53* 30 52* 30 56* Cesar Chavez Math 42 32 31 61* 31 70*
English 32 52* 29 43* 41 56* Green Oaks Math 52 71* 37 76* 42 43
School Group B: 2004 & 2005 to 2009 English 34 78* 31 72* 46 66* 41 54* 47 66* 56 75* 60 78* Willow
Oaks Math 50 92* 57 94* 42 70* 36 26 33 41 63 77* School Group C: 2005 & 2006 to 2009
English 46 74* 37 38 52 64* 51 53 43 61* 64 73* 54 76* Belle Haven Math 59 87* 51 58* 48 51 40 28 35 50* 61 57
English 84 80 57 57 59 46 63 59 57 69* 58 90* 51 70* Costano
Math 91 95* 61 62 51 30 45 31 49 42 44 89* English 73 82* 67 68 74 82* 69 61 73 79 74 65 85 68 James
Flood Math 78 79 80 71 52 72* 50 29 53 69* 67 51 State 2009
English 79 72 85 83 82 81 78
Math 83 83 86 78 75 74Note: Percentages are rounded to the nearest whole number. Grades 4 and 5 moved from Cesar Chavez to Green Oaks, hence pre and post percents are shown in different cells. * Grade-level gain was equal to or larger than statewide gain.
D2
Table D.2. Student Performance: RCSD Schools and CA Statewide Norms We used the pre-and-post-NTC approach in determining the improvement of student performance in RCSD Schools. Change in student performance: CST data from 2002 to 2009 = difference between 2009 score and “pre-NTC” scores Cesar Chavez and Green Oaks: 2009 - Average of (02 and 03) Willow Oaks: 2009 - Average of (04 and 05) Belle Haven, Costano, and Flood: 2009 - Average of (05 and 06) We also look at two sets of scores in English and Math separately: % Basic or Above – a more appropriate index for students in RCSD schools. And % Proficient or Above – the state’s goal, the results are shown in Appendix D. % Basic or Above = % Basic + % Proficient + % Advanced % Proficient or Above = % Proficient + % Advanced
D3
Table D.3. California Standards Test: Statewide Results, 2002 to 2009
English: % Proficient or Above English: % Basic or Above Grade 2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2005 2006 2007 2008 2009
2 32 36 35 42 47 48 48 53 63 68 65 70 74 74 77 79 3 34 33 30 31 36 37 38 44 62 63 61 62 68 69 72 72 4 36 39 39 47 49 51 55 61 71 74 73 77 77 80 84 85 5 31 36 40 43 43 44 48 54 71 72 71 75 75 76 81 83 6 30 36 36 38 41 42 47 52 66 71 72 72 72 73 78 82 7 33 36 36 43 43 46 49 54 65 69 69 73 72 74 76 81 8 32 30 33 39 41 41 45 48 66 64 69 72 73 72 74 78 Math: % Proficient or Above Math: % Basic or Above
Grade 2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2005 2006 2007 2008 20092 43 53 51 56 59 59 59 63 68 76 76 78 81 81 80 83 3 38 46 48 54 58 58 61 64 65 71 73 76 79 79 82 83 4 37 45 45 50 54 56 61 66 67 72 73 75 78 79 84 86 5 29 35 38 44 48 49 51 57 59 61 65 67 69 71 75 78 6 32 34 35 40 41 42 44 49 62 64 66 67 68 71 73 75 7 30 30 33 37 41 39 41 43 61 62 63 64 67 68 70 74 8
D4
Table D.4. California Standards Test: RCSD School and Statewide Results in English, % Proficient or Above
CST English: % Proficient or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- Change School Group A: 2002 & 2003 to 2009 Chavez 4 11 8 14 20 12 9.5 37.5 61.0 23.5 5 5 7 7 15 11 8 11 6.0 33.5 54.0 20.5 6 0 6 9 8 16 8 9 19 3.0 19.0 16.0 33.0 52.0 19.0 7 4 4 7 20 20 24 8 19 4.0 19.0 15.0 34.5 54.0 19.5 8 5 3 4 10 16 17 22 14 4.0 14.0 10.0 31.0 48.0 17.0
2 11 4 3 5 12 9 13 17 7.5 17.0 9.5 34.0 53.0 19.0 Green Oaks 3 10 5 4 3 10 8 9 13 7.5 13.0 5.5 33.5 44.0 10.5 4 17 17 16 16.0 37.5 61.0 23.5 5 20 20.0 33.5 54.0 20.5 School Group B: 2004 & 2005 to 2009
2 11 16 5 13 40 31 44 43 9.0 43.0 34.0 38.5 53.0 14.5 Willow Oaks 3 11 23 8 9 8 5 20 25 8.5 25.0 16.5 30.5 44.0 13.5 4 13 18 24 13 21 12 13 27 18.5 27.0 8.5 43.0 61.0 18.0 5 4 6 10 13 11 21 16 13 11.5 13.0 1.5 41.5 54.0 12.5 6 0 3 7 11 13 18 13 23 9.0 23.0 14.0 37.0 52.0 15.0 7 8 2 6 28 15 16 21 40 17.0 40.0 23.0 39.5 54.0 14.5 8 23 3 10 20 14 19 27 34 15.0 34.0 19.0 36.0 48.0 12.0
D5
Table D.4. Continued
CST English: % Proficient or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- Change School Group C: 2005 & 2006 to 2009
2 19 42 12 16 16 21 15 29 16.0 29.0 13.0 44.5 53.0 8.5 Belle Haven 3 14 15 12 11 7 10 19 21 9.0 21.0 12.0 33.5 44.0 10.5 4 19 14 15 19 18 16 22 30 18.5 30.0 11.5 48.0 61.0 13.0 5 27 26 12 17 18 10 9 16 17.5 16.0 -1.5 43.0 54.0 11.0 6 12 9 17 9 15 18 13 20 12.0 20.0 8.0 39.5 52.0 12.5 7 16 13 11 36 16 29 26 23 26.0 23.0 -3.0 43.0 54.0 11.0 8 4 7 18 9 25 17 19 27 17.0 27.0 10.0 40.0 48.0 8.0 Costano 2 39 56 41 69 54 36 32 56 61.5 56.0 -5.5 3 34 35 12 18 11 16 11 26 14.5 26.0 11.5 4 20 20 17 19 27 11 24 18 23.0 18.0 -5.0 5 30 31 37 29 27 26 11 15 28.0 15.0 -13.0 6 22 13 4 19 26 31 21 22.5 21.0 -1.5 7 9 9 2 23 21 29 47 22.0 47.0 25.0 8 2 10 14 11 11 25 27 14 11.0 14.0 3.0 Flood 2 27 28 22 48 36 54 57 53 42.0 53.0 11.0 3 30 28 28 30 33 16 18 38 31.5 38.0 6.5 4 36 33 48 39 27 34 25 39 33.0 39.0 6.0 5 38 35 35 48 20 4 29 25 34.0 25.0 -9.0 6 19 26 24 24 37 26 13 41 30.5 41.0 10.5 7 20 28 41 43 29 48 31 24 36.0 24.0 -12.0 8 35 32 32 29 37 26 33 29 33.0 29.0 -4.0
D6
Table D.5. California Standards Test: RCSD School and Statewide Results in Math, % Proficient or Above
CST Math: % Proficient or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- Change School Group A: 2002 & 2003 to 2009 Chavez 4 12 23 22 40 23 17.5 41.0 66.0 25.0 5 11 8 2 25 17 3 26 9.5 32.0 57.0 25.0 6 3 7 15 21 25 7 12 25 5.0 25.0 20.0 33.0 49.0 16.0 7 9 5 11 20 29 46 51 36 7.0 36.0 29.0 30.0 43.0 13.0 8
2 15 20 16 22 29 26 32 41 17.5 41.0 23.5 48.0 63.0 15.0 Green Oaks 3 12 13 14 14 27 15 30 41 12.5 41.0 28.5 42.0 64.0 22.0 4 18 21 18 18.0 41.0 66.0 25.0 5 19 19.0 32.0 57.0 25.0 School Group B: 2004 & 2005 to 2009
2 45 28 12 17 42 41 53 69 14.5 69.0 54.5 53.5 63.0 9.5 Willow Oaks 3 47 63 29 19 32 46 33 72 24.0 72.0 48.0 51.0 64.0 13.0 4 56 24 27 13 33 11 47 50 20.0 50.0 30.0 47.5 66.0 18.5 5 38 10 8 11 9 25 9 4 9.5 4.0 -5.5 41.0 57.0 16.0 6 30 3 7 15 13 23 18 5 11.0 5.0 -6.0 37.5 49.0 11.5 7 46 4 25 30 16 21 43 48 27.5 48.0 20.5 35.0 43.0 8.0 8
D7
Table D.5. Continued CST Math: % Proficient or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- Change
School Group C: 2005 & 2006 to 2009 2 20 40 19 30 32 18 36 60 31.0 60.0 29.0 57.5 63.0 5.5 Belle
Haven 3 12 21 23 30 22 19 36 36 26.0 36.0 10.0 56.0 64.0 8.0 4 17 17 14 16 16 13 20 27 16.0 27.0 11.0 52.0 66.0 14.0 5 27 10 8 16 18 8 13 7 17.0 7.0 -10.0 46.0 57.0 11.0 6 10 6 16 11 14 18 17 18 12.5 18.0 5.5 40.5 49.0 8.5 7 5 15 6 30 25 43 19 28 27.5 28.0 0.5 39.0 43.0 4.0 8 Costano 2 40 60 63 84 65 47 67 84 74.5 84.0 9.5 3 13 62 30 28 24 24 18 34 26.0 34.0 8.0 4 21 14 23 19 18 17 19 10 18.5 10.0 -8.5 5 25 34 25 26 27 17 7 7 26.5 7.0 -19.5 6 13 18 4 8 24 17 27 16.0 27.0 11.0 7 13 13 2 4 13 27 59 8.5 59.0 50.5 8 Flood 2 23 44 61 66 36 69 81 57 51.0 57.0 6.0 3 29 31 43 57 46 32 32 58 51.5 58.0 6.5 4 18 33 41 40 12 27 21 36 26.0 36.0 10.0 5 50 41 43 38 8 0 14 11 23.0 11.0 -12.0 6 14 7 24 36 0 12 8 38 18.0 38.0 20.0 7 38 20 17 32 32 32 20 32.0 20.0 -12.0 8
D8
Table D.6. California Standards Test: RCSD School and Statewide Results in English, % Basic or Above
CST English: % Basic or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- ChangeSchool Group A: 2002 & 2003 to 2009 Chavez 4 34 47 36 55 51 40.5 72.5 85.0 12.5 5 33 40 40 44 52 37 48 36.5 71.5 83.0 11.5 6 16 38 49 51 44 41 45 53 27.0 53.0 26.0 68.5 82.0 13.5 7 34 26 35 55 56 58 41 52 30.0 52.0 22.0 67.0 81.0 14.0 8 25 35 28 49 54 52 64 56 30.0 56.0 26.0 65.0 78.0 13.0
2 31 33 12 26 31 27 53 52 32.0 52.0 20.0 65.5 79.0 13.5 Green Oaks 3 29 29 19 16 43 24 32 43 29.0 43.0 14.0 62.5 72.0 9.5 4 51 61 41 41.0 72.5 85.0 12.5 5 56 56.0 71.5 83.0 11.5 School Group B: 2004 & 2005 to 2009
2 39 54 19 48 71 62 72 78 33.5 78.0 44.5 67.5 79.0 11.5 Willow Oaks 3 42 64 28 33 38 41 43 72 30.5 72.0 41.5 61.5 72.0 10.5 4 56 52 48 43 65 56 68 66 45.5 66.0 20.5 75.0 85.0 10.0 5 46 46 34 48 45 52 54 54 41.0 54.0 13.0 73.0 83.0 10.0 6 37 37 47 46 41 40 54 66 46.5 66.0 19.5 72.0 82.0 10.0 7 43 42 47 65 50 47 64 75 56.0 75.0 19.0 71.0 81.0 10.0 8 65 48 52 67 65 50 67 78 59.5 78.0 18.5 70.5 78.0 7.5
D9
Table D.6. Continued CST English: % Basic or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- ChangeSchool Group C: 2005 & 2006 to 2009
2 51 75 49 46 46 57 45 74 46.0 74.0 28.0 72.0 79.0 7.0 Belle Haven 3 45 44 49 32 42 41 44 38 37.0 38.0 1.0 65.0 72.0 7.0 4 51 46 43 45 59 52 61 64 52.0 64.0 12.0 77.0 85.0 8.0 5 59 55 45 51 51 41 49 53 51.0 53.0 2.0 75.0 83.0 8.0 6 45 40 54 36 50 54 55 61 43.0 61.0 18.0 72.0 82.0 10.0 7 42 48 57 65 63 71 61 73 64.0 73.0 9.0 72.5 81.0 8.5 8 45 54 55 51 56 51 67 76 53.5 76.0 22.5 72.5 78.0 5.5 Costano 2 72 86 66 90 77 70 81 80 83.5 80.0 -3.5 3 65 77 45 53 60 37 29 57 56.5 57.0 0.5 4 49 66 53 59 59 49 48 46 59.0 46.0 -13.0 5 69 58 54 61 65 70 37 59 63.0 59.0 -4.0 6 56 53 56 59 54 76 69 56.5 69.0 12.5 7 44 36 44 56 59 53 90 57.5 90.0 32.5 8 44 55 52 45 57 65 56 70 51.0 70.0 19.0 Flood 2 57 59 78 79 67 78 87 82 73.0 82.0 9.0 3 66 64 71 66 68 55 52 68 67.0 68.0 1.0 4 75 85 86 82 65 75 54 82 73.5 82.0 8.5 5 92 83 89 82 55 50 79 61 68.5 61.0 -7.5 6 74 82 76 76 69 59 67 79 72.5 79.0 6.5 7 79 72 79 93 54 70 77 65 73.5 65.0 -8.5 8 97 84 76 75 95 70 69 68 85.0 68.0 -17.0
D10
Table D.7. California Standards Test: RCSD School and Statewide Results in Math, % Basic or Above
CST Math: % Basic or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- ChangeSchool Group A: 2002 & 2003 to 2009 Chavez 4 37 47 49 68 61 42.0 69.5 86.0 16.5 5 34 29 22 57 40 18 53 31.5 60.0 78.0 18.0 6 23 38 57 55 62 36 41 61 30.5 61.0 30.5 63.0 75.0 12.0 7 34 27 37 52 65 73 77 70 30.5 70.0 39.5 61.5 74.0 12.5 8
2 53 50 39 51 52 49 59 71 51.5 71.0 19.5 72.0 83.0 11.0 Green Oaks 3 30 44 37 37 50 38 54 76 37.0 76.0 39.0 68.0 83.0 15.0 4 43 57 42 42.0 69.5 86.0 16.5 5 43 43.0 60.0 78.0 18.0 School Group B: 2004 & 2005 to 2009
2 83 58 48 52 82 72 75 92 50.0 92.0 42.0 77.0 83.0 6.0 Willow Oaks 3 90 108 62 51 69 75 64 94 56.5 94.0 37.5 74.5 83.0 8.5 4 91 61 47 37 73 46 77 70 42.0 70.0 28.0 74.0 86.0 12.0 5 82 29 40 31 28 44 36 26 35.5 26.0 -9.5 66.0 78.0 12.0 6 86 29 31 35 44 49 48 41 33.0 41.0 8.0 66.5 75.0 8.5 7 84 36 56 69 56 50 72 77 62.5 77.0 14.5 63.5 74.0 10.5 8
D11
Table D.7. Continued
CST Math: % Basic or Above School CA Statewide School Grade 2002 2003 2004 2005 2006 2007 2008 2009 Pre- Post- Change Pre- Post- Change School Group C: 2005 & 2006 to 2009
2 46 65 57 64 54 54 66 87 59.0 87.0 28.0 79.5 83.0 3.5 Belle Haven 3 32 47 56 56 45 51 65 58 50.5 58.0 7.5 77.5 83.0 5.5 4 41 40 33 42 53 42 52 51 47.5 51.0 3.5 76.5 86.0 9.5 5 46 35 24 33 47 23 39 28 40.0 28.0 -12.0 68.0 78.0 10.0 6 35 40 52 36 33 47 44 50 34.5 50.0 15.5 67.5 75.0 7.5 7 37 48 41 56 65 88 51 57 60.5 57.0 -3.5 65.5 74.0 8.5 8 Costano 2 65 79 74 97 84 66 93 95 90.5 95.0 4.5 3 44 87 62 52 70 40 43 62 61.0 62.0 1.0 4 57 46 60 47 54 39 32 30 50.5 30.0 -20.5 5 50 52 44 39 51 43 16 31 45.0 31.0 -14.0 6 43 39 33 37 61 58 42 49.0 42.0 -7.0 7 51 39 36 29 58 58 89 43.5 89.0 45.5 8 Flood 2 47 75 80 92 64 87 92 79 78.0 79.0 1.0 3 55 62 73 81 78 61 69 71 79.5 71.0 -8.5 4 68 63 62 65 39 55 64 72 52.0 72.0 20.0 5 85 82 78 69 31 19 39 29 50.0 29.0 -21.0 6 50 51 65 68 37 43 37 69 52.5 69.0 16.5 7 76 64 58 78 56 73 51 67.0 51.0 -16.0 8
