1
The Ball-RUSD PartnershipFinal Evaluation Report
September 23, 2011
Catherine Awsumb Nelson, Ph.D.
2
Key QuestionsI. What did the partnership look like?
a) Designb) Participationc) Quality
II. What impacts did it have?a) Non-participantsb) Personal transfer of participantsc) Broader impacts
III. What does Ball leave behind?IV. What is the potential for sustainability?V. What was learned about investing in district
transformation?
3
2010-11 Data Collection
DATA SOURCE n Details
Individual interviews
22 Executive Cabinet (5), SIL design team (2), CoP2 (2), CoPs (7), IC (6)
SIL case studies 5 schools30 interviews
At each of 5 schools, fall and spring interviews with principal, new team member, and returning team member, plus document analysis
Event observations
9 SIL (3), Literacy Network (3), IC (1), Efficacious Instruction workgroup (1), Sensing team (1)
Event feedback surveys
10 SIL (5), Literacy Network (4), IC (1)
Milestone monitoring meetings
8 Design teams from LN, SIL, and IC, full Executive CabinetMiddle and end of year
Staff survey 364 131 partnership participants (58% response)233 non-participants (35% response)
Student survey 4949 Overall response rate 56% for Grades 4-12
Participant tracker
895 Certificated staff only
4
Ball approach to district transformation Transform, not reform Focus is on systems change Assumption that most necessary expertise is
already in the district Key strategies:
Capacity building Coherence making
5
Why is “coherence” so important? “I hope we are strong enough to
keep it going. I worry about it. Not a reflection on the work Ball has done but the district…we do amazing things in this district, but we tend to dabble. You have to keep things alive yourself if they are working for you because the district moves on. We do a lot of good things but don’t stick with anything long enough to make it great.” (CoP member)
“The ‘here’s another binder’ mentality is what we fight here all the time. So many things they want you to know and learn about. We dip our toes into so many things and they all blur together. I don’t think we need to try so many things at once. With kids, when you bombard them with stuff, it doesn’t work. When you teach for depth, spend the quality time on a unit, that is when learning happens. (SIL member)
Fewer than 10% of respondents agreed with the statement “Once we start a new program in this district we follow up to make sure it
is working.”
6
Critical Features of the Rowland Partnership Ball provided structures and time for inquiry
and collaboration around instruction at three levels Classroom School District
Collaboration supported through design and coaching
Focus on changing adult learning to change student learning
Working with the willing
7
Flipping the adage
In education, money buys time, which creates opportunity for learning. That is the great gift Ball has given Rowland, along with design and coaching that help the professional time align with what we know about what creates impact in the classroom.
The old adage is time is money.
8
U.S. Schools Lag International Competitors in Providing Professional Learning Time The United States is far behind in providing public school
teachers with the kind of high intensity, job-embedded collaborative learning that research shows is most effective in changing practice and improving learning
U.S. teachers report little professional collaboration in designing curriculum and sharing practices, and the collaboration that occurs tends to be weak and not focused on strengthening teaching and learning.
Compared to other nations that outperform the United States on international assessments, American teachers spend much more time teaching students and have significantly less time to plan and learn together.
“Professional Learning in the Learning Profession: A Status Report on Teacher Development in the U.S. and Abroad.”
(National Staff Development Council, 2009)
9
Building Instructional Capacity in RUSDStructure Function Ball supports #
Communities of Practice
Small groups of educators collaborating around a specific literacy topic
•$ for professional books and training•Literacy Network Days•Cluster days•CoP Garden•Developmental framework
120-150
School Instructional Leaders
Teams of 6-8 administrators and teachers tasked with leading professional learning at each site
•Design and facilitation for monthly cross-district meetings•Learning Walks
110-150
Instructional Cabinet
Representatives across district and role groups charged with identifying and supporting district wide instructional priorities
•Meeting design and facilitation•Support for workgroups
25+
Executive Leadership
Existing district leadership structure
•Coaching 5
10
District Context Budget cuts Accountability pressure Demographic shifts Hollowing out of district instructional support
capacity
11
Partnership Participation, 2009-2011
Never; 65.9%2011 only;
8.4%
2010 only; 8.9%
Both years; 16.8%
12
Intermediate and secondary participation increased
Elementary Intermediate High0%
5%
10%
15%
20%
25%
30%
35%
40%
35%
21%
14%
33%
28%
19%
2010 2011
% o
f cla
ssifi
ed
em
plo
ye
es p
art
icip
at-
ing
in
Part
ne
rsh
ip
13
Quality of the Work
14
Ratings for Quality of Professional Learning remarkably stable
% of participants rating partnership professional learning as “Good “ of “Excellent”
2011 2010
Ensuring that all voices are heard 86% 86%
Having a positive impact on student learning 78% 77%
Making it safe to raise difficult issues 78% 74%
Building on existing professional expertise within the district 78% 76%
Focusing on issues directly relevant to my practice 76% 75%
Being grounded in data and/or other evidence of student learning 71% 67%
Striking a good balance between content and process 69% 71%
Using time well 57% 58%
15
Measures of overall intervention quality increased across the board, with the biggest increase in potential for sustainability
Important aspects of the work will continue once Ball personnel and funds are no longer in the district.
Working with Ball helped RUSD respond more strategically to Program Improvement status and budget cuts than we otherwise would have.
Ball Partnership activities help us deeply examine our approach to teaching and learning.
Ball Partnership work is well integrated into the day-to-day work of the district.
District leaders have demonstrated that they are committed to the Partnership work.
Ball Partnership work has helped RUSD move different parts of the system toward a common focus.
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
65%
59%
78%
45%
61%
71%
42%
49%
69%
33%
61%
57%
2010 Partic-ipants
% agree/strongly agree
16
Impacts on non-participants
17
All avenues for non-participant awareness increased this year, with informal communication about CoP work growing the most
I haven't heard anything about the Partnership this year
Work from a teacher Community of Practice group was shared in a staff development meeting
I was told about the work of the Instructional Cabinet
Participated in a Learning Walk @ my site led by my school's School Instructional Leadership team
Read about Ball Partnership activity on the district website or in printed materials
Participated in a meeting led by my school's School Instructional Leadership team
I heard informally about the work a teacher Community of Practice was doing
A colleague talked to me about work he/she was doing with Ball
Heard about Ball Partnership activity in a staff meeting
0% 10% 20% 30% 40% 50% 60%
6%
20%
21%
29%
33%
36%
39%
47%
56%
10%
18%
13%
25%
24%
31%
22%
42%
46%
20102011
% of non-participants who heard about partnership through…
18
40-50% of non-participants noticed improvements in site meetings
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
29% 30% 32% 35% 38% 40% 36%
12% 13%14% 12%
14% 15% 20%
Much more so More so
% o
f non-p
art
icip
ants
sayin
g s
ite m
eetings
over
the last
tw
o y
ears
have c
hanged
19
Participant Impacts
20
Reported levels of personal transfer are unchanged from last year
0%10%20%30%40%50%60%70%80%90%
100%
9% 9% 9% 9% 9% 7%
31% 32% 37% 37% 34% 26%
40% 38% 36% 34% 33%38%
Revolutionized my practice Major transferModerate transfer
% o
f part
icip
ants
report
ing level of
transfe
r
21
Participant ratings of broader district impacts
Co
llab
ora
tio
n in
pro
fe..
.
Qu
alit
y o
f p
rofe
ssio
na
...
Co
Ps
as
a m
od
el o
f co
...
No
rms
of
refl
ect
ive
p..
.
Op
po
rtu
nit
ies
for
tea
...
Co
he
ren
t vis
ion
of
ef.
..
Use
of
eff
ect
ive
/re
sea
...
SIL
te
am
s d
eve
lop
ing
...
Imp
rove
d d
eci
sio
n-m
ak..
.
Dis
tric
t b
ett
er
stru
ctu
...
IC d
eve
lop
ing
dis
tric
t...0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
43% 40% 38% 34% 33% 32% 30% 26%20% 16% 14%
43%41% 44%
41% 46% 48% 49%
41%
34% 47%44%
Mod-erate
% o
f p
art
icip
an
ts r
ati
ng
im
pa
ct
ma
jor/
mo
de
rate
22
For all impact areas with a direct comparison from last year, ratings of Major impact increased
2011-Participants rating impact "Major"
Increase from 2010
Increased collaboration in professional learning 43% +17%
Increased quality of professional learning 40% +12%
Establishing norms of reflective practice and rigorous dialogue about instruction 34% +17%
Use of effective/research-based practices for instruction 30% +13%
Improved decision-making processes in the district 20% +10%
District better structured to support effective instruction 16% +6%
23
District instructional support capacity
District-level decisions are made with adequate input from school-based personnel
It is clear where and how decisions about instruction get made
Key resources of time, money, and personnel are clearly connected to instructional improvement priorities
District decisions are grounded in data
The district has a coherent vision of quality instruction
District priorities are clearly focused on supporting and improving instruction
Staff are held accountable for realizing the district's vision of quality instruction
0% 20% 40% 60% 80% 100%
13%
20%
28%
30%
34%
35%
39%
% of participants who agree/strongly agree
24
Capacity Building Structures
25
CoPs took off in 2nd half of year
January March June0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
71%
88% 89%
18%
76%
96%
43%
82%
94%
36%
67%69%
58%
81%
93%
Used a cycle of plan/act/reflect
Jointly examined artifacts of student learning
Put in place ways to capture our learning
Shared learning with colleagues not in our CoP
Moved beyond sharing to true group collaboration (joint work, development of shared mean-ing)
% o
f m
em
bers
sayin
g t
heir
CoP
had d
one
"Mostl
y"
or
"Com
ple
tely
"
26
Communities of Practice participant ratings of rigor up sharply over last year
My CoP had a clear question or purpose to focus our inquiry
We routinely agreed to try specific things in our classroom and then discuss with the
group how they worked
We routinely looked at evidence of student work from our own classrooms as we talked
about how well a specific practice worked
We pushed each other to be rigorous about what works and why in the literacy practice
we were focusing on
All of our members stayed engaged and accountable to each other
We routinely used Process Learning Circles and/or specific conversational processes
like ordered sharing
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
84%
76%
54%
57%
63%
59%
90%
90%
81%
81%
77%
66%20112010
% of CoP members agree/strongly agree
27
Greatest SIL accomplishments were in team development; whole staff learning also significantly impacted
Used learning walks to reflect on instruction in our school
Built shared understanding among the whole staff about what efficacious instruction looks like
Used cultural proficiency as a lens to analyze instruction in our school
Become seen by all staff as leaders of learning in the school
Built shared understanding among our team about what efficacious instruction looks like
Made staff meeting time more collaborative and inquiry based
Made staff meeting time more focused on learning
Built our own competencies as instructional leaders
Learned to function well as a team
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
27%
12%
13%
19%
16%
29%
36%
23%
48%
29%
44%
49%
48%
57%
48%
42%
63%
43%
CompletelyMostly
% of members rating extent to which their team accomplished goal (May SIL, n=95)
28
SIL impact ratings up across the board this year
% of SIL participants who agree/strongly agree 2011
Change from last
year
Because of the SIL work, people across the district are starting to use more similar language about instruction 65% 26%
The SIL work has significantly influenced our site-level professional development approach and agenda 57% 11%
Our SIL team had enough representation to effect change in our site 49% 17%
Expectations for implementing the SIL work at our site were clear 43% -3%
Our SIL team will be a driving force in our school's instructional improvement efforts going forward 56% 11%
SIL has given teachers more of a leadership role over instruction in this school 51% 8%
We made progress this year in making instruction more public in this school 59% 11%
Instructional Capacity assessments of their own effectiveness vary widely across goals
29
Monitoring the effectiveness of PI plan implementation
Increasing district wide awareness of IC
Supporting staff in implementing instructional priorities
Monitoring the implementation of the PI plan
Understanding IC's role
Setting district wide instructional priorities
Setting a direction for RTI district wide
Determining training needed for new data system
Building our own understanding of RTI (within IC)
Launching a system wide pilot of the new data system
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
16%
23%
31%
35%
55%
63%
67%
74%
91%
95%
% of IC rating group group “Effective” or “Extremely Effective”
30
Challenges to district-level coherence-making: Imbalance between mandate and resources
Mandate: Build
instructional coherence
through developing and
supporting instructional
priorities
Capacity
Authority
Lines of communicat
ion
31
Summary of Impacts: What does Ball leave behind?
32
Areas of most and least change this year
Greatest progress Least progress
Rigor/depth of learning for partnership participants
Quality of professional learning for ALL district staff
Concrete agreements around Efficacious Instruction
District ownership, confidence in sustainability
Cross strand connections
New structure for district coherence continued to struggle
Staff assessment of district instructional capacity
Levels of personal transfer stayed flat
33
Summary of Impacts
Transitions
with
momentum
Idea of design Expectation that decisions will be
collaborative and inclusive Expectation that professional
learning will be collaborative CoPs as a vehicle for teacher
directed inquiry into practice Learning Walks starting to de-
privatize practice in some sites SIL starting to re-shape site-based
learning
34
“No going back” “One of the most striking places I saw the impact
was when we had this presenter from county on EL issues. The way they presented was just so foreign from how we do things- it showed how far we have come. It was just, throw up a power point and then we will take your questions- boom. Instead of taking a piece and really working it the way we do now. (Principal)
“I think many of us have passed the point of no return this year. We don’t want to go back. There is no way we are going back to professional development that is not collaborative and self-initiated.” (Teacher)
35
Summary of ImpactsC
hallenges to the emerging system
Haven or silo? Struggle to balance accountability
with “learning as a journey” Divergent conceptions of
assessment Traditional conceptions of teacher
“leadership” Search for plug-in solution still
evident in some areas
36
What does Ball leave behind? Morale maintained during difficult time Cuts made with more intentionality Capacity for the design and facilitation of
adult learning (in a much broader base of staff)
District owns new structures for learning and leadership
Norms about adult learning Agreements about efficacious instruction
37
Looking Forward: Sustainability Potential
38
Majority of participants are optimistic that most impacts will be lasting
Improved decision-making processes in the district
IC developing district-level coherence in instruction
CoPs as a model of collaborative inquiry
Quality of professional learning
Norms of reflective practice/rigorous dialogue
Collaboration in professional learning
33%
41%
44%
53%
54%
56%
58%
59%
59%
62%
65%
% of participants saying impact will "Definitely" or "Probably" last
39
Large increase in confidence that impacts are sustainable
Change in Sustainability Rating
Communities of Practice as a model of collaborative professional inquiry +36%
Increased collaboration in professional learning +35%
School Instructional Leadership (SIL) teams as a means to develop site-level coherence in instruction +34%
Increased quality of professional learning +31%
Improved decision-making processes in the district +14%
The Instructional Cabinet as a means to develop district-level coherence in instruction -1%
40
Moving Forward in RUSD:Efficacious Instruction
41
RUSD Learning Paradigm
Linked paradigm shifts in adult and student learning
Both grounded in brain-mind principles
Process of strands is Learning for Effective Teaching
Content for strands is Teaching for Effective Learning
ADULTS:Learning for
Effective Teaching
InquiryCollaboration
DataOwnership
STUDENTS:
Teaching for Effective Learning:Democratic
RelationshipsClarity
Invested CognitionFeedback
Expert learners
Learn
er ce
nte
red
prin
ciple
s
43
Process for creating and enacting the framework embodies “capacity” Bottom up Incorporated research and practitioner
knowledge Back and forth between the strands “Not a thing” Ongoing opportunities for meaning-making vs.
“Rollout”
44
Baseline findings about Efficacious Instruction in RUSD Teacher clarity is the strongest domain,
relationships and engagement are the weakest
No significant differences in how Hispanic students experience instruction
Quality of instruction as experienced by students drops slowly from 4th to 8th grade, bottoms out in 9th, then climbs again until 12th
Compared to students, teachers overestimate the quality of relationships, underestimate quality of feedback
45
What was learned about district transformation? Slower than reform People want to “thingify” Monitor the balance between ownership and
coherence Changing power relationships at the top is
hardest Broadening the teacher role isn’t easy either
People do need to be taught skills of collaboration and inquiry
Capacity and buy-in are easier to build in the process of doing authentic work