©2012KarenA.BlaseandDeanL.Fixsen 1
From Admiring to Using Data to Improve Systems and Practices
Caryn Ward, Ph.D.
Dale Cusumano, Ph.D.
Angela Preston Ph.D.
Objectives
• Understand components of a research-based problem solving model;
• Understand differences and appropriate contexts within which to use problem solving or improvement cycle approaches
• Gain skills in applying improvement model processes including general problem solving approaches to all levels and layers of schools as systems.
Problem
Solution
Effective?Efficient?
Out of
Time
©2012KarenA.BlaseandDeanL.Fixsen 2
Quick Peek at Active Implementation Frameworks
Effective & Usable
Innovations
Stages
©Copyright Dean Fixsen and Karen Blase
DRIVERS
§ Whatcriticalsupportsareneededtomakethischange?Whatistheinfrastructure?
STAGES
§ Whatstepsleadtosuccessfulimplementation?TEAMS
§ Whotakesresponsibilityforandhelpsguidethechangeprocess?EFFECTIVE&USABLEINNOVATIONS
§ Whatexactlyarepeoplesayinganddoingthatmakesthingsbetterforchildrenandfamilies?
IMPROVEMENTCYCLES
§ Howcanwecreatemorehospitableenvironments,efficientlysolveproblemsandgetbetter?
Drivers
Teams
Cycles
Effective & Usable
Innovations
Stages DriversTeams
Cycles
Active Implementation Framework in a Formula for Success
Problem
Identification
Plan
Evaluation
Did it
work?
How will
we do this?Plan
Development
Why?
Problem
Analysis
Is there a
problem?
Horner, R. H., Newton, J. S., Todd, A. W., Algozzine, B. Algozzine,
K., Cusumano, D. L., & Preston, A. I. (2015). Team Initiated Problem Solving. Online at www.PBIS.org
Problem
Identification
Plan
Evaluation
Did it
work?
How will
we do this?Plan
Development
Why?
Problem
Analysis
Is there a
problem?
©2012KarenA.BlaseandDeanL.Fixsen 3
Welcome to Cherry Elementary SchoolMeet the Building
Implementation
TeamTeams What When
Who
Why Where
Problem Identification begins with precision
Welcome to Cherry Elementary SchoolMeet the Building
Implementation
Team
Is there a problem?
What is the problem?
Who is having the problem?
Why?
©2012KarenA.BlaseandDeanL.Fixsen 4
Possible Problem Precision StatementThemajorityofthirdgradestudents(who)arenotreadingfluentlyorcomprehendingwrittentext(what,when,where)because…..??????????(why).
Problem
Identification
Why?
Problem
Analysis
Analyze your Active Implementation Frameworks
Effective & Usable
Innovations
Stages
©Copyright Dean Fixsen and Karen Blase
DRIVERS
§ Arecriticalsupportsthatareneededtomakethechangepresent?Whatistheinfrastructure?
STAGES
§ Weretherightstepsandactivitiestakenattherighttime?TEAMS
§ Aretherightpeopletakingtakingresponsibilityforguidingthechangeprocess?EFFECTIVE&USABLEINNOVATIONS
§ Areinnovationslearnable,teachable,doable,andreadilyassessableinpractice?IMPROVEMENTCYCLES
§ Arehospitableenvironmentsestablishedtosupportefficientproblemsolvingandbarrierbusting?
Drivers
Teams
Cycles
Usable Innovations: How often are…
Core Components Operationally Defined?
Fidelity Assessments Correlated with Outcomes?
18% 7%
1,200+ Outcome StudiesMoncher & Prinz, 1991;Gresham, et al., 1993; Dane & Schneider, 1998; Durlak & DuPre, 2008
©2012KarenA.BlaseandDeanL.Fixsen 5
Hexagon Tool Usable Innovations
Fidelity Assessment
System
Clear Description
Essential FunctionsUsable
Innovation
Operational Definitions
Clear philosophy,
values, and
principles
Inclusion and
exclusion criteria
Identification of what
must be present
(core components)
What is done in
practice
(Practice Profiles).
Did we do what we
said we would do?
Usable Innovation
Performance Assessments
Clear Description
Essential FunctionsUsable
Innovation
Operational Definitions
Teachable
LearnableDoable
Assessable in practice
Performance Assessments
Clear Description
Essential FunctionsUsable
Innovation
Operational Definitions
Teachable
LearnableDoable
Assessable in practice
Does Cherry Elementary have an Effective and Usable Innovation?
✅✅
✅
✅
©2012KarenA.BlaseandDeanL.Fixsen 6
Do we have a research-based usable innovation?
Areweimplementing
itwell?
Analyze your Active Implementation Frameworks
Effective & Usable
Innovations
Stages
©Copyright Dean Fixsen and Karen Blase
DRIVERS
§ Arecriticalsupportsthatareneededtomakethechangepresent?Whatistheinfrastructure?
STAGES
§ Weretherightstepsandactivitiestakenattherighttime?TEAMS
§ Aretherightpeopletakingtakingresponsibilityforguidingthechangeprocess?EFFECTIVE&USABLEINNOVATIONS
§ Areinnovationslearnable,teachable,doable,andreadilyassessableinpractice?IMPROVEMENTCYCLES
§ Arehospitableenvironmentsestablishedtosupportefficientproblemsolvingandbarrierbusting?
Drivers
Teams
Cycles
0
1
2
Clear Instruction Demonstrates Tasks Engages Students Provides Feedback Adjusts to Responses Provides Opportunities
Adjusts to Engagement
Total Average
Observation Tool for Instructional Supports and Systems (OTISS)Average Scores for 3rd Grade - Winter 2017
3rd
Fully in Place
Partiallyin Place
NotObserved
Adequate Quality Instruction
Clear Instruction
DemonstratesTasks
Provides Feedback
Provides Opportunities
Engages Students
Adjusts to Responses
Adjusts to Engagement
©2012KarenA.BlaseandDeanL.Fixsen 7
Meet the District Implementation Team (DIT)
Teams
What systems are in place to support the school?
Competency
Leadership Organization
©2012KarenA.BlaseandDeanL.Fixsen 8
Possible Problem Problem StatementThe majority of third grade students (who) are not reading fluently or
comprehending written text (what, when, where) because reading content has not been delivered with clear instruction, instructional tasks demonstrated and frequent opportunities to practice with feedback provided (why). It is hypothesized that this is due to limited coaching of teachers in the delivery of the reading innovation (why).
Problem
Identification
How will
we do this?Plan
Development
Why?
Problem
Analysis
Is there a
problem?
Goals across teams65% of 3rd grade students will meet
grade level expectations in reading
fluency by May of 2017
OTISS scores will increase by 0.5
overall by March
DCA Competency subscale will
increase to 65% by August of
2017
• Buildrefreshertraining
• Createcoachingplan
Plan Development- Competency Driver
©2012KarenA.BlaseandDeanL.Fixsen 9
Plan
Evaluation
How will
we do this?Plan
Development
Why?
Problem
Analysis
Is there a
problem?
Did it
work?
Problem
IdentificationAnalyze Fidelity and Outcome Data
OTISS scores will increase by 0.5
overall by March
65% of 3rd grade students will meet
grade level expectations in reading
fluency by May of 2017
©2012KarenA.BlaseandDeanL.Fixsen 10
Problem
Identification
Plan
Evaluation
Plan
Development
Problem
Analysis
Celebrate!
“Insanityisdoingthesamethingover
andoverandexpectingdifferentresults”
AlbertEinstein
"Ifwekeepdoingwhat
wehavebeendoing,
we'llkeepgettingwhat
we'vealwaysgotten”
PaulBatalden
“Ifwewantbetteroutcomes,wemustchange
somethinginthesystem.Todothis,weneedto
understandoursystems.”
DonaldBerwick
“Everysystemisperfectlydesignedtoget
theresultsitgets”
Senge Deming
Three Types of Improvement Cycles
Rapid Cycle
Problem SolvingUsability Testing
Practice-Policy
Communication
Cycle
Rapid cycle, short
term, few people
Initially identifying
problems and
solutions
Address immediate and
pressing issues
Make incremental
improvements quickly
Rapid Cycle (PDSA) Problem SolvingRapidcycle(PDSA)problemsolving
Shewhart (1931); Deming (1986); Taylor et al. (2014)
Plan
DoStudy
Act
What do you PLAN to do?
Can you do it in
practice?Did you do it?
What happened?
What can be changed
and improved?
©2012KarenA.BlaseandDeanL.Fixsen 11
Outcomes from a Meta-analysis of Improvement Studies (N = 73)
Plan
DoStudy
Act
(Taylor et al. 2014)
100%
??%15%
14%
19%Three Types of Improvement Cycles
Rapid Cycle
Problem SolvingUsability Testing
Practice-Policy
Communication
Cycle
Medium cycle, medium-
term, more people
Testing the feasibility
of solutions
Developing administrative
supports
Usability Testing Aplannedseriesoftestsofaninnovationorofimplementationprocessesthattest thefeasibilityandimpactofanewwayofworkpriortorollingoutmorebroadly
Rubin (1994); Nielsen (2000); Akin et al. (2013)
Plan
DoStudy
Act Plan
DoStudy
Act Plan
DoStudy
Act
More is learned from 4 cycles with 5 participants each than from 1 pilot test with 20 participants
Three Types of Improvement Cycles
Rapid Cycle
Problem SolvingUsability Testing
Practice-Policy
Communication
Cycle How are supports and barriers
leveraged across the system?
©2012KarenA.BlaseandDeanL.Fixsen 12
ImplementationTeams:Competencies&Functions
EnsuringSupport
EngagingtheCommunity
CreatingEnablingContext
Effective Practices
Implementation Infrastructure
Improvement Cycles
System Change
Linking Team Structure
Building
ImplementationTeam
District
ImplementationTeam
Regional
ImplementationTeam
State/System
ImplementationTeam• EnsureSupport
• EngagetheCommunity
• CreateEnablingContext
Implementation Team
StateManagement
Team
TeachersInnovations
Students
Syst
em
Cha
nge
Adaptive Challenges• Duplication
• Fragmentation
• Hiring criteria
• Salaries
• Credentialing
• Licensing
• Time/ scheduling
• Union contracts
• RFP methods
• Federal/ State laws
“Ext
erna
l” S
yste
m C
hang
e Su
ppor
t
Prac
tice-
Polic
y C
omm
unic
atio
n Lo
opPolicy Enabled Practice
“The fault
cannot lie in
the part
responsible
for the
repair.”
Ashby (1956)
Practice-Policy Feedback Loop
School“BIT”
District“DIT”
Regional Entity“RIT”
Problem Solving PDSA Cycles
Systematic
Use data
Resolve problems
Study and address needs of
practice in context
Process initiated by a
“problem”
Setting event = discrepancy between expected and
observed
Initiated to study the impact of “change”
Setting event = purposeful effort to
study practices
Making things betterClosing gaps
Planned“Reactive”
©2012KarenA.BlaseandDeanL.Fixsen 13
ActivityAnswer these questions:
1. Which approach (e.g., general problem
solving, PDSA cycles, usability testing)
would you consider using and why?
2. What question(s) will you answer or
what is your precise problem statement
you will try to define?
3. What is your goal or outcome?
4. What steps will you follow?
5. What data are needed, from whom,
and what is the frequency?
Example
ActivityAnswer these questions:
1. Which approach (e.g., general problem
solving, PDSA cycles, usability testing)
would you consider using and why?
2. What question(s) will you answer or
what is your precise problem statement
you will try to define?
3. What is your goal or outcome?
4. What steps will you follow?
5. What data are needed, from whom,
and what is the frequency?
Objectives
• Understand components of a research-based problem solving model;
• Understand differences and appropriate contexts within which to use problem solving or improvement cycle approaches
• Gain skills in applying improvement model processes including general problem solving approaches to all levels and layers of schools as systems.
©2012KarenA.BlaseandDeanL.Fixsen 14
Frank Porter Graham Child Development InstituteUniversity of North Carolina
Chapel Hill, NC
http://nirn.fpg.unc.edu/www.scalingup.org
www.implementationconference.org
Caryn Ward, Ph.D.Dale Cusumano, Ph.D.Angela Preston, Ph.D.
For More Information
Citation and CopyrightThis document is based on the work of the National Implementation Research
Network (NIRN).
© 2013-2016 Allison Metz, Leah Bartley, Jonathan Green, Laura Louison, Sandy
Naoom, Barbara Sims, and Caryn Ward
This content is licensed under Creative Commons license CC BY-NC-ND,
Attribution-NonCommercial-NoDerivs . You are free to share, copy, distribute and
transmit the work under the following conditions: Attribution — You must attribute
the work in the manner specified by the author or licensor (but not in any way that
suggests that they endorse you or your use of the work); Noncommercial — You
may not use this work for commercial purposes; No Derivative Works — You may
not alter, transform, or build upon this work. Any of the above conditions can be
waived if you get permission from the copyright holder.
email: [email protected]
web: http://nirn.fpg.unc.edu
The mission of the National Implementation Research Network (NIRN) is to
contribute to the best practices and science of implementation, organization
change, and system reinvention to improve outcomes across the spectrum of
human services.