Toward a Technology of Treatment Integrity
Ronnie DetrichWing Institute
APBS, Denver, Colorado2011
Goals for Today
• Review the link between data-based decision making and treatment integrity.
• Discuss the dimensions of treatment integrity.
• Review methods for assuring treatment integrity.
Data-based Decision Making and Treatment Integrity
• Data-based decision making at heart of PBS and RtI.• The impact of PBS and RtI depends on the
effectiveness of specific interventions.• The effectiveness of interventions is a function of the
integrity with which they are implemented.• The quality of decisions about effects of an
intervention is directly linked to the quality of implementation.
Data-based Decision Making and Treatment Integrity
• Student data provides feedback about progress.• If we know about adequacy of treatment integrity
then can make decisions: Adequacy of intervention Adequacy of implementation
If implementation is inadequate then focus should be on improving implementation.
If implementation is adequate then focus should be on changing intervention so student can succeed.
Grade Level Standard
Aim Line
Trend Line
Data-based Decision Making and Treatment Integrity
Positive NegativeH
igh
Low
Continue Intervention Change Intervention
Unknown reason Unknown reason• Intervention problem?
• Implementation problem?
• Other life changes?
• Unknown intervention?
• Intervention is effective?
OutcomeIn
tegr
ityPositive Negative
Hig
hLo
w
Assumptions about Treatment Integrity
• Students cannot benefit from interventions they do not experience.
• Unless educators know what they are supposed to be doing they cannot do more of it.
Dimensions of Treatment Integrity(Dane & Schneider,
1998)
• Exposure (Dosage): the extent to which participants are exposed to the intervention as prescribed. Curricula usually prescribe frequency and duration of
exposure that is necessary for benefit. Ex: 3/week for 30 minutes/session.
Failing to satisfy either can impact student benefit.Ex: 1/week for 30 minutes.
Dimensions of Treatment Integrity(Dane & Schneider,
1998)
• Adherence: the extent to which the components of an intervention are delivered as prescribed. Most commonly measured dimension. It is necessary but not sufficient to produce benefits.
Adherence with low dosage not likely to produce positive outcomes.
Dimensions of Treatment Integrity(Dane & Schneider,
1998)
• Quality of delivery: qualitative measure of how well the intervention is implemented. Importance has been acknowledged for years.
Have not developed good measures or how to influence it. Possible measures through social validity methods:
Enthusiasm Sincerity Variations in inflection and content of speech.
Variables that Influence Treatment Integrity
• Organizational Systems that are perceived by teachers to be supportive
and provide strong leadership have higher levels of integrity.
• Personal characteristics Burnout: Inverse relation between measures of teacher
burnout and treatment integrity. Personal efficacy: Positive correlation between measures
of personal efficacy and treatment integrity.
Variables that Influence Treatment Integrity
• Characteristics of Intervention Complexity
As complexity increases integrity decreases.
Relationship between Complexity, Precision, and Treatment Integrity
• Be as precise as necessary but no more.co
mpl
exity
precision
Catch’em being good
Check inCheck out
Individualized support plan
Integrity
Variables that Influence Treatment Integrity
• Characteristics of Intervention Perceived effectiveness. Acceptability. Contextual fit. Necessary resources available.
Variables that Influence Treatment Integrity
• Training Training is necessary but not sufficient. Not all training is equal.
OUTCOMES
(% of Participants who demonstrate knowledge, demonstrate new skills in a training setting,
and use new skills in the classroom)
TRAININGCOMPONENTS
Knowledge SkillDemonstration
Use in the Classroom
Theory and Discussion
10%
5% 0%
..+Demonstration in Training 30% 20% 0%
…+ Practice & Feedback in Training
60% 60% 5%
…+ Coaching in Classroom 95% 95% 95%
Joyce and Showers, 2002
Not All Training is Equal
What We Know About Treatment Integrity
• It is estimated that drug prevention programs are implemented with integrity only 19% of the time. (Hallfors & Godette, 2002) Programs have extensive research base as being effective. This may be a generous estimate.
• No reason to assume that other programs are immune to poor implementation.
What We Know About Treatment Integrity
• Most of what we know has been developed at the level of individual student support plans.
• SET is a measure of treatment integrity at the school level. Does not address what is happening at the level of the
individual classrooms or individual students.
What Do We Know About Treatment Integrity and Student Behavior?
• Different levels of integrity result in different levels of student behavior.
(Wilder, Atwell, & Wine, 2006) • High integrity followed by declines in integrity has
limited disruptive effect on student behavior. (Northup, Fisher, Kahng, Harrel, & Kurtz, 1997)
• Low levels of integrity followed by increases in integrity do not produce the same level of student response as when integrity high from the beginning.
(Groskreutz,, Groskreutz, & Higbee, 2011)
What Do We Know About Treatment Integrity and Student Behavior?
• Implications Make sure that integrity is high at the beginning of
intervention. It is better to start with high levels of integrity and let it decline
than to start with low integrity and try to increase it. Maximizes impact of intervention.
What Do We Know About Integrity of Interventions at the Universal Level?
• Kovaleski, Gickling, Morrow, & Swank (1999)Evaluated high vs low implementation of Instructional Support Teams
(IST). School-wide organizational change. Students benefited from IST processes only when implemented with high
fidelity. Implementing with low fidelity resulted in no better outcomes for
students than control group not exposed to IST processes. Having structures in place was not sufficient to assure high fidelity. Fidelity assessed one time per year.
What Do We Know About Integrity of Interventions at the Universal Level?
• Horner (2005) Effect of high fidelity vs low fidelity on office discipline
referrals. Schools that implemented with high fidelity had 25% fewer
office referrals for major rule violations than schools that did not meet fidelity criterion.
Fidelity measures taken 2 times per year.
How Do We Assure High Levels of Integrity?
• Technology for assuring treatment integrity is emerging.
• Most of the research is at level of individual support plans.
• Feedback is most common approach.
Mortenson & Witt, 1998
Effective Performance Feedback
• More frequent the feedback better effects (Jones, Wickstrom, & Friman, 1997Mortensen & Witt, 1998).
• Daily better than weekly.• Immediate better than delayed.• Immediate more preferred than delayed.
Limitations of Performance Feedback
• Requires direct observation.• May be too resource intensive to implement at large
scale. Would require significant restructuring to implement
effectively.
Alternative Approaches: Teacher Self Report
• Teacher rate their own implementation. Teacher ratings tend to overestimate accuracy of
implementation ( Wickstrom, Jones, Lafleaur, & Witt). Teachers rated integrity at 54% accurate. Direct observation= 4%.
Integrity may be increased by rating immediately following intervention session.
Integrity may be improved by having teacher score video tape.
It may be possible to teach teachers to rate more accurately (see self-evaluation literature).
Alternative Approaches: Quizzes
• Quizzes (Detrich et al., 2001)
Staff quizzed weekly on elements of multi-component individualized behavior support plans.Given feedback on quiz but no feedback on actual implementation
of support plan.4 versions of the quiz. One question per element of the plan
(student preferences, antecedent interventions, teaching replacement behavior, responding to misbehavior).
Alternative Approaches: Quizzes
• Advantages: Easier to implement than direct observation. May be easy to implement at scale when standard
protocols are utilized.
• Limitations Does not produce maintenance effects in the absence of
quizzes.
Alternative Approaches: Video Modeling
• Has been utilized to train a standard protocol as well as individualized interventions. Problem solving curriculum. Discrete trial instruction. Functional assessment.
Alternative Approaches: Video Modeling
• Advantages: For standard protocols can be used to train many
implementers. Demonstration of implementation can be standardized. Flexible scheduling of training.
• Limitations Has not been evaluated at large scale. Coaching resources required to assure modeling was
effective.
Alternative Approaches: Checklists
Alternative Approaches: Checklists
• Advantages: Standardize implementation. Increase integrity especially adherence.
• Limitations: Upfront effort to develop. Often low acceptability.
Alternative Approaches: Intensive Professional Development
• University faculty/coach assigned to school for full year (Klingner, Vaughn, Hughes, & Arguelles, 1999). Provided in-depth training on three reading programs. Classroom based coaching. Problem solving implementation. Produced moderate levels of treatment integrity at three
years follow-up.
Alternative Approaches: Intensive Professional Development
• Advantages: Highly trained resource on site.
• Limitations Difficult to replicate at large scale. Expensive.
Unresolved Issues
• Adoption or adaptation?Must we implement exactly as prescribed or can we
adjust to fit local circumstances?
• Research suggests that programs are almost always adapted.
• Presumably adapted to improve outcomes. Some adaptation for other reasons:
Better fit teaching style. Do not like some elements of program.
Unresolved Issues
• What does this mean for treatment integrity? Are adaptations systematic?
If so, then we can assess integrity. What parts of program can we adapt without doing
harm to effectiveness? Core elements?
If adapted is program still research-based? If teachers allowed to adapt then program more
acceptable. Teacher’s made better adaptation of reading programs if they
were well grounded in principles. (Klingner, Vaughn, Hughes, & Arguellas, 1999).
Treatment Integrity and PBS
• Regardless of the level of the intervention, it is necessary to know that it was implemented with integrity. High integrity is necessary in a data based decision
making approach.
• Integrity should be assessed at the same level that the intervention is being evaluated.
Treatment Integrity and PBS
• A program or intervention is a set of protocols that guides behavior of the adult. If protocols are not followed then by definition the
program has not been implemented or sustained.
• PBS is an excellent model for making decisions about when, where, and how to intervene. Intervention without process for assuring integrity is likely
to result in wasted effort.
Taking Treatment Integrity Measurement to Scale: Data-based Decision Making
• Insufficient resources to frequently observe all educators working with students to determine adequacy of implementation.
• One solution: Measure student behavior and analyze at different units of analysis:SchoolClassroomIndividual student
• Any level there is gap between obtained and expected performance assess levels of treatment integrity.
Taking Treatment Integrity Measurement to Scale: Data-based Decision Making
• Requires that data are routinely collected across all levels. RtI:
CBM on all students 3/year. PBS
Continuous measurement of Office Discipline Referrals.
• Systems have to be in place to assure integrity data-based decision making process.
Where are We?
• Implementation is where good interventions go to die unless there is active plan to assess and influence integrity.
• Research-based approaches to influencing treatment integrity are emerging.
• We have ethical obligation to allocate resources to influencing treatment integrity.
Thank you
Copies may be downloaded atwinginstitute.org