Evidence-based Practice for Applied Evidence-based Practice for Applied Behavior Analysts: Necessary or Behavior Analysts: Necessary or
RedundantRedundant
Ronnie Detrich, Wing Institute
Tim Slocum, Utah State University
Teri Lewis, Oregon State University
Trina Spencer, Northern Arizona University
Susan Wilczynski, Ball State University
Goals for TodayGoals for Today
Describe basic concepts of evidence-based practice.
Describe integrated decision making framework.
Two Ways of Thinking about EBPTwo Ways of Thinking about EBP
An intervention found to have strong research support. (Cook, Tankersley, & Landrum, 2009)
Decision making process that informs all professional decisions. (Sakett, Straus, Richardson, Rosenberg, & Haynes, 2000)
Two Ways of Thinking about Two Ways of Thinking about EBPEBP
Using same term (EBP) to describe two different constructs creates confusion:
Empirically supported treatments (EST)-interventions that meet defined standards of quality and quantity.
Evidence-based practice-decision making process.
Evidence-based Practice and Evidence-based Practice and Applied Behavior AnalysisApplied Behavior Analysis
EBP is framework for guiding decisions of practitioners.
Decisions are based on the integration of:
Best available evidence
Client values and context
Professional judgment
Consistent with foundational principles of applied behavior analysis:
Data-based decision making
Consideration of client values
Considerations of contextual fit
Commitment to research-based treatment
Evidence-based Practice and Evidence-based Practice and the Research-Practice Gapthe Research-Practice Gap
Across disciplines, great concern about the discrepancy between what is known from research about effective treatments and the interventions practitioners routinely employ.
EBP is basis for closing the gap.
How Does EBP Narrow the How Does EBP Narrow the Research to Practice Gap?Research to Practice Gap?
Practitioners must make decisions “now” even when research evidence is absent or incomplete. What is to be the basis for decisions?
Decisions informed by best available evidence allows practitioners to: Select
Adapt to fit local circumstances
Modify on the basis of progress monitoring data
These decisions require professional judgment.
Dilemma for PractitionersDilemma for Practitioners
Practitioners must make many decisions daily.
EBP assumes there is evidence for all decisions and that the relevant evidence is accessible.
How do practitioners incorporate evidence into all decisions?
The challenge of The challenge of Best available EvidenceBest available Evidence
GoalGoal
The best available evidence should pervade the practice of Applied Behavior Analysis.
What is the “Best Available Evidence” for ABA practice?
How do we systematically identify it?
Best Available EvidenceBest Available Evidence What do we mean by “best”?
Quality: research methods and outcomes
Relevance: close match with our applied question in terms of:
• Participants
• Treatment
• Outcomes
• Context
Amount: number of participants, studies, investigators
Best available evidenceBest available evidence
Quality
High
Low
Low High
Better e
vidence
Relevance(P, T, O, C)
Empirically Supported Treatments
Empirically Supported Treatments
Best Available EvidenceBest Available Evidence
What do we mean by “best available”?
We should use the best of what is available,
• This may mean using extremely high quality evidence
• Or it may mean using evidence that is less certain.
“Unlimited skepticism is equally the child of imbecility as implicit credulity.”
Dugald Stewart
“Unlimited skepticism is equally the child of imbecility as implicit credulity.”
Dugald Stewart
Best available evidenceBest available evidence
Quality
High
Low
Low High
Relevance(P, T, O, C)
Best available evidenceBest available evidence
Quality
High
Low
Low High
Empirically-Supported Treatment
Empirically-Supported Treatment
Relevance(P, T, O, C)
Need to GeneralizeUncertainty
The practical problemThe practical problem
Practitioners must often make decisions with insufficient empirical support.
What are they to do?
Make the best possible inferences from imperfect evidence?
Make decisions without using systematic evidence?
The practical problemThe practical problem
If Evidence-Based Practice of ABA is to be a pervasive model for professional decision-making…
then we need ways to identify the best available evidence when the evidence is imperfect.
Accessing the Best Available Accessing the Best Available EvidenceEvidence
1. Systematic reviews – The foundation
Identifies empirically supported treatments
2. Alternative types of review
Improve our ability to glean recommendations from imperfect literature
3. Other units of practice - beyond the package
Using what we know about intervention elements and principles
4. Progress Monitoring
The best evidence about what works for this particular case
1. Systematic reviews1. Systematic reviews
Systematic ReviewsSystematic Reviews
• Systematic EBP review (e.g., WWC, BEE, NAC)
Establish standards for:
Identifying research base
o Participantso Interventionso Comparisonso Outcomeso Settings
Quality of evidence
Quantity of evidence
Unit typically limited to “programs” (treatment packages)
Systematic ReviewsSystematic Reviews
Reduced bias
Transparency
Objectivity
Rigorous methods
Reduced risk of false positives
Exclusive reliance on high quality evidence
Often fail to identify sufficient evidence
Not informed by lower quality evidence
Higher risk of false negative.
Strengths Limitations
2. Alternative types of 2. Alternative types of reviewsreviews
a. What Works Practice Guidesa. What Works Practice Guides
Practice guide system:
Provide:
expert recommendations and
specific level of supporting evidence.
Allows broader generalization from specific studies
Allows for expert recommendations on topics where evidence is spare
a. What Works Practice Guidesa. What Works Practice Guides
Expert interpretation of research
Includes “level of evidence” ratings – based on systematic review.
Expert interpretation may introduce bias & increase uncertainty
Strengths Limitations
b. Best Practice Panelsb. Best Practice Panels
• Best practice panel
Panel selection
Organization selects “expert” panelists
Panel can be broadly or narrowly constructed
Key to validity
o Face validity o Functional aspects of validity
Recommendations based on group consensus
b. Best Practice Panelsb. Best Practice Panels
May include diverse perspectives:
Researcher
Practitioner
Consumer
Allows for interpretation of research
Tend to include factors other than research
social validity?
contextual fit?
May lack transparency:
Selection of panel
Criteria for “best practice”: politics or science?
Tend to include factors other than research
Bias?
Strengths Limitations
c. Narrative Reviewsc. Narrative Reviews
• Narrative review
E.g., APA Handbook of Behavior Analysis;NASP Best Practices;
• Experts review research base to establish recommendations
• Allows experts to draw on wide range of evidence
• Allows for expert interpretation of patterns of relevant findings
c. Narrative Reviewsc. Narrative Reviews
Allows for broad generalization from specific studies to implications for practice.
Can incorporate many sources of evidence and logic/theory
No methodology to reduce bias.
Selecting experts
Relevant research base
Criteria for “best practice”
Strength of evidence rating
Strengths Limitations
3. Other units of practice3. Other units of practice
What is a “treatment”?What is a “treatment”?
The best available evidence can validate:
← Comprehensive schoolwide systems
← Multi-component instructional or behavior packages
← Specific components/elements/kernels
← Principles of behavior and learning
a. Practice Elements and a. Practice Elements and KernelsKernels
ExamplesDifferential praise, self-monitoring, frequent
student responses with corrections
How they are validatedKernels
• Kernels are implemented, outcomes are measured.
Practice elements
• Effective treatment packages are analyzed.
• Practice elements are components included in most effective packages.
a. Practice Elements & Kernelsa. Practice Elements & Kernels
More flexible than multi-component programs.
May be used to assemble custom interventions directed at specific problems
Can supplement information on multi-component packages
A set of effective components may not produce an effective package.
Assembling package is time-consuming and requires high level of skill.
Strengths Limitations
b. Principles of Learning and Behaviorb. Principles of Learning and Behavior
Examples
Differential reinforcement, extinction
Principles of using examples and non-examples
How they are validated
Numerous studies across wide variety of populations and contexts.
b. Principles of Learning and Behaviorb. Principles of Learning and Behavior
The most basic and flexible guides to intervention.
Apply to huge range of educational problems including modifications & adaptations.
Can supplement evidence on packages and components
Do not provide specific interventions. Principles must be applied – this process is uncertain.
Application requires extremely high level of expertise.
Strengths Limitations
4. Progress monitoring4. Progress monitoring
4. Progress Monitoring4. Progress Monitoring
Progress monitoring can validate:
The specific treatment (as modified)
with the specific client
on the specific outcomes
in the specific context
No other evidence can substitute.
4. Progress Monitoring4. Progress Monitoring
Strengths
The best evidence on whether this particular program is working.
Provides basis for data based decision making
Limitations
Does not help:
initial selection of treatments
select modifications
ConclusionConclusion
The best available evidence to support pervasive evidence-based practice can be derived from:Empirically supported treatments
Evidence from alternative types of reviews
Evidence on alternative units of practice
And should always include progress monitoring.
Client Values and ContextClient Values and ContextTeri Lewis
Oregon State University
Beyond ESTBeyond EST
Even when there are EST, practitioners may not choose these interventions and/or implement them when they are recommended.
In education innovations come and go in 18-48 months (Latham, 1988).
So, if we have the right So, if we have the right answers why aren’t EST answers why aren’t EST adopted and sustained?adopted and sustained?
Face Validity
Client-values and context need to be included and respected
Historical PerspectiveHistorical Perspective Baer, Wolf & Risley (1968)
Immediate and important change in behavior that has practical value as determined “…by the interest which society shows in the problems.” (p. 92)
Wolf (1978) Social significance of the goals.
Societal value
Social appropriateness of the procedures. Treatment acceptability
Social importance of the effects. Consumer satisfaction
““Client”Client”
Individual(s) who are invested in outcomes and/or are critical to the behavior change process (e.g., Baer et al, 1968; Heward et al, 2005)
Individual who is the focus of the behavior change
Parents and family members, teachers, mentors, colleagues, employers
Organization, society
• In contrast to EBP, BACB (2010) relies on a narrow definition
• Individual receiving services from a behavior analyst
• Decision about who is a client based on:
• Ethics
• Acceptability & Fidelity
• Effectiveness
• Maintenance and sustainability
Context Context
Just as we focus on behavior within an environmental context , we need to consider the context for selection and implementation of interventions
Contextual-fit (e.g., Albin et al 1996)
Values, skills, goals and stressors of the implementors and those impacted by the target behavior
Implementation and Implementation and AcceptabilityAcceptability
Motivating operations such as values, goals and stressors
Compatibility with other aspects of the context (e.g., routines, resources, policies)
Reinforcement and punishment associated with implementation
Clients ultimately decide whether the intervention effects are judged successful(Wolf, 1978)
Strain, Barton & Dunlap (2012) assert that incorporating client-values successfully informs decision-making:
Design of service delivery
Technical support to key implementors
When to fade intervention components
Identification of unanticipated events
Focus of future research needs
Ethical PerspectiveEthical Perspective
Three basic and fundamental questions (Cooper, Heron & Heward, 2007)
What does it mean to be a behavior analyst?
What is the right thing to do?
What is worth doing?
• Social Validity
• Client Values
SummarySummary
Including client values into assessment, intervention selection and implementation
Respects all individuals perspectives
Increases acceptability
Improves decision-making
Increases fidelity and sustainability
Specifically, client values inform selection of treatment and the methods of treatment
Art of behavioral interventions
But, most importantly, it allows behavior analysts the opportunity to implement our EST in applied settings and to produce important changes in behavior that have practical value
Professional judgmentProfessional judgment
Professional JudgmentProfessional Judgment
Process by which the best available evidence is applied to individual cases in specific contexts.
An evidence-based practice framework provides an opportunity for behavior analysts to understand the role of professional judgment so that we can ensure all behavior analysts are wise decision makers and ultimate improve our effectiveness.
Evidence-based Practice Evidence-based Practice FrameworkFramework
Spencer, Detrich, & Slocum, 2012
Client and Context
Behavior analysts use Behavior analysts use professional judgment…professional judgment…
…when formulating a practical question based on the presenting needs of the client and context.
Hudson engages dangerous behaviors such as climbing houses to get attention and because climbing is fun. What treatment should I use?
Client and Context
…to determine what evidence to look for, where to look for it, and to make sense of it in relation to the practical question and client.
Conduct a search for a systematic review or studies on treatments for dangerous behaviors that involve attention and automatic reinforcement functions.
Client and Context
Behavior analysts use Behavior analysts use professional judgment…professional judgment…
…to weigh the quality and relevance of the currently available scientific knowledge in relation to the client and context to select a treatment.
Differential reinforcement of other behaviors and a regular routine of appropriate replacement activities.
Behavior analysts use Behavior analysts use professional judgment…professional judgment…
Client and Context
Client and Context
…to adapt a selected treatment to be suitable for the specific client and context based on the best available evidence.
For every day that Hudson does not engage in dangerous behaviors, he earns 30 minutes of a special activity. In addition, he participates in a rock climbing club for children every week.
Behavior analysts use Behavior analysts use professional judgment…professional judgment…
Client and Context
…to implement and monitor a selected treatment that was adapted for a specific client and context, based on the best available evidence.
Hudson can monitor his dangerous behaviors and record their absence on a calendar. He can report to his mother every evening and request 30 minutes of a special activity.
Behavior analysts use Behavior analysts use professional judgment…professional judgment…
Client and Context
Behavior analysts use Behavior analysts use professional judgment…professional judgment…
…to determine when a socially meaningful outcome has been achieved, based on the specific client and the best available evidence.
Spencer, Detrich, & Slocum, 2012
Professional JudgmentProfessional Judgment
Behavior analysts use professional judgment in their practice every day.
Importantly, these judgments are a necessary part of Applied Behavior Analysis.
The pervasiveness of professional judgment suggests that we should be very skilled at this activity.
Bases of Professional JudgmentBases of Professional Judgment
Behavior analysts can draw from the available scientific bases to strengthen professional judgments.
Principles of learning and behavior
Ongoing progress monitoring
Bases of Professional JudgmentBases of Professional Judgment
Principles of behavior, grounded in decades of experimental and applied research, serve as a foundation for judgments made by behavior analysts.
Behavior analysts can identify the principles that contribute to specific decisions (e.g., discrimination, differential reinforcement, extinction, schedules of reinforcement).
Bases of Professional JudgmentBases of Professional Judgment
Most of our decisions can and should be informed by close and continual contact with relevant outcome data.This does not mean that we do not use
professional judgment.
It means our judgments are data-based.
Ongoing progress monitoring is THE best evidence of what works with specific individuals in specific contexts regardless of the strength of the empirical research.
Professional JudgmentProfessional Judgment
Given professional judgment is ubiquitous, necessary and it has a scientific basis, it serves our profession to embrace it, develop it, and promote it.
Doing so, has implications for training and professional development.
Training ProfessionalsTraining Professionals
A great deal of training is necessary to build the expertise required to be effective behavior analysts.
The BACB training sequence is designed to ensure that behavior analytic practitioners are prepared to make good judgments.
Training ProfessionalsTraining Professionals
Coursework is important, but professional judgment is primarily shaped through supervised field experiences.
Guided or mentored experiences are necessary to teach the subtle discriminations required to be an effective behavior analytic practitioner.
Take Home PointsTake Home Points
Professional judgment is pervasive in our practice and necessary for behavior analysts to be effective.
Professional judgment can be strengthened through an explicit recognition of its role, regular contact with empirical research, careful integration of principles and tactics, and ongoing measurement of relevant outcomes.
Professional judgment can be developed and enhanced via quality preservice training, supervised field experiences, and real world contingencies.
RecommendationsRecommendations
Adopt a decision-making framework to help guide practitioners to make judgments based on the best available evidence and client and context.
Organize our scientific knowledge so that principles and tactics are more accessible to practitioners who rely on them as a basis for professional judgment.
Infuse behavior analytic training programs with professional decision making.
Conclusions & Contingencies for Conclusions & Contingencies for EBP of ABAEBP of ABA
Formerly named: Contingencies and Evidence-Based Practice: Are we seeking punishers for
our scientists and practitioners?
Two Ways of Thinking about EBPTwo Ways of Thinking about EBP
An intervention found to have strong research support. (Cook, Tankersley, & Landrum, 2009)
Decision making process that informs all professional decisions. (Sakett, Straus, Richardson, Rosenberg, & Haynes, 2000)
EBP of ABAEBP of ABA
EBP is process for guiding decisions of practitioners.
Decisions are based on:
Best available evidence
Client values and context
Professional judgment
An EBP to ABA framework promotes translational methods at the practitioner level.
Translational MethodsTranslational Methods
Translational MethodsTranslational Methods
EBP of ABAEBP of ABA
Consistent with foundational principles of ABA
Data-based decision making
Consideration of client values
Considerations of contextual fit
Consonance of Behavior Analysis and EBPBehavior Analysis Evidence-Based Practice
Analytic BAE* – goal: establish a functional relationship based on local data collection and progress monitoring
Technological BAE* – methods that are sufficiently well defined to allow for treatment fidelity & replication
Conceptually Systematic
BAE* – well documented interventions are understood as examples of a unifying base
Professional Judgment – selecting/adapting tx based on an integrated system in lieu of ‘bag of tricks’
Applied Client values and context* – improve quality of life targeting socially significant behavior
Effective Client values and context* – effective change of practical and social significance
Generality BAE & Client values and context* – programming across time, settings, people
Behavioral BAE* – must be measureable, precise, reliable
Baer, Wolf, & Risley (1968)*BAE & Client values/Context always involve professional judgment
Consonance of ABA and EBPConsonance of ABA and EBP
Behavior Analysis Certification Board
Best Available Evidence
• Identify interventions based on “best available scientific evidence”
Client Values & Context
• Select treatments based on a range of variables including but not restricted to: client preferences; client’s current repertoires; supporting environments including the constraints on those environments; social validity
Professional Judgment
• Making determinations about treatment effectiveness based on graphical display of data (in various formats)
Implications of EBP of ABA: Implications of EBP of ABA: PracticePractice
Help provide an explicit rationale for decision-making that is consistent with the rationale forwarded by virtually every other discipline.
Decision-making framework is:
Transparent about how information regarding evidence is evaluated
Clear about why evidence is most appropriate for a given case
Cogent regarding how client values and contextual factors influence treatment selection (and the decision to retain/alter a treatment)
Highlights critical analytic thinking and why this analytic process should be applied to all treatments considered
Implications of EBP of ABA: Implications of EBP of ABA: PracticePractice
Demonstrates the strengths of ABA in a way that is accessible to clients and other stakeholdersClient and contextual factors
• Define the goals for treatment and acceptable methods for intervention
• Serves to prompt us to focus on these factors that have always been held important to our field
• To meet this goal: practitioners must be prepared to discuss the goodness of contextual fit
• By referring to these factors in practice, we undermine the fallacious argument that we are ‘cold’ in our decision-making
Implications of EBP of ABA: Implications of EBP of ABA: PracticePractice
Demonstrates the strengths of ABA in a way that is accessible to clients and other stakeholdersProfessional judgment
• An EBP of ABA framework gives us a way to discuss the expertise we bring to the table
• Common framework and language adopted by other fields but allows us to highlight our unique strengths in treatment evaluation on an individual basis
• Allows us to reject practitioners who argue EBP involves following a ‘list’
Implications of Ignoring Implications of Ignoring EBP of ABA: PracticeEBP of ABA: Practice
Portrayed as ‘ignorant’ of the most pervasively adopted framework across disciplines
Our professional practice may be undermined with: Practitioners in other disciplines
Clients & Families
Administrators
Insurance Reimbursement
Implications of Ignoring Implications of Ignoring EBP of ABA: ScienceEBP of ABA: Science
Federal agencies and funding
Institute for Educational Sciences
NIMH
DHHS
Future DirectionsFuture Directions
By highlighting ‘best available evidence’ we identify that further refinements are necessary
As highlighted by Smith (2013), we must continue to validate practice elements or kernels
Future DirectionsFuture Directions
Dissemination remains the great challenge of all fieldsHow do we get our information about
effective treatments organized well enough that it is accessible to practitioners?
How do we make decisions when the best available evidence is not ‘good enough?’
How do we set guides for optimal care for behavior analysts that will improve the overall quality of the services we provide?
Future DirectionsFuture Directions
Training of new behavior analysts
The EBP of ABA framework suggests the decision-making process is the cornerstone of good practice – not a list of treatments
Shifting greater emphasis on client values and contextual variables
Requiring a rationale for how decisions are made by students in all training activities and providing feedback will enhance professional judgment (and thus practice)