Date post: | 14-Apr-2017 |
Category: |
Health & Medicine |
Upload: | iamrareval2015 |
View: | 316 times |
Download: | 0 times |
1
Evaluating Revalidation: Lessons from the Evaluation of Complex
Interventions
IAMRA Revalidation Symposium
Sanjeev Sridharan The Evaluation Centre for Complex Health
InterventionsUniversity of Toronto &St. Michael’s Hospital
3
What is evaluation? A useful but perhaps incomplete
definition • Evaluation is defined both as a means of
assessing performance and to identify alternative ways to deliver
• “evaluation is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.”
4
Purpose of evaluation
• Assessing merit and wortho Causal questions, RCT, observational studies
• Programme and organizational improvemento Formative evaluation
• Oversight and compliance• Knowledge development
o Neglected purpose of many evaluationso Processes and active ingredientso Testing assumptions
6
Features of complex interventions
The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not
linear, and involve negotiation and feedback at each stage.
Interventions are embedded in social systems and how they work is shaped by this context.
Interventions are prone to modification as they are implemented.
Interventions are open systems and change through learning as stakeholders come to understand them.
7
System Dynamic Approaches (Sterman,
2006)
• Constantly changing;• Governed by feedback;• Non-linear, History-dependent;• Adaptive and evolving;• Characterized by trade-offs;• Policy resistance: “The result is policy
resistance, the tendency for interventions to be defeated by the system’s response to the intervention itself.”
8
“Solutions” Can Also Create New Problems
Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904.Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.
Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention
itself. -- Meadows, Richardson, Bruckman
9
System-as-Cause
Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999. Available at <http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf>.Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991.Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.
11
Intervention System
Components
of the Interventio
n
Stabilityof the
components
Impacts on evaluation
design
1. Elements of the intervention: What are the elements of a revalidation
system?
12
2. What is the theory of change by which revalidation can enhance quality of care? What are the connections between the various domains (trust, knowledge, communication, quality)?
• Context and Support Conditions• Mechanisms• Heterogeneity• Leverage
14
4. Anticipated Timeline of Impact: What is your timeline of impact? Would revalidation impact quality of care within the time frame of the evaluation?
Outcome 1
Outcome 2
Outcome 3
Program
Years1 2 3 4 5 60
16
5. What do you wish to learn from the
evaluation of revalidation? Improvement vs.
Merit and Worth
17
Evaluation/Performanc
e Measureme
nt
Pathways of
Evaluation Influence: Collective, Individual
and Interperso
nal
6. How will the evaluation of Revalidation impact practice?
Impacts on Policy,
Practice and the Political Agenda
19
Some ideas for design• Is a comparison group possible?• Longitudinal • Ecology of Evidence• Involving the stakeholders• Balance an improvement vs. Merit and worth lens• Multilevel• Attribution vs. Contribution
20
ProgramEvaluation Design/Met
hodProgram Impacts
Initial Program Theory
Initial Impacts
Areas of Uncertainty
Learning from
Innovative Methods
Emergent Program TheoryScenario 2
Scenario 1
(8) Learning from Methods: How do you learn along the way?
22
Program
Larger System
Spread Innovations
Components of System
Innovative Practice
Contextual Learning
(9) Framework of Spread
23
Framework of Learning
Decision to Sustain
Match between
Anticipated Findings of
Impact/Performan
ce Trajectory and Actual Performan
ce
Evaluation Results
Other Factors
(10) A Framework for Sustainability
24
How will this approach help with learning about key domains discussed by GMC?
• 1. Knowledge, skills, and performance• 2. Safety and quality• 3. Communication, partnership, and teamwork• 4. Maintaining Trust.
25
Clarifying revalidationo Theory of Change; Realist Review
• Connect the different domains• Connections between different mechanisms• Support conditions
o Timeline of Impact
o What are you uncertain of? What evidence do you need? Whom are you trying to persuade? • Ecology of Evidence
o ProcessesTough tests of processesMechanisms and differential mechanismsFormative vs. Summative
o Knowledge Translation
o Learning, accountability and accountability towards learning