Date post: | 16-May-2018 |
Category: |
Documents |
Upload: | trinhxuyen |
View: | 236 times |
Download: | 1 times |
Using Contribution Analysis to Address Cause-Effect
Questions:Theory and Concepts
John MayneAdvisor on Public Sector Performance
2
John MayneAdvisor on Public Sector Performance
Outlinethe ideatheories of changecontribution analysiscomplexityconclusions
3
John MayneAdvisor on Public Sector Performance
The challengeAttribution for outcomes always a challengeStrong evaluations (such as RCTs) not always available or possibleA credible performance story needs to address attributionComplexity significantly complicates the issueWhat can be done?
4
John MayneAdvisor on Public Sector Performance
The ideaBased on the theory of change of the program,Buttressed by evidence validating the theory of change, Reinforced by examination of other influencing factors,Contribution analysis builds a reasonably credible case about the difference the program is making
5
John MayneAdvisor on Public Sector Performance
Proving CausalityThe gold standard debate (RCTs et al)AEA and EES: many methods capable of demonstrating scientific rigourMethodological appropriateness for given evaluation questionsCausal analysis: auto mechanic, air crashes, forensic work, doctors
6
John MayneAdvisor on Public Sector Performance
Contribution analysis: the theory
There is a postulated theory of changeThe activities of the program were implementedThe theory of change is supported by evidenceOther influencing factors have been assessed
7
John MayneAdvisor on Public Sector Performance
outputs(goods and services
produced by the program)
activities(how the program carries
out its work)
intermediate outcomes(the benefits and changes resulting from the outputs)
end outcomes(the final or long-term
consequences)
Examplesnegotiating, consulting, inspecting, drafting legislation
Exampleschecks delivered, advice given, people processed, information provided, reports produced
Examplessatisfied users, jobs found, equitable treatment, illegal entries stopped, better decisions made
Examplesenvironment improved, stronger economy, safer streets, energy saved
Immediate outcomes(the first level effects of the
outputs)
Examplesactions taken by the recipients, or behaviour changes
A results chain
External Factors
8
John MayneAdvisor on Public Sector Performance
outputs(goods and services
produced by the program)
activities(how the program carries
out its work)
intermediate outcomes(the benefits and changes resulting from the outputs)
end outcomes(the final or long-term
consequences)
Examplesnegotiating, consulting, inspecting, drafting legislation
Exampleschecks delivered, advice given, people processed, information provided, reports produced
Examplessatisfied users, jobs found, equitable treatment, illegal entries stopped, better decisions made
Examplesenvironment improved, stronger economy, safer streets, energy saved
Immediate outcomes(the first level effects of the
outputs)
Examplesactions taken by the recipients, or behaviour changes
Why will these
immediate outcomes
come about?
�Results chain links
External Factors
9
John MayneAdvisor on Public Sector Performance
Theories of changeA results chain with embedded assumptions and risks identifiedAn explanation of why the results chain is expected to work; what has to happen
Reduction in smoking
Anti-smoking campaign
Assumptions: target is reached, message is heard, message is convincing, no other major influences at work
Risks: target not reached, poor message, peer pressure very strong
10
John MayneAdvisor on Public Sector Performance
Theory of change analysisNeed to identify which of the links in the results chain have the weakest evidenceSome may be supported by prior researchSome may be well acceptedBut some may be a large leap of faith, or the subject of debateWith limited resources, these contested links are where effort should be focused
11
John MayneAdvisor on Public Sector Performance
Steps in Contribution Analysis
1. Set out the attribution problem to be addressed
2. Develop the postulated theory of change3. Gather the existing evidence on the ToC4. Assemble & assess the contribution story5. Seek out additional evidence6. Revise & strengthen the contribution
story7. Develop the complex contribution story
12
John MayneAdvisor on Public Sector Performance
1. Set out the attribution problem
Acknowledge the need to address attributionScope the attribution problem– What is really being asked– What level of confidence is needed?
Explore the “expected contribution”What are the other influencing factors?How plausible is a contribution?
13
John MayneAdvisor on Public Sector Performance
2. Develop the ToC and Risks to It
Build the postulated results chain and ToC– Identify roles played by other influencing
factors– Identify the risks to the assumptions– Determine how contested the ToC is
14
John MayneAdvisor on Public Sector Performance
3. Gather existing evidence
Assess the logical robustness of the ToCGather available evidence on– Results– Assumptions– Other influencing factors
15
John MayneAdvisor on Public Sector Performance
4. Assemble and assess the contribution story
Set out the contribution storyAssess its strengths and weaknesses
16
John MayneAdvisor on Public Sector Performance
5. Seek out additional evidence
Determine what additional evidence is neededIf needed, refine the ToCGather new evidence
17
John MayneAdvisor on Public Sector Performance
Strengthening Techniques
Survey knowledgeable others involved Track program variations and their impacts (time, location, strength)Undertake case studiesConduct a component evaluationIdentify and synthesize relevant research or evaluationUse multiple lines of evidence
18
John MayneAdvisor on Public Sector Performance
6. Revise and strengthen the contribution story
Build a more credible contribution storyReassess its strengths and weaknessesRevisit step 5
19
John MayneAdvisor on Public Sector Performance
Causality in ComplexityMany factorsMany interventionsNo room for experimentingNot everything is plannedCausality a real mess– How to know? Many theory strands– What to know?
20
John MayneAdvisor on Public Sector Performance
7. Develop the complex contribution story
Develop the contribution story for each theory strandDevelop the contribution story for any general theory of change
21
John MayneAdvisor on Public Sector Performance
Evaluating in ComplexityGoal should be understanding not determining what works per se (Sanderson, Pawson)CA seeks to understand what is working and whyRobust predictions cannot be made but we can observe the consequences and learnToC will be revised frequentlyWe will learn incrementally; CA can help
22
John MayneAdvisor on Public Sector Performance
Some ReferencesSanderson (2000). Evaluating in Complex Policy Systems. Evaluation 6(4):433-454Pawson (2006). Simple Principles for the Evaluation of Complex Programmes. In Public Health Evidence. OxfordSampson (2007). Developing Robust Approaches to Evaluating Social Programmes. Evaluation 13(4): 477-493Eoyang & Berkas (1998). Evaluating Performance in a Complex Adaptive System. http://www.winternet.com/~eoyang/CAS_Abstract.htm
23
John MayneAdvisor on Public Sector Performance
Levels of contribution analysis
Minimalist contribution analysisContribution analysis of direct influenceContribution analysis of indirect influence