www.3ieimpact.orgPhilip Davies

How is a Policy Supposed to Work? – Theory of Change Analysis

Philip Davies

International Initiative for Impact Evaluation [3ie]

BCURE Evidence-Informed Decision-Making Capacity Building Workshop1st and 2nd June 2015Pretoria, South Africa

www.3ieimpact.orgPhilip Davies

• Evidence is information or data that supports, or rejects, a conclusion• Evidence is any thing that increases the estimate of the probability of the truthfulness of a proposition• Evidence is particularistic and generalisable• Often disagreement agreement on what counts as ‘evidence’• Evidence is always contestable/contested• Not all research is of equal value/sufficient quality• Single studies can misrepresent the balance of evidence• Hence, the need for systematic reviews/synthesis of evidence

Some Features of Evidence

www.3ieimpact.orgPhilip Davies

Quantitative Evidence

• Social Surveys – cross sectional and time series

• Longitudinal Studies – cohort and panel data

• Experimental Designs (random allocation)

•Quasi-Experimental Designs (Matched samples,

Interrupted Time Series, Regression analysis)

• Economic Methods - Cost-Benefit and Cost-

Effectiveness Analysis

• Statistical / Econometric Modelling

www.3ieimpact.orgPhilip Davies

Qualitative Evidence

• In-Depth interviews

• Focus Groups

•Other Consultative Designs

•Observational and Participant-Observational Studies

• Ethnography

•Documentary Analysis

•Oral Histories

• Case Studies

www.3ieimpact.orgPhilip Davies

Theory of Change/Logic Model/Programme Theory

• How is a policy/programme supposed to work?

• What activities, mechanisms, people, outputs have to be in place?

• And in what sequence – what is the causal chain?

• What resources are required – and are available?

• What data are required – and are available?

• Is the policy/programme feasible/achievable?

www.3ieimpact.orgPhilip Davies

Building a Theory of Change: From Inputs to Outcomes

www.3ieimpact.orgPhilip Davies

Constituent Features of a Theory of Change

Assumptions?

www.3ieimpact.orgPhilip Davies

Constituent Features of a Theory of ChangeData Required

•Surveys, statistics, demographic data •Qualitative data•Costs/benefits data•Systematic review data•Documentary analysis

• Performance data• Historical data• Diversity data• Qualitative data• Effectiveness data

• Stakeholder data• Qualitative data• Public opinion data• Effectiveness data

• Performance data• Effectiveness data• Stakeholder data• Qualitative data• Costs/benefits data

• Administrative data• Performance data• Costs/benefits data

• Administrative data• Performance data• Qualitative data

• Counterfactual data• Administrative data• Survey data, statistics• Cost/benefit data

www.3ieimpact.orgPhilip Davies

www.3ieimpact.orgPhilip Davies

Different Levels of Intervention

www.3ieimpact.orgPhilip Davies

Cognition Lifestyle Skills and BehavioursBiological

Individual Level

Demographic

Coping Ability Obesity/Health Eating

FamilyRole Modeling

Social SupportFriends

Peers

Social Norms Obesity/Health Eating

Social Environment

HomeAccess Availability

Bars/Pubs

Supermarkets

Opportunities Obesity/Health Eating

Physical Environment

GovernmentLegislation Tax and Price

Industry

Health Systems

LicensingAdvertising

(Availability)

Obesity/Health Eating

Macro Level Environment (Policy)

www.3ieimpact.orgPhilip Davies

Establishing the Policy Logic/Theory of ChangeBasic Principles

• Map out the causal chain

• Test the underlying assumptions with evidence

• Understand context and culture

• Anticipate heterogeneity (variance) across the population

• Rigorous evaluation of impact using an appropriate counterfactual

• Use mixed methods of evidence gathering

www.3ieimpact.orgPhilip Davies

Group Exercise

In groups, take a policy issue of your choice and:

• Map out how the policy is supposed to work, identifying:

i. What are the outputs and outcomes to be achieved?

ii. What people/agencies are likely to be involved?

iii.What mechanisms (training, planning, zoning, financing, communications, delivery etc) are likely to be involved?

iv.What time will be required to achieve the above factors?

v. What are the likely costs involved, and where should they fall?

www.3ieimpact.orgPhilip Davies

Thank you

Philip Davies on

Email: pdavies@3ieimpact.org

+44 (0)207 958 8350

Visit www.3ieimpact.org