Evaluation Designs in the IMATCHINE project:

Regression DiscontinuityCluster Randomized Trial

Presentation by Manoj Mohanan, Duke University

1

CO

HE

SIV

E-I

ND

IA

Commercial Break! A word about COHESIVE-India• COHESIVE-India: Collaboration for Health Systems Impact

Evaluation in India.• Jerry La Forgia (World Bank)• Grant Miller (Stanford U. & NBER)• Manoj Mohanan (Duke U.)• Marcos Vera-Hernandez (U. College London & IFS)

• Focus on evaluation in health sector using a combination of quasi-experimental and experimental methods

• Provide critical input into the design of policies and interventions, to provide rigorous evidence on how to improve performance as part of evaluation.

• Collaborating with SAMBODHI and DFID-India on IMATCHINE project

2

CO

HE

SIV

E-I

ND

IA

Key Question in Impact Evaluation• What is the counterfactual? What would have happened if

this program did not exist?• Identifying CAUSAL effects is the central to impact

evaluation• Two common fallacies in making causal inference:

• Cum hoc ergo propter hoc• Post hoc ergo propter hoc

Cum hoc ergo propter hoc

Source: http://ssgreenberg.name/PoliticsBlog/2009/04/03/diversion-highway-fatalities-and-lemons/

The post hoc fallacy• Observe people on the street @ 9AM to predict weather?• In medicine, there is a disease progression, hence you can

use temporal changes to make causal claims. In social sciences, less so.

Visualizing problems in evaluation

Participants

Counter factual

This is why the Before-After method is sometimes called The “Counterfeit Counterfactual” method!!

TIME -------------------------------------

OU

TC

OM

ES

---

----

----

----

----

----

----

Outline of Today’s Presentation

• Projects being evaluated in the IMATCHINE project• Gujarat: (Regression Discontinuity)

Chiranjeevi Yojana (CY)

• Karnataka: (Experimental Evaluation)Thayi Bhagya Yojana (TBY)

7

CO

HE

SIV

E-I

ND

IA

Gujarat: Chiranjeevi Yojana (CY)

• Introduced in 2005• Response to acute shortage of OBGYNS in

public sector• Leveraging presence of private providers in

rural areas• Pays approx Rs. 1700 to accredited provider

per delivery• Eligibility: BPL card holder or BPL eligible

(~23% of population; total pop 55 million)8

CO

HE

SIV

E-I

ND

IA

More on CY

• First introduced in 5 backward districts 2005-2007 and then rolled out across the state Jan 2007-08 onwards

• B/w 2005 – Feb 2008, CY had covered over 165,000 deliveries provided by 852 providers

• Claims: (in 2009)Has increased institutional births from a national

average of 57% to over 80%Has reduced MMR & IMR Won WSJ Innovations Award & is now widely

looked upon as the “model” 9

CO

HE

SIV

E-I

ND

IA

Previous ‘Evaluations’ of CY

Source: Mavalankar, D. et al. 2009. Saving mothers and newborns through an innovative partnership with private sector obstetricians: Chiranjeevi scheme of Gujarat, India. International Journal of Gynecology and Obstetrics 107: 271–276.

• Have typically used data from CY facilities to extrapolate estimated utilization and health benefits in the population

10

CO

HE

SIV

E-I

ND

IA

Gujarat: Regression Discontinuity Evaluation Design

• The program uses the BPL line as the eligibility criteria, which allows a quasi-experimental RD design

Since the BPL score is continuous, households immediately next to each other across the BPL line are comparably similar to each other

Discontinuity in program eligibility across the BPL line allow us to test for differences in outcomes that can be attributed to the program

11

CO

HE

SIV

E-I

ND

IA

Gujarat: RD Design …contd…Two main challenges with BPL criteria:

Lots of other state run programs use BPL eligibility (such as JSY, food subsidies),:

− SOLUTION: We use a “Difference-in-Difference” framework by relying on timing of introduction and expansion of the program to 5 districts in 2005 and all over the state in 2007 to try to identify program effects of CY.

•

Manipulation of BPL criteria, resulting in misclassification− SOLUTION: We rely on a “Fuzzy” regression discontinuity

strategy, where we calculate the “true” eligibility and then instrument for CY participation using simulated eligibility

12

CO

HE

SIV

E-I

ND

IA

Year Pilot Districts Expanded Districts

2005 BPL + CY BPL

2008 BPL + CY BPL + CY

Gujarat: RD Analysis• First Stage: (instrumenting for CY Participation)

• Second Stage: (Change in avg. outcomes)

• Key outcomes:Rates of institutional delivery Study is not powered to detect effects on IMR or MMR,

but we will collect data on these measures anyway in addition to measures of morbidity that are more common. Also will collect data on HH characteristics

13

CO

HE

SIV

E-I

ND

IA

ihk khhih CONTROLSBPLBELOWeparticipat

ikikk khihih CONTROLSBPLeparticipatoutcome

Gujarat: RD Analysis using the Geographic Spread

• Accounting for CY roll out in time and space (2nd difference estimate)

• Second Stage: (change on avg. outcomes : same equation accounting for roll out)

14

CO

HE

SIV

E-I

ND

IA

ihdyydk ikkh

dyhdyhihdy

CONTROLSBPL

xCYBELOWCYBELOWeparticipat

)(

ihdyydk ikk

hihdydyhihdy

CONTROLS

BPLeparticipatCYBELOWoutcome

BREAK B/W RD & CRT

15

CO

HE

SIV

E-I

ND

IA

Experimental Methods• The Randomized Controlled Trial

• Clearly, the gold standard• An obvious solution to the Prog Eval problem, although not

always practical:• Assigns treatment in a manner that is unrelated to outcomes

• Two important steps in randomized evaluation1. Randomly selecting potential participants from population

2. Randomly assign treatment to the group

• The RCT in IMATCHINE Project…

Karnataka: Thayi Bhagya Yojana (TBY)

• New Conditional Cash Transfer program of Rs. 1000 for women who prefer to give birth in the private sector

• Two components • (1) Prospective evaluation• Since the program uses a BPL eligibility like the

CY program, we will use a RD based method, combined with a difference-in-difference

We are working with the Govt. of KN to implement a baseline survey 17

CO

HE

SIV

E-I

ND

IA

KN : Experimental Evaluation of Provider Incentives (2nd

component)• Cluster randomized trial of incentives for

providers to estimate effect of incentives for improvements in process measures v/s outcome measuresArm 1: Provider incentives evaluated based on

performance on quality of care indicators

Arm 2: Provider incentives evaluated based on improvement of MCH outcomes in catchment area population

Arm 3: Control group, with no incentives

18

CO

HE

SIV

E-I

ND

IA

KN Incentives Experiment: Design Issues

• 180 rural clusters, in 3 groups of 60 each

• Each cluster is approximately at the level of the HOBLI (called kasba in N. India)

• On average: 3 OBGYN providers in each cluster

• Sample for the study: approx 550 doctors and 18,000 households (100 women who have a baby in catchment area of each cluster)

19

CO

HE

SIV

E-I

ND

IA

Key challenge from the RCT perspective• Recall the two important steps in randomized evaluation

1. Randomly selecting potential participants from population

2. Randomly assign treatment to the group

• The second one is relatively easy – just write a STATA code• The first one is the big challenge

• Need to identify providers whom we can include in the study• Both a conceptual and logistic challenge• Need to define eligibility based on objective of the

experiment

20

CO

HE

SIV

E-I

ND

IA

KN Incentives Experiment: Design Issues …contd…

• There are three key issues related to measurement in this study:

Definition of clusters and catchment area

Measurement of process measures of quality and health outcomes in a population

Identification of women who have had a baby and interviewing them in time 21

CO

HE

SIV

E-I

ND

IA

Measurement Issues – contd..• Some things to keep in mind:

• Sample Sizes in Cluster Randomized Trials … the devil called Intra-Cluster Correlation

• Careful attention to defining eligible population – mapping is VERY effort and resource intensive, but has HUGE pay offs

• How practical is it to do RCTs? • “Politically Robust Randomization” – Gary King et al.

• Policy Relevance….• Very important topic, but we don’t have time for this today.• Working in conjunction w state governments to ensure buy-

in and policy impact.• Findings from our research in 2011.

22

CO

HE

SIV

E-I

ND

IA

Thanks

• 3ie• Government of Gujarat• Government of Karnataka• DFID• World Bank • My colleagues at Sambodhi• For further details on project contact:

Ms. Manveen Kohli, Project Manager, IMATCHINE, imatchine@gmail.com 23

CO

HE

SIV

E-I

ND

IA