Di erences in Di erence and Randomized...

M2R “Development Economics”Empirical Methods in Development Economics

Université Paris 1 Panthéon Sorbonne

Differences in Difference and RandomizedExperiment

Rémi [email protected]

Semester 1, Academic year 2016-2017

1 / 57

Introduction

I We’ve already reviewed together a number of populareconometric methods to isolate causal/treatment effects:

I Propensity Score Matching;

I Instrumental Variables;

I Heckman procedure;

I Regression Discontinuity Design;

2 / 57

Introduction

I The purpose of the class today is to present two other keyimpact evaluation methods: Difference-in-Difference andRandomized Experiments.

I Outline:

1. Back to the selection bias

2. Solving the selection bias with a Difference-in-Differenceapproach

3. Solving the selection bias with a Randomized Experiment

3 / 57


I Suppose we want to measure the impact of textbooks onlearning.

I We denote:

I YTi the average test score of children in a given school i if theschool has textbooks (i.e. if treated);

I Y Ci the average test score of children in a given school i if theschool has no textbooks (i.e. if untreated).

4 / 57


I The objective of policy makers is to be able to estimate the“treatment effect”, defined as follows:

E (Y Ti − Y Ci |T ).

I This quantity indeed allows to measure how schools whichhad textbooks would have fared in the absence oftextbooks, compared to those which had textbooks.

I For sure, we cannot compute this quantity by observing aschool i both with and without books at the same time: whileevery school has two potential outcomes, only one is observedfor each school.

5 / 57


I Imagine you get access to data on a large number of schoolsin one developing country.

I Some schools have textbooks and others do not.

I Would you manage to measure the treatment effectE (Y Ti − Y Ci |T ) by computing the difference between theaverage test scores in schools with textbooks and in schoolswithout textbooks?

6 / 57


I Computing the difference between the average test scores inschools with textbooks and in schools without textbooks boilsdown to computing the following quantity:

D = E (Y Ti |T )− E (Y Ci |C ).

I Is D equal to E (Y Ti − Y Ci |T )?

7 / 57


I This second term (E (Y Ci |T )− E (Y Ci |C )) is the selectionbias.

I It captures the fact that treatment schools may have haddifferent test scores on average even if they had not beentreated.

I Put differently, the selection bias will exist (i.e.: it won’t beequal to 0) as soon as schools in the treatment group andschools in the control group initially differ with respect toobserved and unobserved characteristics that influencestudents’ grade (“initially” means that these characteristicsare different prior to any treatment).

9 / 57


I For instance:

I if schools that received textbooks were schools where parentsconsider education a priority, then the selection bias will beupward (i.e.: E (Y Ci |T ) > E (Y Ci |C )): one will conclude thatthe impact of textbooks on student’s score is more positivethan it actually is.

I if schools that received textbooks were targeted because theywere located in particularly disadvantaged communities, thenthe selection bias will be downward (i.e.:E (Y Ci |T ) < E (Y Ci |C )): one will conclude that the impact oftextbooks on student’s score is more negative than it actuallyis.

10 / 57


I How can one eliminate the selection bias?

11 / 57

2. Solving the selection bias with a Diff-in-Diff approach

I To implement a Diff-in-Diff, one needs information onpre-period differences in outcomes between treatment andcontrol group so as to control for pre-existing differencesbetween the groups (and therefore neutralize the selectionbias).

12 / 57


I We suppose 2 periods of time:

I at date t = 0, a baseline (or pre-treatment) survey isconducted in a set of different schools. Just after this baselinesurvey, some of the schools are selected by an NGO (in a nonrandom manner) for being treated (i.e: for getting access totextbooks);

I at date t = 1, a post-treatment survey is conducted in boththe set of treated and untreated schools.

13 / 57


I More specifically:

I the baseline survey allows to get information on E (Y Ci0 |T ) andon E (Y Ci0 |C ), the test score in period 0 (where none of theschools is treated) in schools that are destined to be treatedand in schools that will remain untreated respectively;

I the post-treatment survey allows to get information onE (YTi1 |T ) and E (Y Ci1 |C ), the average test score in period 1 inschools that have been treated right after period t = 0 and inschools that belong to the control group respectively.

14 / 57

2. Solving the selection bias with a Diff-in-Diff approachI Illustration in case the most disadvantaged schools are

selected by the NGO for being treated:

School

characteristics

Students’

characteristics

Students’

characteristics

School

characteristics

Changes in

time-varying

characteristics

Treatment

effect

School

characteristics

School

characteristics

Students’

characteristics

Students’

characteristics

Changes in

time-varying

characteristics

T0:

value of Y in

the treatment

at date t=0

Factors

influencing

Y

T1:

value of Y in

the treatment

at date t=1

C0:

value of Y in

the control at

date t=0

C1:

value of Y in

the control at

date t=1

15 / 57


I The “difference-in-difference” estimator denoted DD is givenby:

DD = (T1 − C1)− (T0 − C0)= [E (Y Ti1 |T )− E (Y Ci1 |C )]− [E (Y Ci0 |T )− E (Y Ci0 |C )]= [E (Y Ti1 |T )− E (Y Ci0 |T )]− [E (Y Ci1 |C )− E (Y Ci0 |C )]= (T1 − T0)− (C1 − C0)= (CTVCT + TE )− CTVCC ,

where CTVCT and CTVCC stand for “changes in time-varyingcharacteristics” in the treatment and in the control grouprespectively, and TE stands for the “treatment effect”.

16 / 57


I In others words, DD coincides with the treatment effect TEunder the critical assumption that, in the absence of anytreatment, both the treatment and the control groups ofschools would have followed parallel trends over time (nomean reversion effect, no anticipation of the treatment... etc).

I This means that, when many years are available, you shouldplot the series of average outcomes for Treatment and Controlgroups and see whether trends are parallel and whether thereis a sudden change just after the reform for the Treatmentgroup.

17 / 57


I The regression counterpart to obtain DD is given by

Y = α + βTreat + γTime + δ(Treat ∗ Time) + u, (1)

where:

I Treat is equal to 1 if the school belongs to the treatmentgroup (it is equal to 0 if it belongs to the control group);

I Time is equal to 1 if the period is post-treatment (it is equalto 0 if the period is pre-treatment).

18 / 57


I δ captures DD (which coincides with the treatment effect assoon as one can reasonably assume that, in the absence ofany treatment, both the treatment and the control groups ofschools would have followed parallel trends over time):

δ = (T1 − T0)− (C1 − C0).

19 / 57

2. Solving the selection bias: a Diff-in-Diff approach

I Relying on a regression analysis has a big advantage: it allowsto partly relax the “parallel trend” assumption.

I It indeed allows to control for the effect of schoolcharacteristics that are likely to not evolve over time in aparallel way in the treatment and in the ontrol group.

I Moreover, as soon as there is more than one period in thepre-treatment phase and more than one period in thepost-treatment phase, one can control for group specific timetrends (they capture the linear evolution over time of theoutcome Yi in each group).

I To do so, one must introduce the following controls inEquation (2): Trend and (Treat ∗ Trend) where Trendcaptures the value of the period.

20 / 57


I Keep in mind that the treatment should not be concomitantto policies which might differentially impact the treatmentand the control group.

I For instance, the introduction of textbooks by the NGOshould not be accompanied by other interventions by theNGO aiming at improving test scores in the treatment group(e.g. organisation of remedial classes).

I Otherwise, it is impossible to attribute the treatment effect tothe introduction of textbooks only.

21 / 57

2. Solving the selection bias with a Diff-in-Diff approachDiff-in-Diff in practice

I We can try to find a “natural experiment” that allows us toidentify the impact of a policy

I E.g. An unexpected change in policyI E.g. A policy that only affects 16 years-olds but not 15

years-oldsI In general, exploit variation of policies in time and space

I The quality of the comparison group determines the quality ofthe evaluation

22 / 57


I When there is more than two periods, one can use aregression with individual and time fixed effects

I Individual fixed effects: control for time-invariant individualcharacteristics

I Time fixed effects: effects that are common to all groups atonce particular point in time, or common trend

I Valid only when the policy change has an immediate impacton the outcome variable

I If there is a delay in the impact of the policy change, we doneed to use lagged treatment variables

23 / 57


I Diff-id-Diff attributes any difference in trends between thetreatment and control group that occur at the same time asthe intervention, to that intervention

I If there are other factors that affect the difference in trendsbetween the two groups, then the estimation will be biased!

I → The common/parralel trend assumptionI Cannot be tested directly but one can show graphical evidence

24 / 57

Source: World Bank (2011) “Impact Evaluation in Practice”

25 / 57


I Sensitivity Analysis for diff-in-diffI One need to convince that the effect is not driven by other

factorsI Placebo tests:

I Use a “fake” treatment groupI For instance for previous yearsI Using a treatment group a population that was NOT affectedI If the DD estimate is different from 0, trends are not parallel,

and our original DD is likely to be biased

I Different comparison groupI You should obtain the same estimates

I Different outcome variableI Use an outcome variable which is NOT affected by the

intervention, using the same comparison group and treatmentyear

I If the DD estimate is different from zero, we have a problem

26 / 57


I Other issuesI Bertrand et al. (2004): When outcomes within the unit of

time/group are correlated, OLS standard errors understate thestandard deviation of the DD estimator

I Solution: Cluster at the i levelI Solution 2: Collapsing the data into pre- and post- periods

produce consistent standard errors

I Autor (2003): add specific linear individual time trend (orquadratic individual time trend)

I To check that results are not driven by a specific trend

27 / 57

2. Solving the selection bias with a Diff-in-Diff approachDuflo (2001), “Schooling and Labor Market Consequences of School Construction inIndonesia: evidence from an unusual policy experiment”, American Economic Review

I Research QuestionI What is the effect of school infrastructure on educational

achievement?I What is the effect of educational achievement on salary level?

I Program DescriptionI 1973-1978: the Indonesian Government built 61000 schools

(one school per 500 children between 5 and 14 years-old)I Enrollment rate increased from 69% to 85% between

1973-1978I The number of schools built in each region depended on the

number of children out of school in those regions in 1972,before the start of the program

28 / 57


I IdentificationI Two sources of variations in the intensity of the program for a

given individualI By region: there is variation in the number of schools received

in each regionI By age: (1) Children who were older than 12 years in 1972

did not benefit from the program (2) The younger a child was1972, the more it benefited from the program because shespent more time in the new schools.

29 / 57


I DataI 1995 population census w/ individual data on birth date, 1995

salary level, 1995 level of educationI The intensity of the building program in the birth region of

each individualI Focus on men born between 1950 and 1972

I First stepI We simplify the intensity of the program (high vs low) and the

groups of children (young who benefited and older who did notbenefit)

30 / 57


31 / 57


32 / 57


33 / 57


34 / 57


35 / 57


36 / 57


37 / 57


38 / 57

3. Solving the selection bias with a Randomized Exp.

I Randomized experiments are far from being a new estimationmethod.

I They were institutionalized by researchers in psychology andmedicine, at the end of the 19th century.

I However, this estimation method was extended to research inDevelopment Economics only recently, by Esther Duflo, whohas been Professor at the MIT (Massachusetts Institute ofTechnology) since 1999 and co-founder (with Abhijit Banerjeeand Sendhil Mullainathan) of the Jameel Poverty Action Lab(J-PAL) in 2003.

I J-PAL was established to estimate the impact of a wide rangeof development policies, all over the world.

39 / 57

3. Solving the selection bias with a Randomized Exp.2.1. Mechanics

I A randomized experiment aiming at measuring the impact oftextbooks on students’ test scores would consist in:

I First, selecting a sample of N schools;

I Second, randomly selecting half of them to assign them to thetreatment group.

40 / 57


I In the context of a randomized experiment, will an approachconsisting in computing the following quantityD = E (Y Ti |T )− E (Y Ci |C ) based on a post-treatment surveyallow to measure the treatment effect?

I Remember that D can be rewritten as follows:

D = E (Y Ti − Y Ci |T ) + (E (Y Ci |T )− E (Y Ci |C )).

I Answering the question therefore amounts to determiningwhether the selection bias measured byE (Y Ci |T )− E (Y Ci |C ) is equal to 0.

41 / 57


I The fact that the treatment has been randomly assignedensures that, on average, students in schools belonging to thetreatment group are identical to students in schools belongingto the control group prior to the treatment.

I We therefore have that E (Y Ci |T )− E (Y Ci |C ) = 0.

42 / 57


I In other words, differences in the outcome Y between thetreatment and the control group after the random assignmentof the treatment are only attributable to their differences inexposure to the treatment.

I Therefore:

D = E (Y Ti − Y Ci |T ) = E (Y Ti − Y Ci ) = ATE ,

where ATE is the Average Treatment Effect (the effect ofbeing treated in the population if individuals are randomlyassigned to treatment).

43 / 57


I The regression counterpart to obtain the ATE is given by

Yi = α + βT + ui , (2)

where T is a dummy for assignment to the treatment group.

I Indeed,β = E (Y Ti − Y Ci ).

44 / 57

3. Solving the selection bias with a Randomized Exp.2.2. Other sources of bias and short overview of potential solutions

I Although randomized experiments allow to get rid of theselection bias, other sources of bias can arise:

1. Externalities: it happens when untreated individuals areaffected by the treatment.

2. Attrition: it happens when some treated individuals leave theoriginal sample.

3. Hawthorne and John Henry effects: it happens when theevaluation itself may cause the treatment or comparison groupto change its behavior.

I Biases related to 1. and 2. are also called “partialcompliance” biases (“perfect compliance” meaning that thetreatment did reach ALL the individuals in the treatmentgroup, and ONLY them).

45 / 57


I Concerning externalities, they will lead to an underestimationof the treatment effect if they are positive, and to anoverestimation of the treatment effect if they are negative.

I Some techniques can be implemented to estimate themagnitude of such externalities and therefore neutralize thebias they induce.

I For instance, when evaluating the impact of using fertilizerson crop yields, one may worry about information externalities:individuals in the treatment group (those who receive anincentive to use fertilizers) may talk to individuals in thecontrol group about the benefits/drawbacks of usingfertilizers.

46 / 57


I One way to solve the problem is to ask farmers in both thetreatment and the control group the name of the 3 farmersthey discuss agriculture with the most often (we refer to themas “friends” in the following).

I To get an idea of the magnitude of the information spilloversbetween the treatment and the control group, one cancompare:

I the use of fertilizers by friends in the control group of farmersin the treatment group with

I the use of fertilizers by friends in the control group of farmersin the control group.

47 / 57


I Concerning attrition, it won’t induce a bias if it is random;however, it will lead to a bias as soon as it is correlated withthe impact the treatment has on each individual.

I For instance, bias will arise if those who are benefiting theleast from a program tend to drop out of the sample.

I Therefore, managing attrition during the data collectionprocess is essential.

48 / 57


I More precisely, this requires collecting good information in abaseline questionnaire on how to find each individual again,should he decide to leave the group after the treatment (byasking for instance the names of relatives that can beinterviewed if the respondent cannot be found during thepost-treatment survey).

I For sure, following up with ALL attritors is too expensive, butfollowing up with only a random sample of the attritors is agood alternative.

49 / 57


I What are the Hawthorne and John Henry effects exactly?

I Hawthorne effect: individuals in the treatment group,because they are conscious of being observed, may alter theirbehavior during the experiment (compared to what it usuallyis) to please the experimenter (for instance, teachers in schoolswhich received textbooks may also decide to work harder).

I John Henry effect: individuals in the control group, in casethey are aware of being a control group, may feel offended toget this experimental status and therefore could react by alsoaltering their behavior (for instance, teachers in schools whichreceived no textbooks may decide either to work harder or toslack off).

50 / 57


I One way to get rid of the HJH effects is to continue tomonitor the impact of a development program, after theofficial experiment is over.

I The fact that the measured impact is similar when theprogram is not being officially evaluated any more and whenthe program is officially evaluated means that it is not due tothe HJH effects. If they are not similar, the estimation of thetreatment effect should rely on the “post-post-treatment”survey.

51 / 57

3. Solving the selection bias with a Randomized Exp.2.3. What about external validity?

I So far, we have mainly focused on issues of internal validitywhich is whether an OLS estimate of β in Equation (2)captures the treatment effect without bias.

I External validity is about whether the treatment effect wemeasure would carry over to other samples or populations.

I Ensuring external validity is considered as the greatestchallenge faced by randomized experiments.

52 / 57


I Discussing the external validity of a randomized experimentrequires to:

I First, discuss whether the results obtained among members ofcommunity X at date t would hold if the randomizedexperiment is conducted among members of anothercommunity at another point in time;

I Second, discuss whether the results obtained among asub-sample of individuals in a given country would hold if therandomized experiment is conducted among the entirepopulation of this country.

53 / 57


I To address the first requirement:

I One surely has to replicate the randomized experiment indifferent communities and at different points in time (thesereplications constitute an important activity at J-PAL);

I One can also identify the observed individual characteristicsthat magnify or mitigate the treatment effect. For instance, ifone finds that a development program works for poor womenin country i but that the magnitude of the positive effect ofthis program decreases with these women’s income, then thissuggests that the development program wouldn’t be assuccessful among a set of middle-income women in country i .

54 / 57


I To address the second requirement one must think about thepossible effect of scaling up the program.

I Scaling up the program can indeed trigger off generalequilibrium effects that are non existent when one runs arandomized experiment in a small area.

55 / 57


I Indeed, the fact that the randomized experiment is conductedon a small scale means that one observes a partial equilibriumeffect: one looks at the impact of the treatment every otherthings being held constant.

I Put differently, the treatment affects a portion of a country’spopulation that is too small to have an impact onmacro-economic variables (like wages or prices) which couldhave a feedback effect on the outcome of interest.

I If the randomized experiment is scaled up however, thesefeedback effects are possible. They can considerably challengethe results obtained on a small scale.

56 / 57

IntroductionIntroduction1. Back to the selection bias1. Back to the selection bias1. Back to the selection bias1. Back to the selection bias1. Back to the selection bias1. Back to the selection bias1. Back to the selection bias1. Back to the selection bias2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias: a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach2. Solving the selection bias with a Diff-in-Diff approach3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.3. Solving the selection bias with a Randomized Exp.

Date post:	08-Feb-2021
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Di erences in Di erence and Randomized...

Documents