Field Experiments:A Brief Overview of

Core Topics

Don Green Columbia University

Why Experiment?

Causal explanation vs. description: parameters and parameter estimation

The role of scientific procedures/design in making the case for unbiased inference

The value of transparent & intuitive science that involves ex ante design/planning and, in principle, limits the analyst’s discretion

Experiments provide benchmarks for evaluating other forms of research

Experiments at the border between theory and practice

Experiments allow us to study the effects of phenomena that seldom occur naturally (e.g., out-of-equilibrium phenomena)

Systematic experimental inquiry can lead to the discovery and development of new interventions

Momentum of experimental programs: each finding leads to new questions

Experimentation is not an assumption-free research procedure

Unbiased estimation of the sample average treatment effect hinges on assumptions of

Random assignment: independence of treatments and potential outcomes

Excludability: the only systematic way in which treatment assignments affect outcomes is via the treatment itself

Non-interference: potential outcomes are affected solely by the treatment that is administered to the subject, not to treatments administered to other subjects

Core assumptions must be kept in mind at all times when designing, analyzing, and interpreting experiments

To the extent possible, try to address core assumptions through experimental design rather than argumentation

What follows are suggestions for making experiments more convincing

Many of these suggestions grow out of lessons learned from my own mishaps and mistakes

Implementing random assignment

• CONSORT suggests that random allocation be performed by an outside team/person

• Should be carefully documented (for later RI purposes)

• If you are working with a naturally occurring random assignment, make sure you understand it in detail and verify its statistical properties

• Use a random number seed• Good to make the process transparent (sometimes

they are public lotteries), but beware of excludability violations

Working out the random assignment with research partners

• Your research partners may not fully understand what you mean when you say that treatments will be allocated randomly and that they won’t get to adjust or approve the random assignment

• Be pro-active with more palatable designs• Stepped wedge• Blocking to ensure that “preferred” or

“inexpensive” subjects have a high probability of being included

90% of life is just showing up

• Your experiment may get out of hand if you don’t supervise it, especially when interventions are launched and outcomes are measured

• Don’t expect your research partners to understand your rationale for doing things

• Beware of their good intentions (e.g., their desire to treat everyone)

Noncompliance is often a problem of miscommunication or misrepresentation

• Your partners may exaggerate how many subjects they are actually able to treat

• With supervision, you can use a randomly sorted treatment list and an exogenous stopping rule

• Measure compliance with the treatment as accurately as you can

• Irony that when research partners exaggerate their contact rate, the produce an underestimate of the CACE

Trade offs of orchestrating the treatments yourself

• On the one hand, implementing your own interventions is a pain in the neck and often expensive

• All the more reason to do a pilot test to see what you’re getting yourself into

• On the other hand, you get to do more interesting (and often more publishable) work that is less prone to collapse

• “Program evaluation” work is good for practice/apprenticeship but may involve uninteresting treatments or outcomes

Do not confuse random sampling with random assignment

• Random sampling refers to the procedure by which subjects are selected from some broader population.

• Random sampling facilitates generalization but not necessarily unbiased causal inference.

• Random assignment refers to the procedure by which subjects are allocated to treatment conditions. Its role is to make assignments independent of potential outcomes.

Maintain the integrity of random assignment

When randomly allocating subjects to experimental groups, follow and document a set procedure.

Do not redo the random assignment because you are disappointed that certain observations ended up in the “wrong” experimental group.

Design your experiment in ways that minimize attrition or cause attrition to be (arguably) unaffected by assignment

Maintain symmetry between treatment and control groups

Apart from receiving different treatments, subjects in different experimental conditions should be handled in an identical fashion.

The same standards should be used in measuring outcomes across the groups, and where possible, those tasked with measuring outcomes should be blind to experimental condition.

Make sure your analysis is appropriate to your random assignment procedure

• When the probability of being assigned to the treatment group varies across subjects, you must either analyze the experimental results separately for each stratum or apply inverse probability weights before pooling the data

• If you use restricted randomization, your estimation (weighting) and inference procedures (simulations) should take these restrictions into account

When estimating average treatment effects using regression (i.e., regressing outcomes on the treatment as well as a set of covariates), do not attach a causal interpretation to the covariates' coefficients• Covariates help reduce disturbance variability and

thus improve the precision with which treatment effects are estimated.

• Regression coefficients associated with the covariates are not unbiased estimates of average treatment effects because the covariates are not randomly assigned

• Present results both with and without covariates (or follow a pre-specified plan)

“Agnostic” regressions may include pre-treatment covariates ONLY.

Do not include post-treatment variables as predictors in order to detect the "mediating paths" by which a treatment influences the outcome, as this method of assessing mediation relies on implausible assumptions.

When interpreting a null finding, consider the sampling distribution

A noisy estimate that is indistinguishable from zero is often called a “null” finding, but its confidence interval may encompass quite large effects.

Do not interpret the failure to find an effect as a failure. A precisely estimated zero may have important theoretical implications and may help redirect scholarly attention toward more effective interventions.

When analyzing a cluster-randomized experiment, be sure to account for clustering when conducting estimation and inference.

When analyzing a cluster-randomized experiment, be sure to account for clustering when estimating standard errors and confidence intervals.

The power of a cluster-randomized experiment is more strongly determined by the number of clusters than the number of observations per cluster.

Clusters of unequal size may lead to bias in difference-of-means and regression estimation; if possible, block on cluster size

When analyzing experiments that encounter noncompliance, do not compare subjects who receive the treatment with subjects who do not receive the treatment. Never discard noncompliers.

Start with a model that defines the estimand and shows the conditions under which it can be identified.

Analysis should be grounded in a comparison between the randomly assigned treatment group and the randomly assigned control group, which have the same expected potential outcomes

Groups formed after random assignment, such as the “treated” or “untreated,” may not to have comparable potential outcomes.

Keep the estimand in mind when reporting and interpreting results.

ATE In your sample versus in some larger

population ITT CACE ATE among “Always-Reporters” Treatment effects in the presence of spillovers

of various kinds

Beware of interference and consider its empirical implications

Unless you specifically aim to study spillover, displacement, or contagion, design your study to minimize interference between subjects. Segregate your subjects temporally or spatially so that the assignment or treatment of one subject has no effect on another subject’s potential outcomes.

If you seek to estimate spillover effects, remember that you may need to use inverse probability weights to obtain consistent estimates

Show appropriate caution when interpreting interaction effects, or subgroup differences in average treatment effects

Adjust p-values for multiple comparisons so as to reduce the risk of attaching a substantive interpretation to what is in fact chance variation in average treatment effects.

When interpreting interactions between covariates and treatments, bear in mind that you are describing how treatment effects vary across different covariate values.

A factorial design is a more informative way to investigate whether changes in one variable cause changes in the influence of another factor.

Expect to perform a series of experiments.

In social science, one experiment is never enough.

Experimental findings should be viewed as provisional, pending replication, and even several replications and extensions only begin to suggest the full range of conditions under which a treatment effect may vary across subjects and contexts.

Every experiment raises new questions, and every experimental design can be improved in some way.