Field Experiments in Political Science:

A Brief History and Some Illustrative Examples

Don GreenColumbia University

What is an experiment? Units of analysis are randomly assigned with

known probability to treatment and control conditions

Distinction between random and haphazard assignment

Distinction between random assignment and random sampling

In the physical sciences, perfectly controlled experiments substitute for randomized trials

What is a field experiment? Random assignment takes place in a

naturalistic setting, enhancing generalizability Ideally, experimenters play an unobtrusive role,

reducing the risk of a violation of symmetry between treatment and control

Four dimensions of naturalism: subjects, treatments, contexts, outcome measures

Early (near random?) field experiments in Political Science Harold Gosnell: Mobilizing voters in the 1924

and 1925 elections George Hartmann: Using rational or emotional

appeals to increase the socialist vote in Allentown, PA in the 1935 elections

Underhill Moore and Charles Callahan: examined the effects of varying New Haven’s parking regulations, traffic controls, and police enforcement in 1930s

Early Field Experiments using Random Assignment Hovland, Lumdsdaine, and Sheffield: Propaganda

studies conducted for Experimental Section of the Research Division of the War Department

Samuel Eldersveld: Partisan and nonpartisan voter mobilization campaigns in the 1953 and 1954 Ann Arbor, MI elections

Ironic that randomized field experiments should die out just as they are gaining prominence in other disciplines (e.g., polio vaccine trials)

Dominant Modes of Behavioral Research in Political Science Surveys: spearheaded in the 1950s by

American National Election Studies using random sampling

Econometric analysis of aggregate data over time and/or space: growing computing power and technical facility of the discipline during 1970s

Fundamental trade off between research design and post hoc statistical correctives

Illusion of Observational Learning Theorem (Gerber, Green, and Kaplan 2004) When confronted with mixture of observational

and experimental evidence, Bayes’ Rule says assign zero weight to observational evidence unless you have informative priors about its bias

When confronted with laboratory experimental evidence, assign it zero weight unless you have informative priors about the biases associated with your extrapolation to some population/setting of interest

Implications of the “Illusion” for scientific practice Researchers using observational data are

oblivious to the fact that they routinely underreport the true degree of statistical uncertainty associated with their findings

Misallocation of the discipline’s research portfolio: No field experiments conducted between 1985 and 1998

Misconceptions about meta-analysis

Reclaiming the Experimental Tradition in Social Science Reassessment of evidence for a variety of basic

behavioral propositions by subjecting them to the same level of scrutiny as pharmaceutical evaluations

Secondary aim is to stimulate experimental reflection even in domains of political science where experimentation is infeasible

What follows is a brief overview of selected projects, some of which refute the “it can’t be done” critique

Project 1: Voter Mobilization(Authors: Gerber and Green) 1998 New Haven Study, N=31,098 Interventions: nonpartisan face-to-face

canvassing, commercial phone banks, direct mail Voter turnout measured using public records Subsequently replicated with hundreds of

thousands of observations in a variety of sites

Synthesis of recent randomized experiments on voter turnout: door-to-door canvassing, leafleting, phone calls, direct mail, and e-mail

Forest Plot of 85 direct

mail experiments – excluding

“social pressure“Studies

ATE = 0.109 ppts

(-0.07,0.290)

Lessons Learned

Face-to-Face canvassing raises turnout by approximately 7-9 percentage-points

Volunteer phone banks have moderate effects (3-5 percentage-points)

commercial phone banks are typically ineffective, as are robo-calls

Direct mail has weak effects (except as noted below)

E-mail has no apparent effect

Project 2: Habit Formation(Authors: Gerber, Green, and Shachar)

Infer habit from long-term effects of randomized intervention

Example of “downstream experimentation” Follow up study of people in the 1998 New

Haven Study showed that treatment group voted at higher rates

Subsequently replicated in 4 of 5 studies For each 100 additional votes generated in this

election, an additional 33 votes are generated in the next election

Project 3: Interpersonal Influence(Author: Nickerson)

Inference without problems of unobserved heterogeneity, which plague other influence studies

Placebo-control design: 2 voter households receive either get-out-the-vote message or a recycling appeal

Also, a control group gets nothing (as expected, turnout is significantly higher in GOTV vs. control)

Project 3: Interpersonal Influence(Author: Nickerson)

Findings show that housemates of registered voters who were contacted in the treatment group were significantly more likely to vote than housemates of those who were contacted in the placebo group

Example: Nickerson’s influence experiment in Denver 2002

05

101520253035404550

Vot

er T

urno

ut

Voters at the Door Housemates

Recycling (Placebo)GOTV (Treatment)

Project(s) 4: Influence of Television and Radio

Cable system experiments in 2003, 2004: influence of GOTV ads on turnout

Radio ads in 2005 and 2006 municipal elections: influence on competitiveness

Broadcast TV, cable TV, and radio in the context of a $2 million gubernatorial campaign

Influence of radio on ethnic reconciliation in Rwanda

Spanish language radio and voter turnout

Project 5: Social Pressure and Voter Turnout (Gerber, Green, and Larimer)

Using field experiments to test basic behavioral theories

Longstanding interest in “prescriptive” norms dating back to Gosnell’s work in the 1920s

To what extent can one manipulate the salience of “extrinsic” incentives associated with voting?

Study Design: August 2006

Sample: 180,002 households in Michigan, registered voters who voted in 2004

Setting: August primary election, “open” but contested only on the Republican side

Assignment: 10,000 clusters of 18 households each; in each cluster, households assigned at random to one of five groups: Control, Civic Duty, Hawthorne, Self, and Neighbors

Outcome: Voting in the primary election, as indicated by official records for each individual

For more information: (517) 351-1975email: etov@grebner.comPractical Political ConsultingP. O. Box 6249East Lansing, MI 48826

ECRLOT **C050THE WAYNE FAMILY9999 OAK STFLINT MI 48507

Dear Registered Voter:WHO VOTES IS PUBLIC INFORMATION!Why do so many people fail to vote? We've been talking about the problemfor years, but it only seems to get worse.This year, we're taking a different approach. We are reminding peoplethat who votes is a matter of public record.The chart shows your name from the list of registered voters, showingpast votes, as well as an empty box which we will fill in to show whetheryou vote in the August 8 primary election. We intend to mail you anupdated chart when we have that information.We will leave the box blank if you do not vote.DO YOUR CIVIC DUTY - VOTE!-----------------------------------------------------------OAK ST Aug 04 Nov 04 Aug 069999 ROBERT SMITH Voted ______9999 LAURA BETH Voted Voted ______

21

For more information: (517) 351-1975email: etov@grebner.comPractical Political ConsultingP. O. Box 6249East Lansing, MI 48826

ECRLOT **C050THE JACKSON FAMILY9999 MAPLE DRFLINT MI 48507

Dear Registered Voter:WHAT IF YOUR NEIGHBORS KNEW WHETHER YOU VOTED?Why do so many people fail to vote? We've been talking about the problem foryears, but it only seems to get worse. This year, we're taking a new approach.We're sending this mailing to you and your neighbors to publicize who does anddoes not vote.The chart shows the names of some of your neighbors, showing which have voted inthe past. After the August 8 election, we intend to mail an updated chart. Youand your neighbors will all know who voted and who did not.DO YOUR CIVIC DUTY - VOTE!-----------------------------------------------------------MAPLE DR Aug 04 Nov 04 Aug 069995 JOSEPH JAMES SMITH Voted Voted ______9995 JENNIFER KAY SMITH Voted

______9997 RICHARD B JACKSON Voted ______9999 KATHY MARIE JACKSON Voted

______9999 BRIAN JOSEPH JACKSON Voted ______9991 JENNIFER KAY THOMPSON Voted ______9991 BOB R THOMPSON Voted

______9993 BILL S SMITH

______9989 WILLIAM LUKE CASPER Voted ______9989 JENNIFER SUE CASPER Voted

______9987 MARIA S JOHNSON Voted Voted

______9987 TOM JACK JOHNSON Voted Voted

______

22

2006 August PrimaryElection

Experimental Group

Control Civic Duty Hawthorne Self Neighbors

Percent Voting

29.7% 31.5% 32.2% 34.5% 37.8%

N of Individuals

191,243 38,218 38,204 38,218 38,201

All contrasts with the control group are significant at p < .001, two-tailed test, using robust cluster standard errors (clustered at the household level).

Project 6: The Effects of Criminal Sentences on Recidivism (Green and Winik)

Random assignment of judges creates analytic leverage

~1000 defendants in Washington, DC drug courts randomly assigned to nine “calendars” with different sentencing proclivities

No effect of incarceration or length of sentence on recidivism

Miscellaneous Randomized Experiments of Interest in the Domain of Political Attiudes and Actions Randomly varying rules about legislative

seniority, term length, and floor recognition Discrimination experiments focusing on the

responsiveness of legislators to constituents of varying ethnic, racial, or partisan profile

Effects of audits and “accountability” interventions

Exposure to draft, school, or visa lotteries and their effects on attitudes and behavior

Bottom Line Randomized experimentation in real world

settings give social scientists access to the kinds of practical knowledge that outsiders care about

Possibility, of course, for premature extrapolation of experimental results: e.g., class size experiments

Importance of creating firm empirical foundation for theoretical development and policy intervention (e.g., prejudice reduction)

For a recent (2012) discussion of social science experiments and experimental design, see…