Dana BurdeVisiting Assistant Professor

Department of Humanities and Social Sciences

New York Universitydana.burde@nyu.edu

Panacea or Straitjacket? A Qualitative Researcher Reflects

on a Randomized Trial in Afghanistan

Randomized Trial Assessing Community-Based Schools in Afghanistan

Co-Investigator (for the large-scale randomized trial): Leigh Linden, Assistant Professor, Economics Department and School of International and Public Affairs, Columbia University

Agency studied: Catholic Relief ServicesResearch assistants: Matt Hoover, Dr. Saeed Mahmoodi, Nathan Falkner, Nafi Olomi, Nicole Rigg, Amy Kapit.

Funders (from the inception of the study): Columbia University Institute for Social and Economic Research & Policy ($15,000); Weikart Family Foundation ($75,000); United States Institute of Peace ($40,000); Spencer Foundation ($40,000—pilot; $483,000—large scale study); National Science Foundation ($100,000).

Outline of today’s talk

1. Brief overview of arguments: Heated pro/con debate between qualitative researchers and quantitative researchers regarding the uses and effectiveness of randomized trials

2. Study design: Description of the randomized trial of community-based schools in Afghanistan and the ways in which qualitative study components complemented the quantitative design

3. Discussion: Design, execution, ethics, and research culture—how these issues affected our randomized trial

The Pro/Con Debate

Those in favor of randomized trials say they provide: Evidence-based findings that show more robust

results (than other kinds of studies) Answers to key questions in education research

This position is epitomized by this quotation:“…[randomized controlled] experiments provide a

better warrant [for causal conclusions]…than any other method. So if experiments can be conducted in schools they should be. Not to use them requires a very strong justification” (Cook, 2003, p. 117).

The Pro/Con Debate

Those who oppose randomized trials accuse them of: Weakness in the research design and execution of the

study; Ethical shortcomings; and Promoting a culture of privileging quantitative methods

over qualitative.

This position is epitomized by this quotation:“…there are formidable limits to the randomized control approach—limits

which the proponents of this methodology have not sufficiently recognized. To achieve ‘rigor’ in education research, we need to adopt a more realistic grasp of the problems that plague attempts to implement research results as well as a greater openness to other approaches that offer crucial insights into social processes in organizations. For these purposes, qualitative methods, including participant observation and in-depth interviews, are likely to be the most promising” (Lareau, forthcoming, p. 146).

Argument

Based on our study in Afghanistan, I argue that: Randomized trials are excellent tools for assessing program

impact and are underused to study education programs conducted by international NGOs

Strong design and execution of place-based randomized trials can overcome some weaknesses identified by critics

Questions raised regarding ethics can be addressed and should not preclude using randomized trials

In agreement with the critics, I argue randomized trials should not eclipse other forms of research; they are not appropriate for every impact study; qualitative methods can address their weaknesses and complement randomized trials to great effect

Overview of StudyThe study: investigates whether community-based education programs

improve child welfare by comparing villages that receive schools to villages that have not yet received them and by comparing children in these villages.

examines attendance/enrollment, academic achievement, child labor, and social benefits (protection, social networks).

Phases: Qualitative pilot study: Tested field procedures 2005-2006 Randomized trial 2007-2008: Worked with a large international

US-based NGO, Catholic Relief Services (CRS), to randomly assign their community-based schools to eligible villages (i.e., villages that had none previously)

Complementary qualitative case study of program

Program: Partnership for Advancing Community Education in Afghanistan (PACE-A)

PACE-A is a five-year, $24 million USAID-funded program: meant to “expand quality learning and life opportunities for

marginalized communities and their children in Afghanistan” (PACE Summary, 2006, p. 1)

Consortium of 4 NGOs tasked with providing hundreds of community-based schools in 20 provinces across Afghanistan

The CRS portion of the program was intended to reach approximately 100 villages by the end of 2008 Schools were to be phased in over several years

CRS Community-Based Schools

CRS selects districts and communities according to: Security; Availability of teachers; Level of community

interest; Community willingness to mobilize resources (teachers’ salaries); Support from the Ministry of Education (agreement); Community willingness to provide space for the school (room in a home or mosque).

CRS provides training to teachers, materials for the classroom (government curriculum), and regular monitoring to track progress over time.

Study Hypotheses Hypothesis 1 Attendance/Enrollment: Children living in a

village that receives a CRS school will be more likely to enroll in school and will attend more regularly than children in other villages.

Hypothesis 2 Learning: Children who live in villages that receive a CRS school will perform better on academic achievement tests on math and language skills than children in the villages that do not receive a CRS school.

Hypothesis 3 Indirect Social Outcomes: Children who live in villages that receive a CRS school will report higher levels of social integration, higher levels of security, and higher levels of safety than children in other villages.

Original Study Design Yes - CBS in 2007 No - CBS in 2007

(but yes for 2008)

Group 1:

Normal CRS Class,

10 Villages, 500 HHs Group 5

No CRS Schools,

40 Villages,

2,000 HHs

Group 2:

Community education,

10 Villages, 500 HHs

Group 3:

Disciplinary Instruction,

10 Villages, 500 HHs

Group 4:

Community Education and Disciplinary Instruction,

10 Villages, 500 HHs

Actual Study Design

CRS community-based schools

N = 31 villages; N = 1,253 households;

N = 2,036 children aged 6-11

Yes in 2007 No in 2007

Group 1: Treatment

CRS Schools,

13 Villages,

637 HHs

Group 2: Control

No CRS Schools,

18 Villages,

616 HHs

Qualitative component

Case study of the CRS program to complement the RT and learn more about:

program implementation; place, i.e., country, province, districts, and villages in

which we are collecting data; institutions that are part of our study or influence it: i.e.,

households, families, schools (government and CBS), mosques, SMCs, village shuras, ministry of education.

individuals that are part of our study or influence it, i.e., children, parents/heads of households, teachers, arbobs, CRS education staff members, CRS managers, PACE-A directors

Map of Afghanistan

Ghor Province, Afghanistan

Ghor Province

Discussion: Design Criticism: Addressing only narrow questions misses significant

variables in education

Our study: “Schools” constitute a significant variable

Criticism: Extent to which variables can be manipulated accurately, e.g., networks

Our study: Intent was not to manipulate networks, but rather to measure the impact of schools on networks, and other secondary effects

Limitation: We can say that the program did not have an impact on friendship networks, but need qualitative methods to say why

Discussion: Execution Criticism: Adherence to treatment; “contamination” of

control by treatment

Difficulties our study faced in execution:• Persuading CRS to carry out a randomized trial

• Avoiding contamination • Preserving the integrity of the randomization:

• Communication breakdowns (sample size decreased)• Inclement weather (delayed start to survey)• Security (attacks on staff, tribal warfare, etc. sample size

decreased to 31 villages)• Hiring and training survey staff

Discussion: Ethics

Criticism: Unethical to deny treatment

Ethical issues our study faced: Sample selection: “Needs assessment” versus

randomized selection Temporary denial of treatment: Program was

intended to be phased in over time, regardless Program benefits not clear given the basic program

intervention

Discussion: Ethics

Additional ethical issues our study faced: Protection for the researchers was a far greater

problem than the protection of subjects IRB was silent on this point In fact, IRB protections for subjects put researchers

at risk: Confidentiality Signed documents

Discussion: Research CultureA culture of preference for randomized trials pervades among some

research circles.

Qualitative methods are invaluable in addressing weaknesses of randomized trials in several ways:

A strong qualitative pilot study enhances the randomized study design and increases the probability of successful program intervention as well as successful execution of the study.

Qualitative methods used in conjunction with a randomized trial counteract the tendency of randomized trials to focus on narrow cause and effect relationships.

Complementary qualitative case study and ethnographic methods provide more confidence in quantitative data analysis, interpretation, and provide additional detail to explain findings that randomized trials cannot.

Conclusion

Randomized trials -- crucial method to test multiple empirical questions across the social sciences.

Critics have valid concerns -- randomized trials should not be allowed to eclipse qualitative methods.

As in all research, ethical questions raised by randomized trials must be taken seriously.

Properly designed and executed, randomized trials can produce robust and significant data even in the most difficult circumstances.