+ All Categories
Home > Documents > EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring...

EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring...

Date post: 28-Mar-2015
Category:
Upload: declan-dayton
View: 213 times
Download: 0 times
Share this document with a friend
Popular Tags:
34
EVAL 6970: Experimental and Quasi-Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012
Transcript
Page 1: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

EVAL 6970:Experimental and Quasi-

Experimental DesignsDr. Chris L. S. Coryn

Dr. Anne CullenSpring 2012

Page 2: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Agenda

• Basic design elements and notation• Quasi-experimental designs that

either lack a control group or lack pretest observations on the outcome

• Midterm examination• Case study

Page 3: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Questions to Consider

• What are the limitations of designs lacking either control groups and/or pretest observations?

• What simple strategies can be used to improve these types of designs?

• Why are such designs sometimes the only ones that can be used?

Page 4: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Basic Design Elements and Notation

Page 5: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Assignment

• Random assignment• Cutoff-based assignment• Other nonrandom assignment• Matching and stratifying• Masking

Page 6: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Measurement

• Posttest observations– Single posttests– Nonequivalent dependent variables– Multiple substantive posttests

• Pretest observations– Single pretest– Retrospective pretest– Proxy pretest– Repeated pretests over time– Pretests on independent samples

• Moderator variable with predicted interaction• Measuring threats to validity

Page 7: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Comparison Groups

• Single nonequivalent groups• Multiple nonequivalent groups• Cohorts• Internal versus external controls• Constructed contrasts– Regression extrapolation contrasts– Normed contrasts– Secondary data contrasts

Page 8: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Treatments

• Switching replications• Reversed treatments• Removed treatments• Repeated treatments

Page 9: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Notation

X = treatmentO = observationR = random assignmentNR = nonrandom assignmentX = removed treatmentX+ = treatment expected to produce an

effect in one directionX- = conceptually opposite treatment expected to reverse an effectC = cutting score- - - = non-randomly formed groups… = cohort

Page 10: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Logic of Quasi-Experimentation

Page 11: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Rationale

• Quasi-experiments are often a necessity given practical and logistical constraints– Greater emphasis on construct or external validity

rather than cause-effect associations (least common)– Funding, ethics, administration (somewhat common)– The intervention has already occurred (most common)

• Sometimes they are the best alternative, even if causal inferences are weaker than is possible with other designs

• Even so, great care must be taken when planning such studies as numerous threats that cannot be controlled are often operating

Page 12: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Central Principles

• Identification and study of plausible threats to internal validity– Careful scrutiny of plausible alternative

explanations for treatment-outcome covariation

• Primacy of control by design– Use carefully planned and implemented design

elements rather than statistical controls for anticipated confounds

• Coherent pattern matching– Complex (a priori) causal hypotheses that

reduce the plausibility of alternative explanations

Page 13: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Designs without Control Groups

Page 14: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

One-Group Posttest Only Design

• Absence of pretest makes it difficult to know if change has occurred and absence of a control group makes it difficult to know what would have happened without treatment

X O1

Page 15: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

One-Group Pretest-Posttest Design

• Adding a pretest provides weak information concerning what might have happened to participants had the treatment not occurred

O1 X O2

Page 16: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

One-Group Pretest-Posttest Design with Double Pretest

• Adding multiple pretests reduces the plausibility of maturation and regression effects

• Additional pretests can confirm maturational trends

O1 O2 X O3

Page 17: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

One-Group Pretest-Posttest Design Using a Nonequivalent Variable

• Measure A is expected to change because of treatment, B is not

• Both A and B are expected to respond to the same validity threats in the same way

{O1A , O1B} X {O2A , O2B}

Page 18: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

• Lottery ticket sales in convenience stores after introduction of signs in store windows reading “did you buy your ticket?”

A = sale of lottery ticketsB = sale of alcoholC = sale of tobacco

A

A

B

B

CC

Page 19: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Removed-Treatment Design

• Demonstrates that outcomes rise and fall with the presence or absence of treatment

O1 X O2 O3 X O4

Page 20: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

• Generally interpretable outcome pattern

O1

Outc

om

e

Wors

eB

ett

er

O2 O3 O4X X

Uninterpretableoutcome

Interpretableoutcome

Page 21: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Repeated-Treatment Design

• Few threats could explain a close relationship between treatment introductions and removals and parallel outcome changes

O1 X O2 X O3 X O4

Page 22: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

• Mean narcotics use over multiple Methadone maintenance on/off conditions

off MM on MM #1

off MM on MM #2

off MM on MM #3

off MM on MM #4

off MM on MM #5

0%

10%

20%

30%

40%

50%

60%

70%

Page 23: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

A-B Designs

• Multiple-baseline design (a class of single-subject designs), or collection of A-B designs, to assess the effects of an intervention across separate baselinesA = baselineB = treatment

• The intervention is introduced in a staggered manner and the baseline provides a predicted level of the dependent variable in absence of the treatment

• A-B-A designs are sometimes called removal designs (i.e., the treatment is removed)

Page 24: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Nu

mb

er

of

Accid

en

ts

Weeks

Baseline

Baseline

Baseline

Treatment

Treatment

Treatment

Sit

e 1

Sit

e 2

Sit

e 3

Effect

Effect

Effect

Page 25: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Designs that use a Control Group but no Pretest

Page 26: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Posttest-Only Design with Nonequivalent Control Group

• Unknown pretest group differences make it extremely difficult to separate treatment effects from selection effects

NR X O1

NR O1

Page 27: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Posttest-Only Design using an Independent Sample Pretest

• Assumes overlapping group membership• Useful when pretest measurements may

be reactive, cannot follow same groups over time, or when interested in studying intact communities whose members change over time

NR O1 X O2

NR O1 O2

Page 28: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Posttest-Only Design using Proxy Pretest

• Proxy measures should be conceptually related to and correlated with outcome

• Can be used for a variety of purposes including indexing selection bias and/or attrition

NR OA1 X OB2

NR OA1 OB2

Page 29: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Case Control Studies

• Predominant method for many forms of epidemiological research

• Used to identify factors that may contribute to a condition by comparing subjects who have that condition (i.e., 'cases') with those who do not have the condition but are otherwise similar (i.e., 'controls')

• Famously, the association between smoking and lung cancer

• Similar in many respects to Scriven’s GEM and MOM

Page 30: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Midterm Examination

Page 31: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Midterm Examination

• The examination will consist of 50-75 multiple-choice items, scored as 0 or 1

• You will have 2½ hours to complete the examination

• You may use one page of notes (front and back) on 8½” X 11’’ paper– You will be asked questions about

statistical power, but will not be required to calculate power

Page 32: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Case Study

Page 33: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Case Study Activity

• An aid agency implemented a project in Bangladesh with the objective of improving the nutritional and health status of men and women

• The intervention consisted of a package of services including: nutrition education, primary health care, and other activities

• To determine whether the intervention might be effective, the project was field-tested in a small rural community prior to large-scale implementation throughout the country

• A small monetary incentive was provided and slightly more than half of the community’s men and women participated in the study

• All men and women in the community were weighed prior to the intervention and then were measured for body mass index (BMI) six months after the intervention

• Those who did not participate were used as a control group and the evaluators found significant improvements in nutritional and health indicators for the treatment group contrasted with the control

Page 34: EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Questions

• What is the design of the study?• What internal validity threats are

most plausible?• How might the design feasibly be

improved?


Recommended