Chapter Four

Experimental & Quasi-experimental Designs

Experiment

• Tests cause & effect relationships

• Controls for other factors

• Measures on 2 or more groups before and after an intervention

• Measure change in a dependent variable with other things being equal

Internal validity

• Experimental treatment is having an impact on the outcome

• Other sources of influence have been controlled

External Validity

• Extent to which one can extrapolate findings from the particular study to other groups in general

Key Elements in Experimental DesignsKey Elements in Experimental Designs

•Measures taken at different times•Dependent Variable•Independent Variables

• Treatment Variable• Control Variable• Confounding Variables• Random Variables

Key Elements in Experimental Designs (Cont’d)

• 2 or more different treatment levels

• 2 x 2 design is the simplest multiple-variable design

• the first “2” represents the 2 levels of the treatment variable

• the second “2” the 2 levels of the control variable

• effect of University CD-ROM with Nature CD-ROM

Pre Experimental DesignsPre Experimental Designs

Same Group: Pre-Test/Post-Test

Time 1 Time 2 Pre-Test CD-ROM Post-Test

Threats to Internal ValidityThreats to Internal Validity

•History•Maturation•Testing•Instrument Decay•Statistical Regression

Statistical RegressionStatistical Regression

If cases are selected on the basis of extreme scores (either high or low) then when retests are done, there will be a tendency ofscores to move toward the mean

Reading scores example

Rule: Be cautious when evaluating programs where participantshave been selected on the basis of some extreme measure. Retesting will almost certainly lead to changes in the direction of the mean

Exposed/Comparison GroupExposed/Comparison Group

Treatment Group University CD Obs.

Comparison Group Nature CD Obs.

In this design there are measures taken at only one point in time. The problem is that the groups may not have been similar initially. The result may, or may not, be due to the University CD.

Classic Experimental Design

• Between-subjects design

• Also known as the pretest, posttest control group design

• Three elements of classical experiment:– manipulation of a treatment– control over extraneous variables– methods of dealing with pretreatment similarity

Between Subjects Design

.Treatment

Time 1

ControlTime 1

ControlTime 2

TreatmentTime 2

Nature CD-ROM

UniversityCD-ROM

Key to Experimental Design

Treatment and Control groups constructed to be as similar as possible before the experiment begins. This is done by:

– precision matching– randomization– a combination of the above two methods

What does Randomization Do?What does Randomization Do?

Provides control over both known and unknown factors (control and random variables)

% University Aspirations by% University Aspirations by CD-ROM Exposure CD-ROM Exposure

Group Time 1 Time 2 Diff.

Treatment 57.0 73.0 73-57=16

Control55.0 61.0 61-55= 6

Estimated impact of CD-ROM 10

Demonstrating a Causal RelationDemonstrating a Causal Relation

• Changes in treatment variable occur prior to changes in the dependent variable

• The treatment and dependent variables are associated: as the treatment variables goes up, the dependent varies systematically

• Nothing but the treatment variable has influenced the dependent variable

Ruling out Confounding EffectsRuling out Confounding Effects

• Context the same

• Background characteristics similar

• Neutralizing confounding (sources of spuriousness) variables

• Deal with Random Variables

Within-Subject DesignsWithin-Subject Designs

• Also known as repeated measures design

• Subjects are exposed to the various treatments

• Subjects’ own scores are compared

• Subjects act as their own control: perfect control in that it is the same person--no need to control for gender, I.Q., or SES!

Crossover Designs

• Both treatments are given to the same individuals rather than separately to two different samples

• Subjects serve as their own controls, hence, sample size required to detect an effect size is less

• Assign subjects randomly to a specific sequencing of treatment conditions

Hawthorne Effect

• This phenomenon refers to differences in the dependent variable that are not the direct result of changes in the treatment variable

Quasi-Experimental DesignsQuasi-Experimental Designs

• Approximation of experimental design: not possible to use random assignment or to control the nature or timing of the treatment

• Nonequivalent control group

• One-group pretest-posttest (pre-experimental)

• Time series design

• One-time case study

Panel Study TipsPanel Study Tips

• Name of relative or friend who stays put

• original phone number

• use phone directory from original time and present time one

• contact employer

• contact neighbors

• call people with same last name

Advantages & Disadvantages Advantages & Disadvantages of Experimental Designsof Experimental Designs

• Advantages: ease of making clear causal inferences (internal validity)

• Disadvantages: low on external validity, poor on probing, poor on dealing with many variables simultaneously