22
McGraw-Hill/Irwin © 2004 by The McGraw-Hill Companies, Inc. All rights reserved. Appendix B Conducting Experiments
McGraw-Hill/Irwin © 2004 by The McGraw-Hill Companies, Inc. All rights reserved.
Appendix B
Conducting Experiments
B-2
Experimentation and Causality
• Survey Research• Inferences are based on what respondents say about the topic
• Experimental Research• Inferences are based on what subjects do
regarding the treatment• Experiments manipulate conditions and
measure the effects on behavior
B-3
Types of Experiments
• Laboratory Experiments• Behavioral science laboratories• Test kitchens and tasting rooms• Special experimental facilities
• Field Experiments• Advertising and marketing
situations• Human resources and work settings• Public facilities and services
B-4
Pro’s and Con’s of Laboratory Experiments
The Con’s• Small number of subjects
demands precise sampling• Artificiality threatens
external validity• Special facilities or
equipment may be needed• Subjects must travel to the
laboratory or facility• Disruption of subjects’ daily
lives may create problems• Knowing they are being
tested may sensitize them
The Pro’s• Small number of subjects
permits precise sampling• Laboratories provides great
control• Subjects can be randomly
assigned• Instructing subjects is more
thorough• “Take” measures can be
obtained• Conditions permit more
elaborate designs
B-5
Experimental Terminology
• Experiment• A procedure for measuring the effect of some
treatment or condition on subjects
• Treatment or Treatment Condition• The experimental manipulation of some factor or
condition to determine its effect on subjects
• Control Group or Condition• Subjects manipulated identically to treatment groups
except for the experimental treatment
• Confound• Anything other than the experimental manipulation
that might cause a difference between the treatment and control group measurements
B-6
Types of Experimental Designs
• Pseudoexperiments• Designs too flawed to be regarded as legitimate
• Genuine Experiments• Designs that control threats to validity
• Extended Genuine Experiments• Designs that include multiple treatment groups
• Nonequivalent Groups Experiments• Designs lacking random assignment to groups
• Times Series Experiments• Designs that compare before and after effects
B-7
Threats to Internal Validity
• External Events Over Time
• Internal Processes Over Time
• Premeasurement Sensitization
• Measurement Instability
• Systematic Selection
• Experimental Attrition
• Regression to the Mean
B-8
Pseudoexperimental Designs
• Postmeasure Only, Experimental Group Only
Group AGroup A
TreatmentTreatment
Postmeasure A2Postmeasure A2
Treatment GroupTim
e L
ine
B-9
Pseudoexperimental Designs
• Pre- and Postmeasurement, Experimental Group Only
Group AGroup A
Postmeasure A2Postmeasure A2
Treatment GroupTim
e L
ine
Premeasure A1Premeasure A1
TreatmentTreatment
B-10
Genuine Experimental Designs
• Pre- and Postmeasure, Experimental and Control Group
Group AGroup A Group BGroup B
RandomRandom RandomRandom
Premeasure A1Premeasure A1 Premeasure B1Premeasure B1
TreatmentTreatment ControlControl
Postmeasure A2Postmeasure A2 Postmeasure B2Postmeasure B2
Treatment Group Control Group
Tim
e L
ine
B-11
Genuine Experimental Designs
• Postmeasure Only, Experimental and Control Group
Group AGroup A Group BGroup B
RandomRandom RandomRandom
TreatmentTreatment ControlControl
Postmeasure A2Postmeasure A2 Postmeasure B2Postmeasure B2
Treatment Group Control Group
Tim
e L
ine
B-12
Genuine Experimental Designs
• Four-Group, Six-Measure Design
Tim
e L
ine
Group AGroup A Group BGroup B Group CGroup C Group DGroup D
RandomRandom RandomRandom RandomRandom RandomRandom
Premeasure A1Premeasure A1 Premeasure B1Premeasure B1 No MeasureNo Measure No MeasureNo Measure
TreatmentTreatment ControlControl TreatmentTreatment ControlControl
Postmeasure A2Postmeasure A2 Postmeasure B2Postmeasure B2 Postmeasure C2Postmeasure C2 Postmeasure D2Postmeasure D2
Treatment Group Control Group Treatment Group Control Group
Pre- and Postmeasure Postmeasure Only
B-13
Extended Genuine Experimental Designs
• Pre- and Postmeasure, Extended Design
Tim
e L
ine
Group AGroup A Group BGroup B Group CGroup C Group DGroup D
RandomRandom RandomRandom RandomRandom RandomRandom
Premeasure A1Premeasure A1 Premeasure B1Premeasure B1 Premeasure C1Premeasure C1 Premeasure D1Premeasure D1
Treatment ATreatment A Treatment BTreatment B Treatment CTreatment C Treatment DTreatment D
Postmeasure A2Postmeasure A2 Postmeasure B2Postmeasure B2 Postmeasure C2Postmeasure C2 Postmeasure D2Postmeasure D2
Treatment Group Treatment Group Treatment Group Treatment Group
B-14
Extended Genuine Experimental Designs
• Postmeasure Only, Extended Design
Tim
e L
ine
Group AGroup A Group BGroup B Group CGroup C Group DGroup D
RandomRandom RandomRandom RandomRandom RandomRandom
Treatment ATreatment A Treatment BTreatment B Treatment CTreatment C Treatment DTreatment D
Postmeasure A2Postmeasure A2 Postmeasure B2Postmeasure B2 Postmeasure C2Postmeasure C2 Postmeasure D2Postmeasure D2
Treatment Group Treatment Group Treatment Group Treatment Group
B-15
Nonequivalent Groups Experimental Designs
• Pre- and Postmeasure, Matched Groups
Group AGroup A Group BGroup B
MatchedMatched MatchedMatched
Premeasure A1Premeasure A1 Premeasure B1Premeasure B1
TreatmentTreatment ControlControl
Postmeasure A2Postmeasure A2 Postmeasure B2Postmeasure B2
Treatment Group Control Group
Tim
e L
ine
B-16
Nonequivalent Groups Experimental Designs
• Postmeasure Only, Matched Groups
Group AGroup A Group BGroup B
MatchedMatched MatchedMatched
TreatmentTreatment ControlControl
Postmeasure A2Postmeasure A2 Postmeasure B2Postmeasure B2
Treatment Group Control Group
Tim
e L
ine
B-17
Time Series Experimental Designs
• Single Group, Pre- and Postmeasurement Series
Tim
e L
ine
Group AGroup A
Measurement A1Measurement A1
Measurement A2Measurement A2
Measurement A3Measurement A3
Measurement A4Measurement A4
Treatment ATreatment A
Measurement A5Measurement A5
Measurement A6Measurement A6
Measurement A7Measurement A7
Measurement A8Measurement A8
Pre
measu
reS
eri
es
Postm
easu
reS
eri
es
B-18
Time Series Experimental Designs
• Two Groups, Pre- and Postmeasurement Series
Tim
e L
ine
Group AGroup A
Pre
measu
reS
eri
es
Postm
easu
reS
eri
es
Group BGroup B
Measurement A1Measurement A1 Measurement B1Measurement B1
Measurement A2Measurement A2 Measurement B2Measurement B2
Measurement A3Measurement A3 Measurement B3Measurement B3
Measurement A4Measurement A4 Measurement B4Measurement B4
Treatment ATreatment A Treatment BTreatment B
Measurement A5Measurement A5 Measurement B5Measurement B5
Measurement A6Measurement A6 Measurement B6Measurement B6
Measurement A7Measurement A7 Measurement B7Measurement B7
Measurement A8Measurement A8 Measurement B8Measurement B8
B-19
Threats to External Validity
• External Validity• The degree to which results can be generalized to
the population
• Artificiality• The manner in which the experimental setting
differs from the subjects’ daily environment
• Reactivity• The tendency for the experimental situation to
effect subjects’ responses to treatments
B-20
Blind Testing in Experiments
• Blind Testing• The procedure does not reveal to the subjects what
is being measured or tested
• Double-Blind Testing• Neither the subjects nor those running the
experiment are told what is being measured
• The Ultimate Blind Test• A field experiment where subjects are not aware
they are participants in an experiment• Test marketing of new products• “Split run” direct response advertising• Selective modification of public facilities
B-21
Experimentation/Survey Commonalities
• Project Initiation• Definition of information needs• Project planning procedures• Sampling design
• Measurement Instrumentation• Questionnaires, instructions, and scales• Observation and recording
• Analysis and Reporting• Data editing and processing• Statistical analysis and interpretation• Report composition and presentations
McGraw-Hill/Irwin © 2004 by The McGraw-Hill Companies, Inc. All rights reserved.
End of Appendix B
