Selecting a Research Design

Research Design

• Refers to the outline, plan, or strategy specifying the procedure to be used in answering research questions

• Determines the when (a procedural sequence) but not the how of:

1. Sampling Techniques and

representativeness of data sources

2. Data Collection• Time frame of measurement• Methods of measurement

3. Data Analysis

Three Major Approaches to Research

Designs

1. Descriptive Approach2. Experimental Approach3. Quasi-Experimental Approach

1. Descriptive Approach

• Represent or provide accurate characterization of phenomenon under investigation

• Provide a “picture” of a phenomenon as it naturally occurs

Key Features:

• Not designed to provide information on cause-effect relationships, therefore internal validity is not a concern

• Because focus is on providing information regarding some population or phenomena, external validity is major concern

Variations:

• Exploratory

• Goal: to generate ideas in field of inquiry that is relatively unknown

• Least structured to gather more descriptive information

• Frequently used as first in series of studies on a specific topic

•Process evaluation• Goal: to identify the extent to which a program

(or policy) has been implemented, how that has occurred, and what barriers have emerged

• Program as implemented vs. program as intended• Designed to guide development of new program

(normative), summarize the structure of program prior to studying its effects, or assess feasance of pre-existing program

•Strengths (Descriptive):

• Generally lower costs (depend upon sample size, number of data sources, and complexity of data collection methods)

• Relative ease of implementation• Ability to yield results in relatively short

amount of time• Data analysis straight-forward• Results easy to communicate to non-technical

population

•Limitations (Descriptive):

• Cannot answer questions of causal nature

2. Experimental Approach

• Primary purpose is to empirically test the existence of causal relationship among two or more variables

• Systematic variation of independent variable (IV) and measure its effects on dependent variable (DV)

Kerlinger (1973) MAX-MIN-CONApproach

• MAXimize systematic variance (exp cond as different as possible)

• MINimize error variance (accuracy of assessment)

• CONtrol extraneous systematic variance (homogeneity of conditions)

Key Features:

• Random assignment of individuals or entities to the levels or conditions of the study• Control biases at time of assignment• Ensure only independent variable(s)

differs between conditions

Key Features (cont):

• Emphasis placed on maximizing internal validity by controlling possibly confounding variables

• Creation of highly controlled conditions may reduce external validity

Variations:

1 .Between Group Designs

A. Post-only design•Subjects randomly assigned to

experimental\control groups •Introduction of IV in experimental

condition•Measurement of DV (single or

multiple instances)

Group 1RXOGroup 2RO

Randomized IV DV

Post-only design

B. Pre and post design:

•Subjects randomly assigned to experimental\control groups

•Preliminary measurement of DV before before treatment (check of random assgn)

•Introduction of IV in experimental condition

•Measurement of DV (single or multiple instances)

Group 1 R O1 XO2Group 2 R O1O2

Pre- and Post- design

Randomized IV DVDV

C. Multiple Levels of single IV:

•Subjects randomly assigned to experimental\control groups

•Introduction of multiple levels of IV in experimental condition

•Measurement of DV across different conditions

Group 1RX1OGroup 2RX2OGroup 3RX3OGroup 4RO

Multiple Levels design

Randomized IV DV

D. Multiple Experimental and Control Groups (Solomon Four-Group design):

•Subjects randomly assigned to experimental\control groups

•Preliminary measurement of DV in one exp\control pair

•Introduction of IV in both experimental conditions

•Measurement of DV (assess effects of pretest)

Group 1 RO1XO2Group 2 RO1O2

Group 3 RXO2Group 4 RO2

Multiple Levels design

Randomized IVDV DV

E. Multiple IVs (Factorial Design):

•Subjects randomly assigned to experimental\control groups

•Introduction of multiple levels of IVs in experimental condition

•Measurement of DV across different conditions (cells)

Group 1RX1Y1OGroup 2RX2Y2OGroup 3RX1Y2OGroup 4RX2Y1O

Multiple IVs design

Randomized IV DVIV

2 .Within-Group Designs

•Repeated Measures

•Each Subject is presented with two or more experimental conditions

•Comparisons are made between conditions within the same group of subjects

Within SS design

Subject 1 RX1O1X2O2Subject 2 RX1O1X2O2Subject 3 RX1O1X2O2Subject 4 RX1O1X2O2

Randomized IV DVIV DV

•Strengths (Expl):

•Experimental control over threats to internal validity

•Ability to rule out possible alternative explanations of effects

•Limitations (Expl):

•More resemble controlled study, less resembles usual real-world intervention (decrease external validity)

•Experimental Realism = engagement of subject in experimental situation

•Mundane Realism = correspondence of experimental situation to ‘real-world’ or common experience

•Randomized experiments difficult to implement with integrity(practical or ethical reasons)

•Attrition over time = non-equivalent designs

•Limitations (Expl):

3 .Quasi-Experimental Approach

•Primary purpose is to empirically test the existence of causal relationship among two or more variables

•Employed when random assignment and experimental control over IV is impossible or impractical

•Key Features:

•Other design features are substituted for randomization process

•Quasi-experimental comparison base:•Addition of non-equivalent

comparison groups

•Addition of pre- and post-treatment observations

•Variations:A. Non-equivalent Comparison Group

•Post Only, Pre-Post, Multiple Treatments

•No random assignment into exp\con

•“Create” comparison groups

•Selection criteria or eligibility protocol

•Partial out confounding variance •( statistical control)

B. Interrupted Time Series

•Multiple observations before and after treatment or intervention is introduced

•Examine changes in data trends (slope and intercept)

•Investigate effects of both onset and offset of interventions

0

2

4

6

8

10

12

14

16

Intervention

Interrupted Time Series

C. Regression Discontinuity

•Separate sample based on some criterion (pre-test)

•One group administered treatment, other is control group

•Examine trends in both groups; hypotheisze equivalent

Pre-test Cut-off Point

TreatmentNo Treatment

PostTestScores

Regression Discontinuity

•Strengths (Quasi):

•Approximation of experimental design, thereby allowing causal inference

•Garner internal validity through statisticalcontrol, not experimental

•Use where experimental designs are impractical or unethical

•Limitations (Quasi):

•Uncertainty about comparison base:Is it biased?

•Statistical control based on known factors. If unknown or unmeasureable, threat to validity

•Data collection schedule and measures very important