Aleem Ashraf

Department of Psychology

University of Sindh, Jamshoro.

Experimental Design

• Assignment of subjects to different groups or conditions

• Manipulation of one or more independent variables by experimenter (IV)

• Measurement of effects on one or more dependent variables (DV)

• Control of other variables

• In order to make causal judgments about what causes variation in dependent variable

Ale

em

Ashra

f

2

Manipulation

• Applying certain treatments (levels of independent

variables) to the different conditions

Control

• The researcher’s attempts to make the groups as

similar as possible on most variables other than

independent variable.

• Direct control e.g. focus on certain characteristics

by exclusion

• Age, intelligence, socioeconomic class, previous

experience, good school system etc.

• Most variables cannot be controlled in the direct

fashion, for that randomization is used

Ale

em

Ashra

f

3

Control by Randomization

• Randomized selection of participants from

known population. Required but not often

done in the experiments. Only the willing

volunteers are selected

• Done to give external validity =

generalizability

• Randomized allocation of recruited

participants to different conditions (control or

experimental group)

• Done to give internal validity = the effects in

the DV are really due to IV?

Ale

em

Ashra

f

4

Randomization

• Each subject in the study has an equal

chance of being assigned to the control or

experimental group

• It evenly distributes the important

variables across groups that may effect

the dependent variable

Research Process

R

Random

Selection

R

Random

Assignment

O1

Observation

1

X

Treatment

O2

Observation

2

Ale

em

Ashra

f

5

The Experimental Process

• Flow chart of a classic randomized clinical

trial

Sample is

selected from

the population

Baseline data

are collected

Subjects are

randomized

Experimental

group

Control group

Post test data

are collected

Post test data

are collected

Ale

em

Ashra

f

6

Example

• Does noise effect the amount that people

can remember when learning?

• Participants are selected and randomly

assigned to three conditions:

• Control group = no noise at all

• Treatment group 1 = loud unpredictable

noise

• Treatment group 2 = soft, rhythmic music

• Respondents are given a standard list of

items to remember in a given time

Ale

em

Ashra

f

7

Illustration of the Design

Control group

No noise at all

Treatment group 1

Loud unpredictable

noise

Treatment group 2

Soft, rhythmic music

Ale

em

Ashra

f

8

Procedure

• Randomly allocate participants to the three

groups

• All groups perform the same task (memorize

the list) under different conditions i.e. no noise,

loud noise, soft music

• All participants are given the same test after

the treatment

• Compare the mean score of recall for each

group

• Use statistical tests i.e. t-test or ANOVA to see

if the differences in the mean are significant

Ale

em

Ashra

f

9

Graphical representation of the three groups

0

2

4

6

8

10

12

14

16

18

No noise Loud noise Soft music

Ale

em

Ashra

f

10

Internal Validity

• Internal validity is the degree to which

differences in performance on a

dependent variable can be attributed

clearly and unambiguously to an effect of

an independent variable, as opposed to

some other uncontrolled variable.

Ale

em

Ashra

f

11

Threats to Internal & External Validity

• Is the researcher’s claim about the cause

and effect correct?

• Are the changes in the independent

variable indeed responsible for the

observed variation in the dependent

variable?

• Might the variation in the dependent

variable be attributable to other causes?

Ale

em

Ashra

f

12

Importance of Internal Validity

• We often conduct research in order to determine

cause-and-effect relationships.

• Can we conclude that changes in the independent

variable caused the observed changes in the

dependent variable?

• Is the evidence for such a conclusion good or

poor?

• If a study shows a high degree of internal validity

then we can conclude we have strong evidence of

causality.

• If a study has low internal validity, then we must

conclude we have little or no evidence of causality.

Ale

em

Ashra

f

13

Variables & Internal Validity

• Extraneous variables are variables that may

compete with the independent variable in

explaining the outcome of a study.

• A confounding variable is an extraneous

variable that does indeed influence the dependent

variable.

• A confounding variable systematically varies or

influences the independent variable and also

influences the dependent variable.

• Researchers must always worry about extraneous

variables when they make conclusions about

cause and effect.

Ale

em

Ashra

f

14

Three Criteria of Causality

• Association: The cause and effect must

be associated with each other (Correlation)

• Direction of influence: The effect appears

after the cause. (The effect must respond

to the cause, not vice versa).

• Nonspuriousness: There should be good

reason to believe that there are no hidden

factors that could have created an

accidental relationship between the

variables.

Ale

em

Ashra

f

15

Threats to Internal Validity

1. History

2. Maturation

3. Testing

4. Instrumentation

5. Statistical regression

6. Differential selection of participants

7. Mortality

8. Design contamination

9. Compensatory rivalry

10.Resentful demoralization

Ale

em

Ashra

f

16

History

• Unexpected events occur between the

pre- and posttest, affecting the dependent

variable.

• Example: In a short experiment designed

to investigate the effect of computer-

based instruction, a participant missed

some instruction because of a power

failure at the school.

Ale

em

Ashra

f

17

Maturation

• Changes occur in the participants, from

growing older, wiser, more experienced

etc. during the study.

• Example: The performance of first

graders in a learning experiment begins

decreasing after 45 minutes because of

fatigue.

Ale

em

Ashra

f

18

Testing

• Taking a pretest alters the result of the posttest.

• A pre-test may sensitize participant in unanticipated ways and their performance on the post-test may be due to the pre-test, not to the treatment.

• Example: In an experiment in which performance on a logical reasoning test is the dependent variable, a pre-test cues the subjects about the post-test.

Ale

em

Ashra

f

19

Instrumentation

• Did any change occur during the study in

the way the dependent variable was

measured?

• Example: Two examiners for an

instructional experiment administered the

post-test with different instructions and

procedures.

Ale

em

Ashra

f

20

Statistical Regression

• Extremely high or extremely low scorers

tend to regress to the mean on retesting.

• Example: In an experiment involving

reading instruction, subjects grouped

because of poor pre-test reading scores

show considerably greater gain than do

the groups who scored average and high

on the pre-test.

Ale

em

Ashra

f

21

Differential Selection of Participants

• Participants in the experimental and

control groups have different

characteristics that affect the dependent

variable differently.

• Random assignment of participants into

different groups usually resolve this

threat.

Ale

em

Ashra

f

22

Mortality

• Different participants drop out of the study

in different numbers, altering the

composition of the treatment groups.

• Example: In a health experiment

designed to determine the effect of

various exercises, those subjects who

find the exercise most difficult stop

participating.

Ale

em

Ashra

f

23

Design Contamination

• Did the comparison group know (or find

out) about the experimental group? Did

either group have a reason to want to

make the research succeed or fail? Often,

investigators must interview subjects after

the experiment concludes in order to find

out if design contamination occurred.

• Example: In an expectancy experiment,

students in the experimental and

comparison groups “compare notes” about

what they were told to expect.

Ale

em

Ashra

f

24

Compensatory Rivalry

• When subjects in some treatments receive goods or services believed to be desirable and

this becomes known to subjects in other groups,

social competition may motivate the latter to attempt to reverse or reduce the anticipated

effects of the desirable treatment levels.

• Known as “John Henry” effect in honor of the steel driver who, upon learning that his output

was being compared with that of a steam drill, worked so hard that he outperformed the drill

and died of overexertion.

Ale

em

Ashra

f

25

Resentful Demoralization

• If subjects learn that their group receives

less desirable goods or services, they

may experience feelings of resentment

and demoralization.

• Their response may be to perform at an

abnormally low level, thereby increasing

the magnitude of the difference between

their performance and that of groups that

receive the desirable goods or services.

Ale

em

Ashra

f

26

External Validity

• External validity refers to the extent to

which findings from a research study can

be generalized to individuals, settings,

and conditions beyond the scope of the

specific study.

Ale

em

Ashra

f

27

Threats to External Validity

1. Pretest–treatment interaction

2. Selection–treatment interaction

3. Multiple-treatment interference

4. Specificity of variables

5. Treatment diffusion

6. Experimenter effects

7. Reactive arrangements

Ale

em

Ashra

f

28

Pretest–treatment Interaction

• The pretest sensitizes participants to

aspects of the treatment and thus

influences posttest scores.

• Example: In a physical performance

experiment, the pre-test clues the

subjects to respond in a certain way to

the experimental treatment that would not

be the case if there were no pre-test.

Ale

em

Ashra

f

29

Selection–treatment Interaction

• An effect of some selection factor of intact

groups interacting with the experimental

treatment that would not be the case if the

groups were randomly selected.

• Example: The results of an experiment in

which teaching method is the

experimental treatment, used with a class

of low achievers, do not generalize to

heterogeneous ability students.

Ale

em

Ashra

f

30

Multiple-treatment Interference

• When participants receive more than one treatment, the effect of prior treatment can affect or interact with later treatment, limiting generalizability.

• Example: In a drug experiment the same animals are administered four different drug doses in some sequence. The effects of the second through fourth doses cannot be separated from the possible delayed effects of preceding doses.

Ale

em

Ashra

f

31

Specificity of Variables

• Poorly operationalized variables make it

difficult to identify the setting and

procedures to which the variables can be

generalized.

Ale

em

Ashra

f

32

Experimenter Effects

• Conscious or unconscious actions of the

researchers affect participants’

performance and responses.

• Example: Differences in which the

different researchers give different

instructions to the different groups.

• Age, gender, attractiveness also make a

difference

• Solution: Double blind experiments

Ale

em

Ashra

f

33

Double-blind Experiment

• The experimental group receives the real

drug, the control group gets the placebo

• There are two experimenters but the one

who administers the drug is unaware if he

is distributing the placebo or the drug

• Neither the experimenter nor the subjects

know if they getting the drug or placebo

• Eliminates experimenter bias and placebo

effects.

Ale

em

Ashra

f

34

Reactive Arrangements

• The fact of being in a study affects

participants so that they act in ways

different from their normal behavior. The

Hawthorne and John Henry effects are

reactive responses to being in a study.

Ale

em

Ashra

f

35

Acknowledgements

• Prof. Graham R. Gibbs, University of

Huddersfield.

• Prof. Robert S Michael, Indiana University. (For giving baseline ideas for the slides)

Reference

• Lorraine R. Gay, 2011. Educational

Research: Competencies for Analysis and

Applications (10th Edition). 10 Edition.

Pearson.

• John J. Shaughnessy, 2011. Research

Methods In Psychology, 9th Edition. 9th

Edition. McGraw-Hill.

Ale

em

Ashra

f

36

Ale

em

Ashra

f

37