c. 2
000
Del
Sie
gle
Experimental
RESEARCH DESIGNS
This presentation was prepared by Del Siegle.
Some of the material is from an earlier presentation by Scott Brown.
c. 2
000
Del
Sie
gle
Experimental Research Designs have Two Purposes:
• …to provide answers to research questions
• ...to control variance (differences)
c. 2
000
Del
Sie
gle
The main function of the experimental research
design is to control variance.
• Principle: maximize systematic variance, control extraneous systematic variance, and minimize error variance.
In other words control variance.
c. 2
000
Del
Sie
gle
Therefore the researcher attempts to:• maximize the variance of the variable(s) of the research
hypothesis (i.e., maximize the difference in the dependent variable [outcome] caused by maximizing the differences in the independent variable [treatment]).
• control the variance of extraneous or "unwanted" variables that may have an effect on the experimental outcomes, but which he/she is not interested (limit factors other than the treatment (IV) that could be causing differences in the outcome (DV) .
• minimize the error or random variance (i.e., avoid unreliable measurement instruments which have high errors of measurement ).
c. 2
000
Del
Sie
gle
Maximization of Experimental Variance• experimental variance
– the variance due to the manipulated (i.e., treatment) or attribute (i.e., gender) variables (IV)
research precept: – design, plan and conduct
research so that experimental conditions are as different as possible on the independent variable.
c. 2
000
Del
Sie
gle
Control of Extraneous Variables (EV)• eliminate the variable (i.e., if you are worried
about gender, only include one gender in the study). • randomization (i.e., if you randomly assign subjects to
groups, the extraneous variable should be equally distributed among the groups)
• build it into the design – make it a moderator variable (i.e., if you are worried about gender, build it into the analysis [2-way ANOVA]—we’ll learn about this later)
• match subjects (i.e., match the characteristics of subjects and put one of each matched pair in each group)
• statistically equate groups (i.e., use ANCOVA [Analysis of Covariance] to analyze the data with the extraneous variable used as a covariate—we’ll learn about this later)
c. 2
000
Del
Sie
gle
Minimizing Error Variance has Two Principle Aspects:
• reduction of errors of measurement through controlled conditions (i.e., standardize testing procedures)
• increase in the reliability of measures (i.e., revise test instruments or find more reliable ones)
c. 2
000
Del
Sie
gle
Experimental Designs
Should be Developed to
Ensure Internal and
External Validity of the Study
c. 2
000
Del
Sie
gle
Internal Validity:
• Are the results of the study (DV) caused by the factors included in the study (IV) or are they caused by other factors (EV) which were not part of the study?
There are 16 common threats to internal validity.
c. 2
000
Del
Sie
gle
(Selection Bias/Differential Selection) -- The groups may have been different from the start. If you were testing instructional strategies to improve reading and one group enjoyed reading more than the other group, they may improve more in their reading because they enjoy it, rather than the instructional strategy you used.
Subject Characteristics
Threats to Internal Validity
c. 2
000
Del
Sie
gle
(Mortality) -- All of the high or low scoring subject may have dropped out or were missing from one of the groups. If we collected posttest data on a day when the honor society was on field trip at the treatment school, the mean for the treatment group would probably be much lower than it really should have been.
Loss of Subjects
Threats to Internal Validity
c. 2
000
Del
Sie
gle
Perhaps one group was at a disadvantage because of their location. The city may have been demolishing a building next to one of the schools in our study and there are constant distractions which interfere with our treatment.
Location
Threats to Internal Validity
c. 2
000
Del
Sie
gle
The testing instruments may not be scores similarly. Perhaps the person grading the posttest is fatigued and pays less attention to the last set of papers reviewed. It may be that those papers are from one of our groups and will received different scores than the earlier group's papers
Threats to Internal Validity
Instrumentation Instrument Decay
c. 2
000
Del
Sie
gle
The subjects of one group may react differently to the data collector than the other group. A male interviewing males and females about their attitudes toward a type of math instruction may not receive the same responses from females as a female interviewing females would.
Threats to Internal Validity
Data Collector Characteristics
c. 2
000
Del
Sie
gle
The person collecting data my favors one group, or some characteristic some subject possess, over another. A principal who favors strict classroom management may rate students' attention under different teaching conditions with a bias toward one of the teaching conditions.
Threats to Internal Validity
Data Collector
Bias
c. 2
000
Del
Sie
gle
The act of taking a pretest or posttest may influence the results of the experiment. Suppose we were conducting a unit to increase student sensitivity to prejudice. As a pretest we have the control and treatment groups watch Shindler's List and write a reaction essay. The pretest may have actually increased both groups' sensitivity and we find that our treatment groups didn't score any higher on a posttest given later than the control group did. If we hadn't given the pretest, we might have seen differences in the groups at the end of the study.
Threats to Internal Validity
Testing
c. 2
000
Del
Sie
gle
Something may happen at one site during our study that influences the results. Perhaps a classmate dies in a car accident at the control site for a study teaching children bike safety. The control group may actually demonstrate more concern about bike safety than the treatment group.
Threats to Internal Validity
History
c. 2
000
Del
Sie
gle
There may be natural changes in the subjects that can account for the changes found in a study. A critical thinking unit may appear more effective if it taught during a time when children are developing abstract reasoning.
Threats to Internal Validity
Maturation
c. 2
000
Del
Sie
gle
The subjects may respond differently just because they are being studied. The name comes from a classic study in which researchers were studying the effect of lighting on worker productivity. As the intensity of the factory lights increased, so did the worker productivity. One researcher suggested that they reverse the treatment and lower the lights. The productivity of the workers continued to increase. It appears that being observed by the researchers was increasing productivity, not the intensity of the lights.
Threats to Internal Validity
Hawthorne Effect
c. 2
000
Del
Sie
gle
One group may view that it is in competition with the other group and may work harder than they would under normal circumstances. This generally is applied to the control group "taking on" the treatment group. The terms refers to the classic story of John Henry laying railroad track.
Threats to Internal Validity
John Henry Effect
c. 2
000
Del
Sie
gle
The control group may become discouraged because it is not receiving the special attention that is given to the treatment group. They may perform lower than usual because of this.
Threats to Internal Validity
Resentful Demoralization of the Control Group
c. 2
000
Del
Sie
gle
(Statistical Regression) -- A class that scores particularly low can be expected to score slightly higher just by chance. Likewise, a class that scores particularly high, will have a tendency to score slightly lower by chance. The change in these scores may have nothing to do with the treatment.
Threats to Internal Validity
Regression
c. 2
000
Del
Sie
gle
The treatment may not be implemented as intended. A study where teachers are asked to use student modeling techniques may not show positive results, not because modeling techniques don't work, but because the teacher didn't implement them or didn't implement them as they were designed.
Threats to Internal Validity
Implementation
c. 2
000
Del
Sie
gle
Threats to Internal Validity
Compensatory Equalization of Treatment
Someone may feel sorry for the control group because they are not receiving much attention and give them special treatment. For example, a researcher could be studying the effect of laptop computers on students' attitudes toward math. The teacher feels sorry for the class that doesn't have computers and sponsors a popcorn party during math class. The control group begins to develop a more positive attitude about mathematics.
c. 2
000
Del
Sie
gle
Experimental Treatment Diffusion
Threats to Internal Validity
Sometimes the control group actually implements the treatment. If two different techniques are being tested in two different third grades in the same building, the teachers may share what they are doing. Unconsciously, the control may use of the techniques she or he learned from the treatment teacher.
c. 2
000
Del
Sie
gle
Once the researchers are confident that the outcome (dependent variable) of the experiment they are designing is the result of their treatment (independent variable) [internal validity], they determine for which people or situations the results of their study apply [external validity].
c. 2
000
Del
Sie
gle
External Validity:
• Are the results of the study generalizable to other populations and settings?
External validity comes in two forms: population and ecological.
c. 2
000
Del
Sie
gle
1. ...the extent to which one can generalize from the study sample to a defined population-- If the sample is drawn from an accessible population, rather than the target population, generalizing the research results from the accessible population to the target population is risky.
2. ...the extent to which personological variables interact with treatment effects-- If the study is an experiment, it may be possible that different results might be found with students at different grades (a personological variable).
Threats to External Validity (Population)
Population Validity is the extent to which the results of a study can be generalized from the specific sample that was studied to a larger group of subjects. It involves...
c. 2
000
Del
Sie
gle
Ecological Validity is the
extent to which the results of an experiment can be generalized from the set of environmental conditions created by the researcher to other environmental conditions (settings and conditions).
Threats to External Validity (Ecological)
There are 10 common threats to external validity.
c. 2
000
Del
Sie
gle
(not sufficiently described for others to replicate) If the researcher fails to
adequately describe how he or she conducted a study, it is difficult to determine whether the results are
applicable to other settings.
Threats to External Validity (Ecological) Explicit
description of the
experimental treatment
c. 2
000
Del
Sie
gle
(catalyst effect)If a researcher were to apply several treatments, it is difficult to determine how well each of the treatments would work individually. It might be that only the combination of the treatments is effective.
Threats to External Validity (Ecological)
Multiple-treatment interference
c. 2
000
Del
Sie
gle
(attention causes differences)Subjects perform differently because they know they are being studied. "...External validity of the experiment is jeopardized because the findings might not generalize to a situation in which researchers or others who were involved in the research are not present" (Gall, Borg, & Gall, 1996, p. 475)
Threats to External Validity (Ecological)
Hawthorne effect
c. 2
000
Del
Sie
gle
Threats to External Validity (Ecological)
(anything different makes a difference)A treatment may work because it is novel and the subjects respond to the uniqueness, rather than the actual treatment. The opposite may also occur, the treatment may not work because it is unique, but given time for the subjects to adjust to it, it might have worked.
Novelty and disruption effect
c. 2
000
Del
Sie
gle
(it only works with this experimenter)The treatment might have worked because of the person implementing it. Given a different person, the treatment might not work at all.
Threats to External Validity (Ecological)
Experimenter effect
c. 2
000
Del
Sie
gle
(pretest sets the stage)A treatment might only work if a pretest is given. Because they have taken a pretest, the subjects may be more sensitive to the treatment. Had they not taken a pretest, the treatment would not have worked.
Threats to External Validity (Ecological)
Pretest sensitization
c. 2
000
Del
Sie
gle
(posttest helps treatment "fall into place")The posttest can become a learning experience. "For example, the posttest might cause certain ideas presented during the treatment to 'fall into place' " (p. 477). If the subjects had not taken a posttest, the treatment would not have worked.
Threats to External Validity (Ecological)
Posttest sensitization
c. 2
000
Del
Sie
gle
Interaction of history and treatment effect
Threats to External Validity (Ecological)
(...to everything there is a time...)Not only should researchers be cautious about generalizing to other population, caution should be taken to generalize to a different time period. As time passes, the conditions under which treatments work change.
c. 2
000
Del
Sie
gle
(maybe only works with M/C tests)A treatment may only be evident with certain types of measurements. A teaching method may produce superior results when its effectiveness is tested with an essay test, but show no differences when the effectiveness is measured with a multiple choice test.
Threats to External Validity (Ecological)
Measurement of the dependent variable
c. 2
000
Del
Sie
gle
Interaction of time of measurement and treatment effect
Threats to External Validity (Ecological)
(it takes a while for the treatment to kick in)It may be that the treatment effect does not occur until several weeks after the end of the treatment. In this situation, a posttest at the end of the treatment would show no impact, but a posttest a month later might show an impact.
c. 2
000
Del
Sie
gle
First, and foremost, an experiment must have internal validity. If the researchers cannot be certain that the results of the experiment are dependent on the treatment, it does not matter to which people or situations they wish to generalize (apply) their findings. The importance of external validity is reliant on
having internal validity in much the same way that the validity of a measurement instrument is reliant on the instrument being reliable.
However, the more tightly experimenters design their study, the more they limit the populations and settings to whom they can generalize their findings.
c. 2
000
Del
Sie
gle
c. 2
000
Del
Sie
gle
Suppose a researcher wants to study the effect of a reading program on reading achievement.
She might implement the reading program with a group of students at the beginning of the school year
X
and measure their achievement at the end of the year.
OThis simple design is known as a one-shot case study design.
c. 2
000
Del
Sie
gle
X O
Unfortunately, the students’ end of year reading scores could be influenced by other instruction in school, the students’ maturation, or the treatment.
We also do not know whether the students’ reading skills actually changed from the start to end of the school year.
We could improve on this design by giving a pretest at the start of the study.
OThis is known as a one-group pretest-posttest design.
c. 2
000
Del
Sie
gle
X OO
Unfortunately, the students’ end of year reading scores still could be influenced by other instruction in school, the students’ maturation, or the treatment.
OO
Our researcher may wish to have a comparison group.
This is a static-group pretest-posttest design.
c. 2
000
Del
Sie
gle
If our researcher believes that the pretest has an impact on the results of the study, she might not include it.
X OO
OO
This is a static-group comparison design.
c. 2
000
Del
Sie
gle
X OO
OO
RR
Because our researcher did not pretest, she might wish to randomly assign subjects to treatment and control group.
Random assignment of subject to groups should spread the variety of extraneous characteristics that subjects possess equally across both groups.
This is a randomized posttest-only, control group design.
c. 2
000
Del
Sie
gle
X OO
OO
This is a randomized pretest-posttest control group design.
RR
Of course, our researcher could also include a pretest with her random assignment.
c. 2
000
Del
Sie
gle
Occasionally researchers combine the randomized pretest-posttest control group design with the randomized posttest-only, control group design.
This is a randomized Solomon four-group design.
X OO
RR
X OO
OO
RR
c. 2
000
Del
Sie
gle
With the randomized Solomon four-group design, all groups are randomly assigned
X
X
OO
OO
OO
RR
RR
and given the posttest.
Two of the groups are given pretests.
One of the pretest groups is assigned to treatment and one of the non-pretest groups is assigned to treatment.
c. 2
000
Del
Sie
gle
Each of the designs described in this section has advantages and disadvantages that influence the studies internal and external validity.
This presentation was prepared by Del Siegle.
Some of the material is from an earlier presentation by Scott Brown.