EXPERIMENTAL RESEARCH DESIGN

EXPERIMENTAL RESEARCH

• One of the most important things for you to learn in this course is the difference between correlational research and experimental research. Like correlational research, experimental research concerns relationships between variables. Unlike correlational research, however, experimental research provides strong evidence for causal interpretations. Here we will focus on the two most important features of experimental research.

NON EXPERIMENTAL RESEARCH DESIGN

• In non experimental research, the researcher starts from the effect or outcome and attempts to determine causation.

• Nonexperimental research involves observing and measuring things as they are. Naturalistic observation, interview, survey, case history, and psychometric scales are some of the methods used when it is not possible or unethical to manipulate an independent variable. Nonexperimental research is used to provide solutions to problems. Nonexperimental research can add to what we know by common sense because we can test our beliefs to see how true they are.

QUASI-EXPERIMENTAL

• Quasi-ExperimentsThe prefix “quasi” means, in essence, “sort of.” So a quasi-experiment is a “sort of” experiment. Specifically, a quasi-experiment is a study that includes a manipulated independent variable but lacks important controls (e.g., random assignment), or a study that lacks a manipulated independent variable but includes important controls. So a quasi-experiment has some features of a well conducted experiment but not others.

Types of Quasi-Experiments

• Non-Equivalent Groups Design• A non-equivalent groups design includes an

existing group of participants who receive a treatment and another existing group of participants to serve as a control or comparison group. Participants are not randomly assigned to conditions, but rather are assigned to the treatment or control conditions along with all the others in their existing group.

• Pretest-Posttest DesignIn a pretest-posttest design, a single group of participants is measured on the dependent variable both before and after the manipulation of the independent variable. Imagine that a group of 100 sixth graders is given a test of their attitudes toward drugs. This is the pretest. Then, a week later, a police officer comes to school and presents an anti-drug program (complete with “cool” decorated car and performing police dog). This is the treatment. Then, in another week, the students are given another test of their attitudes toward drugs. This is the posttest. Obviously, the substantive question here is whether the students’ attitudes toward drugs change after being presented with the anti-drug program.

• Interrupted Time-Series Designs• A time series is simply a set of measurements

of a variable taken at various points in time. For example, we could measure the moods of the students in our class each day throughout the semester, and we could see how people’s moods changed (or did not change) over time. In an interrupted time-series design, a time series like this (the dependent variable) is interrupted (usually near the middle) by the manipulation of the independent variable.

UNSUITABILITY OF EXPERIMENTAL RESEARCH DESIGN

* Elimination of extraneous variables is not always possible.* Experimental situation may not relate to the real world.* It may be unethical or impossible to randomly assign people to groups.

LIMITATIONS• The first is that sometimes you cannot do an experiment

because you cannot manipulate the independent variable, either for practical or ethical reasons. For example, if you are interested in the effects of a person’s culture on their tendency to help strangers, you cannot do an experiment. Why not? You cannot manipulate a person’s culture. Or if you are interested in how damage to a certain part of the brain affects behavior, you cannot do an experiment. Why not? You cannot go around damaging people’s brains to see what happens. In such cases, correlational research is the only alternative.

• The second limitation of experimental research is that sometimes controlling extraneous variables means creating situations that are somewhat artificial. A good example is provided research on the effect of smiling on first impressions. To control extraneous variables, people are typically brought into a laboratory and asked standard questions about a small number of posed stimulus photographs. It is legitimate to ask, however, whether the effect of smiling is likely to be the same out in the "real world" where people are actually interacting with each other. For a good discussion of why this is not always a problem, though, see Stanovich's (2007) discussion of the "artificiality criticism" of psychological research.

SOME OTHER LIMITATIONS

• Uses casual relationships that may be bias.• Scientist manipulates values so they may not

be making a completely objective experiment.• People can be influenced by what they see

around them and may give answers that they think the researcher wants to hear rather than how they think and feel on a subject.

A CASE STUDY USING EXPERIMENTAL DESIGN

• Problem: Researchers wanted to evaluate a family-participation drug prevention program in the Boys and Girls Clubs. Four clubs were purposively selected to receive the program because they had directors who would strongly support the promotion of family involvement and would give the program coordinator the flexibility to work in nontraditional ways to encourage family participation.

• A Solution: Other Boys and Girls Clubs that were similar on socioeconomic and other demographic variables to the family-participation program clubs were selected as comparison groups.

• A Drawback to the Solution: Children in the family-participation program were about a quarter of a year younger, on the average, than those in the comparison groups. While all groups were predominantly African American, there were differences in the second most frequent racial/ethnic groups with differences in the Hispanic and Caucasian mix. There were differences in the gender composition of the groups (e.g., 35% female in the family-participation clubs and 41% female in the control clubs).

Validity in Experimental Design

• Internal validity: Internal validity tries to examine whether the observed effect on a dependent variable is actually caused by the independent variables in question.

• External validity: External validity refers to the generalization of the results of an experiment. The concern is whether the result of an experiment can be generalized beyond the experimental situations

Threats to internal validity

• Ambiguous temporal precedenceLack of clarity about which variable occurred first may yield confusion about which variable is the cause and which is the effect.• ConfoundingA major threat to the validity of causal inferences is confounding: Changes in the dependent variable may rather be attributed to the existence or variations in the degree of a third variable which is related to the manipulated variable. Where spurious relationships cannot be ruled out, rival hypotheses to the original causal inference hypothesis of the researcher may be developed.

• Selection biasSelection bias refers to the problem that, at pre-test, differences between groups exist that may interact with the independent variable and thus be 'responsible' for the observed outcome. Researchers and participants bring to the experiment a myriad of characteristics, some learned and others inherent. For example, sex, weight, hair, eye, and skin color, personality, mental capabilities, and physical abilities, but also attitudes like motivation or willingness to participate.

• HistoryEvents outside of the study/experiment or between repeated measures of the dependent variable may affect participants' responses to experimental procedures. Often, these are large scale events (natural disaster, political change, etc.) that affect participants' attitudes and behaviors such that it becomes impossible to determine whether any change on the dependent measures is due to the independent variable, or the historical event.

• MaturationSubjects change during the course of the experiment or even between measurements. For example, young children might mature and their ability to concentrate may change as they grow up. Both permanent changes, such as physical growth and temporary ones like fatigue, provide "natural" alternative explanations; thus, they may change the way a subject would react to the independent variable. So upon completion of the study, the researcher may not be able to determine if the cause of the discrepancy is due to time or the independent variable.

• Instrument change (instrumentalityThe instrument used during the testing process can change the experiment. This also refers to observers being more concentrated or primed, or having unconsciously changed the criteria they use to make judgments. This can also be an issue with self-report measures given at different times. In this case the impact may be mitigated through the use of retrospective pretesting. If any instrumentation changes occur, the internal validity of the main conclusion is affected, as alternative explanations are readily available.

• Experimenter biasExperimenter bias occurs when the individuals who are conducting an experiment inadvertently affect the outcome by non-consciously behaving in different ways to members of control and experimental groups. It is possible to eliminate the possibility of experimenter bias through the use of double blind study designs, in which the experimenter is not aware of the condition to which a participant belongs.

• Repeated testing (also referred to as testing effects)

Repeatedly measuring the participants may lead to bias. Participants may remember the correct answers or may be conditioned to know that they are being tested. Repeatedly taking (the same or similar) intelligence tests usually leads to score gains, but instead of concluding that the underlying skills have changed for good, this threat to Internal Validity provides good rival hypotheses.

Threats to external validity

• Reactive effects of experimental arrangements: It is difficult to generalize to non-experimental settings if the effect was attributable to the experimental arrangement of the research.

• Multiple treatment interference: As multiple treatments are given to the same subjects, it is difficult to control for the effects of prior treatments.

• Situation: All situational specifics (e.g. treatment conditions, time, location, lighting, noise, treatment administration, investigator, timing, scope and extent of measurement, etc. etc.) of a study potentially limit generalizability.

• Aptitude–treatment Interaction: The sample may have certain features that may interact with the independent variable, limiting generalizability. For example, inferences based on comparative psychotherapy studies often employ specific samples (e.g. volunteers, highly depressed, no comorbidity).

• The environment at the time of test may be different from the environment of the real world where these results are to be generalized.

• Pre-test effects: Individuals who were pretested might be less or more sensitive to the experimental variable or might have "learned" from the pre-test making them unrepresentative of the population who had not been pre-tested.

• Post-test effects: If cause-effect relationships can only be found when post-tests are carried out, then this also limits the findings because of the pre-test estimation.

• Treatment at the time of the test may be different from the treatment of the real world.