PSY 3330 Research Methods

Research Methods:Validity and Threats to

Validity

Generating Research Hypotheses (Review – Figure 1.2)

Ideas lead toobservationslibrary research

Initial theoryPredictions or

hypothesesOperationalized

in new studiesFindings used

to support, refute or refine theory.

Characteristics of a Good Study Question

“FINER”F Feasible

I Interesting N Novel E Ethical R Relevant

Actually testing three sets of hypothesesThe null hypothesisThe confounding variable hypothesesThe causal hypothesis

Accept causal hypothesis only if youreject null hypothesis (statistical analysis)rule out each potential confounding variable

hypothesis (based on appropriate controls)

Testing Research Hypotheses

Correlation (also called covariation)Relationship found between variables

Time orderCause must occur before result

NonspuriousnessAlternative explanations must

be eliminated from possibilityExperiments are intended to

reduce or rule out alternative explanations and confoundingvariables

Criteria for Nomothetic Causality (See Table 2.2 on p 34)

Asking “the Question”The PICO format:

P PopulationI Intervention or Interest areaC Comparison intervention or statusO Outcome

Applying the “PICO” formatOriginal: “What is the usefulness or

accuracy of the current 1-10 pain scale assessment in treating a patient’s pain, and what are other options that may prove more useful?”

Research: Does a 10 point pain Visual Analog Scale (____, ____) accurately assess pain in the first day postop abdominal total hysterectomy patient when compared with the Faces Pain Scale (Pasaro, 1997)?

After the experiment, what do you think about the null hypothesis?

What do you think about the research (causal)

hypothesis?

Evaluating Hypotheses

Statistical Validity – carrying out the actual statistical analysis properly

Construct Validity refer most often to a characteristic of an instrument but also to the whole study

Internal Validity refers to a characteristic of a study’s design

External Validity refers to the generalizability of study findings

Types of Research Validity

Do the variables covary?Is the relationship

statistically significant?Threatened by

Unreliable measuresViolations of statistical assumptions How do we detect these problems???

Strengthened byUsing well validated measuresHaving approximately equal sample sizes in

each group)

Statistical Validity

Can we infer higher-0rder (more abstract) constructs from the operations used in our experiment or study?E.g., Milgram’s study – authority? Social conformity?

Is our theory the best explanation for the results?Threatened by

Any alternative explanation for the resultsHOW do we locate these alternative explanations?

Strengthened byUsing well-validated constructs to build the theoretical

predictions for the studySee list of threats on p. 171 in Table 6.2

Construct Validity

Do the results apply to the broader population?

Threatened byUnrepresentative samplesGeneralizing beyond the limits of the sampleHOW do we know when this problem is

present???Strengthened by

Gathering a representative sample (if possible)Clearly describing sample, so that other

researchers will know the limits of generalization

External Validity

Is the independent variable responsible for the observed changes in the dependent variable?

Threatened by Confounding variablesHOW do we detect the presence of

confounding variables????Strengthened by

Adding adequate controls to reduce or eliminate confounding

Internal Validity

Confounding and internal validityMany sources for confounding (covered next)With proper controls, confounding can be

virtually eliminated (see Chapter 9)

Confounding and construct validityMake sure that you have considered alternative

theoretical explanations for the anticipated phenomenon

HOW????

Avoiding Confounding

Pretest-Posttest Research Design

Single-group, pretest-posttest design compares pre-treatment and post-treatment scores to determine improvement

Fails to control most sources of confounding

Many theories include an assumption about human motivations or behavior

Behavioral economists & some psychologists explore deeply whether these assumptions prove accurate

Assumptions threaten validityDan Ariely – Predictably Irrational (TED)

See Dan Ariely’s website, where he often has experiments.

Historical events may occur during the course of the experiment.Remember Pygmalion effect & its story

Maturation of the subjects.Testing and retesting can

influence awareness of variables or behaviorLearn Hawthorne effect & its story.

Instrumentation – measurement methods or procedures may not be equivalent

Sources of Internal Invalidity

Statistical regression of subjects starting out in extreme positions.

Selection biases (we will see several types)

Experimental mortality(a.k.a. attrition) – subjects drop out of the study before it's finished.

Sequence effects – Performance on one measure is related to previous experience with other measures. Outcome depends on the sequence of measures.

Sources of Internal Invalidity

Demoralization subjects incontrol group find out, loseinterest in study, stop trying

Diffusion of treatment (those who get the experimental stimulus spread it to controls)

Rivalry (controls change behavior to try to beat the experimental group)

Equalization of treatment (researcher compensates controls for not getting treatment)

Social threats to validity

Social Psychology Experiment:Conformity to Norms (file)

Experiment: Conformity to gender roles. We placed common male and female signs on opposite transparent doors instructing people to walk through the correct entrance.

(Sorry, no music. Youtube made me remove it)

Participants are not passiveThey try to understand the study to help them

to know what they “should do” (termed subject effects)

Respond to subtle cues about what is expected (termed demand characteristics)

Placebo effect: treatment effect due to expectations that the treatment will work

Subject Effects

Based on the expectations of the researcher

Can affect the outcome of studies if not controlled

May be due to the experimenter providing demand characteristics to the participant

Not the same as scientific fraud (which is deliberate)

Experimenter Effects

Three closely-tied concepts

Validity The accuracy of the study or procedureIncreased by using appropriate control

procedures

The more controls we employ, the higher the level of constraint

Controls may increase some types of validity while, by their unnatural aspect, decreasing other types of validity.

Validity, Control, and Constraint

Risk is balanced by rewardA poorly designed study will provide no useful

information; therefore, any risk would be unacceptable

Informed ConsentVirtually guarantees that you will have

confounding due to selection because some people will refuse to participate

A small price to pay to maintain ethical standards

Ethical Principles

Start by building a research hypothesisTesting the research hypothesis is actually

testing three hypotheses(1) null; (2) confounding-variable; (3) causal

Several types of validityMany potential confounding variablesSubject and experimenter effects can also

affect the outcome of the study

Summary

PSY 3330 Research Methods

Research Methods:Validity and Threats to

Validity