Chapter 8Construct and External Validity in Experimental Research ♣

Construct Validity

External Validity

Cautions in Evaluating the External Validity of Experiments

Relationship between Internal and External Validity

Back to Brief Contents

8.1 Construct Validity -1

Definition—extent to which we can

infer higher-order constructs for our operationsFig 8.1

Constructs are used for Research participants Independent variable Dependent variable Experimental setting

Back to Chapter Contents

8.1 Construct Validity -2

Assessing Construct Validity

Threats to Construct Validity Reactivity to the Experimental Situation

Experimenter Effect

Back to Chapter Contents

Assessing Construct Validity -1

Have a clear definition of the construct of interest Problem is identifying prototypical features of

the constructs

Affects ability to identify the concrete operations used to represent the construct

Back to Chapter Contents

Assessing Construct Validity -2(end)

Assess the match between the constructs and the operations used to represent them Ways to assess

Content validity

Criterion-related validity

— Predictive

— concurrent

Any other source of evidence

Back to Chapter Contents

Threats to Construct Validity

Inadequate explanation of the construct Tab 8.1 Construct confounding Mono-operation bias Mono-method bias Confounding constructs with level of constructs Treatment-sensitive factorial structure Reactive self-report changes Reactivity to the experimental situation Experimenter effects Novelty and disruption effects Compensatory equalization Compensatory rivalry Treatment diffusion

Back to Chapter Contents

Threats to Construct Validity

Inadequate explanation of the constructIf a construct is not adequately explained and analyzed,

it can lead to a set of operations that do not represent the construct adequately

Construct confoundingthe operations used in a study represent more than one construct

Mono-operation biasa study uses only one operationalization of a construct

This typically results in an underrepresentation of the construct and lowers construct validity

Back to Chapter Contents

Threats to Construct Validity

Mono-method biasa study uses only one method (e.g., physiological recording) to operationalize a construct

The method used may influence the results

Confounding constructs with level of constructsa study investigates only a few levels of a construct (e.g., three doses of a drug), but makes inferences about the overall construct (e.g., the overall effect of the dose)

Treatment-sensitive factorial structurean instrumentation change that occurs because of the experimental treatment

Back to Chapter Contents

Threats to Construct Validity

Reactive self-report changeschanges that a research participant may make on self-report measures as a result of a motivational shift after being included in the experimental study

Reactivity to the experimental situationresearch participants’ perceptions and motives can affect the responses they make to the dependent variable

and these responses can be interpreted as part of the treatment construct being tested

Back to Chapter Contents

Threats to Construct Validity

Experimenter effectsthe experimenter’s attributes and expectancies can influence the responses made by the research participants

and these responses can be interpreted as part of the treatment construct being tested

Novelty and disruption effectsResearch participants usually respond better to a new and novel situation and poorly to one that disrupts their routine

These effects are part of the overall treatment effect

Back to Chapter Contents

Threats to Construct Validity

Compensatory equalizationindividuals try to provide the same benefits or services to the control group that are received by the experimental group

Compensatory rivalryindividuals resent being assigned to the control group and respond more negatively than would be expected, because of the resentment they feel

Treatment diffusionindividuals in one treatment group receive some or all of another group’s treatment

Back to Chapter Contents

Reactivity to the Experimental Situation 1/3

Refers to research participants’ motives and perceptions influencing the response to the DV

Participant Effect Demand characteristics

Any of the cues available in an exp, such as

instructions, experimenter, rumors, experimental settings

Primary motive — positive self-presentation

Back to Chapter Contents

Reactivity to the Experimental Situation 2/3

Conditions producing a positive self-presentation motive Tedeschi, Schlenker, & Bonoma (1971)

P believe that others view their behavior as indicative of their true intentions, beliefs, or feelings

Exp. constructed P believe that others think their behavior is externally determined

Back to Chapter Contents

Reactivity to the Experimental Situation 3/3

Implication for research

Intertreatment interaction

Perception by P in different treatment groups that they can fulfill the positive self-presentation motive by responding in different ways

Intratreatment interaction

Perception by P in the same treatment group that they can fulfill the positive self-presentation motive by responding in different ways

Back to Chapter Contents

Experimenter Effect 1/3

Experimenter Effect Fig 8.2

Experimenter has motive of supporting the study hypothesis

Can unintentionally lead to recording errors

Back to Chapter Contents

Experimenter Effect 2/3

Ways experimenter may bias the study Experimenter attributes

Biosocial attributes

(e.g.) age, sex, race Psychosocial attributes

psychometrically determined characteristics

(e.g.) anxiety level, hostility, authoritarianism, intelligence, dominance, warmth

Situational factors

(e.g.) prior contact of P, naive or experienced

Experimenter expectancies (cont.)

Back to Chapter Contents

Experimenter Effect 3/3

Ways experimenter may bias the study Experimenter attributes Experimenter expectancies

Effect on experimenter

(e.g.) recording errors (biased, but low error rate) Effect on research participant Tab 8.2

Mediation of expectancies

— Handling in animal research

— Nonverbal communication in human studies Magnitude of expectancies—can exist in animal and

human research and can be greater than the IV Fig 8.3

Back to Chapter Contents

8.2 External Validity 1/4

Definition—generalizing across people, settings, treatment variations, outcomes and times

Represents a test of interactions

Threats to external validity Population validity

Ecological validity

Temporal validity

Treatment variation validity

Outcome validity

Back to Chapter Contents

8.2 External Validity 2/4

Population ValidityThe extent to which the results of a study can be generalized to the larger population Two-step inferential process Fig 8.4 Tab 8.3

Sample Experimentally accessible population

Target population College students as P

less emotionally and impulsively in laboratory studies Selection by Treatment Interaction

if interaction exists

cannot generalize to the target population

Back to Chapter Contents

8.2 External Validity 3/4

Ecological ValidityThe extent to which the results of a study can be generalized across settings or environmental conditions

Temporal ValidityThe extent to which the results of an experiment can be generalized across time Seasonal variation

A variation that occurs at regular time intervals Cyclical variation

A regular variation that occurs within people and other organisms

(ex) circadian rhythm, MC

Back to Chapter Contents

8.2 External Validity 4/4

Treatment Variation Validity (IV)The generalizability of results across variation of the treatment

Outcome Validity (DV)The generalizability of results across different but related dependent variables

Back to Chapter Contents

8.3 Cautions in Evaluating the External Validity of Experiments

Some studies conducted to increase knowledge and not to generalize to a real-life situation

Some studies assess a theoretical process

Moving out of the laboratory does not insure generalization

Back to Chapter Contents

8.4 Relationship between Internal and External Validity

Relationship between internal and external validity—often an inverse relationship

Back to Chapter Contents