Generalizability & Field Research

Readings on Sakai:Baumeister, Vohs, & Funder, 2007; Mitchell, 2012

External Validity and Generalizability

External validity: The degree to which results are likely to generalize to other samples, situations, etc. (“generalizability”)

Can we generalize findings from data collected in the lab to everyday life?

Establishing external validity or generalizability requires collecting data outside of the lab (e.g., field research)

Is External Validity Always Essential? External validity is not always essential for

establishing that associations exist

Basic vs. applied science

“Theory testing mode” vs. “Generalizing mode”

Generalizing results across participants Most participants in top psychology journals

are WEIRD WEIRD: Western, Educated, Industrialized, Rich,

and Democratic “The weirdest people in the world?” (Henrich,

Heine, & Norenzayan, 2010) 80% of participants in top psychology journals

and “WEIRD” undergraduate students 96% of psychology samples come from

countries that make up only 12% of the world’s population

Can we assume that results from “WEIRD” allow us to draw conclusions about “human nature”?

Generalizing results across cultures What psychological phenomena might be universal?

Emotional experiences? Self-enhancement?

We should not assume that a specific psychological phenomenon is universal

Cross-cultural research is essential for establishing universality

Challenges in cross-cultural research: Recruitment Methods and measures designed in WEIRD populations

may not be equally valid or sensible in other cultures

Generalizing across settings Does behaviour observed in the lab generalize to real

world behaviour?

Ecological Validity - the extent to which laboratory paradigms are similar to natural experiences Ecological validity is not absolutely essential for a study to

have high external validity …but it might help

“Mundane realism” vs. “Experimental realism”

We might also collect data in naturalistic settings (outside of the lab)

Field Research and Naturalistic Methods

Naturalistic Methods Broadly, naturalistic methods

measure/describe behavior in the context of natural, daily life

Maximizes external validity Examining behavior where it naturally occurs But, minimal control and internal validity

Examples: Observational research Experience sampling methods

Field Experiments Research conducted in a naturalistic setting

(outside of the lab) Very high external validity because data are

collected in the context of “real life” However, internal validity can be lower due to a

lack of control over extraneous variables

Because of challenges to internal validity, field experiments are often paired with a lab study

“Psychology as the Science of Self-Reports and Finger Movements: What happened to actual behavior?”(Baumeister, Vohs, & Funder, 2007)

Psychology is the science of behaviour, yet psychologists rarely measure behaviour directly

Many domains in psychology rely on: Self-report questionnaires Performance on cognitive tasks (e.g., implicit

measures) Example: January, 2006 issue of JPSP:

11 articles with 38 studies The only measure of “behaviour” involved asking

participants to select one of two stimulus persons

“Classic” studies in social-personality psychology included behavioural observation

Bystander intervention (Darley & Latané, 1968)

Milgram’s obediance studies

Cognitive dissonance manipulations (Festinger & Carlsmith, 1959)

Does it matter if we measure behaviour? Introspection can be in accurate (Nisbett & Wilson,

1977, “Telling more than we can know”) Self-reported hypothetical behaviour often differs from

actual behaviour! (e.g., West & Brown, 1975) Would you donate money to a passerby asking for money

for medical care? Actual donations were much lower than people predicted

Affective forecasting (e.g., Wilson & Gilbert, 2003) Hypothetical vs. live interaction (Eastwick, Hunt, &

Neff, 2013) Personality and behaviour

But, behavioural observation can be difficult!

Example: Studying familiarity-attraction with different paradigms Norton, Frost & Ariely (2007) used a trait

evaluation paradigm to study the effect of familiarity on attraction Participants receive a list of 4, 6, 8, or 10 randomly

selected trait adjectives describing a hypothetical person

“How much do you think you would like this person?” Number of traits & liking, r(76) = -.23, p = .05

Is this paradigm ecologically or externally valid? Live interaction does not involve sharing randomly

selected traits! People seek out commonalities Interaction might be responsive and enjoyable

Example: Studying familiarity-attraction with different paradigms (cont.) Reis, Maniaci, Caprariello, Eastwick, & Finkel (2011)

used a live interaction paradigm to see if this effect changed with a more ecologically valid design

Study 2: A sample of 220 participants were paired with a stranger and chatted either 1, 2, 4, 6, or 8 times via instant messaging over the same number of days

1 2 4 6 82.8

3

3.2

3.4

3.6

3.8

4

Number of Chats

Attr

acti

on

F-linear(1,101)= 5.15, p<.03

Example: Studying familiarity-attraction with different paradigms (cont.)

1 2 4 6 80

10

20

30

40

Number of Chats

Percentage of dyads

who contacted each other after the

study

2 (1) = 2.83, p=.05

Meta-Analysis and Convergence Between Lab and Field Results

Aggregating Data: Meta-Analysis meta-analysis – a procedure used to

examine every study that has been conducted on a particular topic to assess the relationship between whatever variables are the focus of the analysis.

By looking at effect sizes across many studies, a general estimate of the strength of the relationship between the variables can be calculated

Social and Medical Research relies on Meta-Analysis more than Physics, Chemistry, Etc. Older sciences have a larger base of existing

research Social and medical sciences are often less

integrative In psychology specifically:

People are variable! We cannot control for all sources of error variance

Research with human subjects must follow ethical guidelines

In predicting human behaviour, many effects are small

… and many studies tend to be underpowered

Steps to Conducting a Meta-Analysis

1. Identify your research question2. Collect studies

inclusion criteria—measures, methods, controls, etc.)

Multiple databases Identifying nonsignificant and unpublished

results3. Extract data from studies

Effect size Code characteristics of samples

4. Analyze data; Examine aggregated effects5. Interpret effects across studies

Does behaviour observed in the lab generalize to real world behaviour?

Mitchell (2012) combined results from 82 meta-analyses, with 217 comparisons of effects in lab vs. field studies Effect sizes in the lab vs. field are highly correlated (r = 64; r = .71 excluding 1 outlier)

… suggesting that lab studies in psychology tend to be externally valid!

Does behaviour observed in the lab generalize to real world behaviour? (cont.) The aggregate results support external

validity… … but they also hide considerable variability:

30 of 215 effects (14%) from lab studies changed direction when examined in the field!

Lab and field studies are more consistent for industrial/organizational psychology (r = .89) than for social psychology (r = .53, with 26% changing direction!)

“Homework Assignment”(for discussion in next week’s class) Pretend that you MUST obtain a statistically

significant result in your group project at any cost.

Try to change your analyses to find a statistically significant result. For instance, you could: Exclude participants for any reason Add control variables Change scores that look unusual

The significant result does not need to be relevant to your hypotheses

Could you write a paper that makes sense of this significant result?