Validity = accuracy of study Reliability = consistency of results
VALIDITY- Does your study reveal accurate and representative results?
Reliability
• Is the measure able to give consistent measures over time?
Reliability checks?
• Test-retest• Split-half• Inter-observer
• Convergence…replications
VALIDITY- Does your study reveal accurate and representative results?
• Internal Validity and External Validity
– External Validity- generalizability• Issues
» Sampling» Return rates» …..
…External Validity
• Generalizeability– Across population– Across environments– Across time
Internal Validity
Convergent validity
• Different manipulations• Different measuring devices• Different populations…etc
• Replication
COMMON THREATS TO VALIDITY
• Selection bias– random selection and
assignment– Pretesting and matching
– Recruitment technique
History and Maturation Effects
• Changes in the DV over time• Between pretest and post-test • Or between test sessions
– constancy
Mortality
• Subject loss Over time
• May reflect unintended effect of IV or DV• Subject loss may result in selection bias
STOP FOR FUN!
• ESP in PSYCHOLOGY??– Somewhat critical– Lacking empirical validation– Should we remain open-minded?
– Let’s give it a shot
One of these symbols on each trial
STOP until test complete
Trial Number Symbol
1 red 2 blue 3 green 4 green 5 red 6 yellow 7 yellow 8 blue 9 red 10 blue 11 blue 12 yellow 13 red 14 green 15 green 16 green 17 blue 18 red 19 yellow 20 yellow
What is Chance Performance?
-1 out of 4 for each trial…-20 trials
Chance performance = 5
Just for fun let’s try again..this time seriously!
answers
Trial Number Symbol
1 Blue 2 Blue 3 Yellow 4 Red 5 Green 6 Yellow 7 Yellow 8 Red 9 Red 10 Green 11 Green 12 Blue 13 Red 14 Green 15 Yellow 16 Red 17 Blue 18 Blue 19 Yellow 20 green
Statistical Regression
• Regression to the mean with repeated testing
• Watch out for extreme scores..they may be due to chance
Instrumentation
• Changes in DV due to changes in measurement, not due to IV
Sequence Effects-repeated testing effects
• Practice effects• Test sensitization• Boredom/fatigue• Carry-over
– Randomization and counterbalancing
Subject Bias
• Subjects can have attitude! RESPONSE SETS that produce inaccurate results– Soc desireability– Cooperitivity– Negativity– Volunteerism– -blindin-recruitment
-deception or disguiselie detectors
Experimentor Bias
• Experimentors like to be correct! CONFIRMATIONAL BIAS
• Demand characteristics– constancy– Automation– Double blinding
How would you use blinding in a MJ drug study?
PLACEBO EFFECTS ARE REAL
• Expectation
• What might someone expect MJ to do?
• Beliefs/expectations can lead to real results….that are not related to the real effects of the IV.
DOUBLE-BLIND PLACEBO CONTROL DESIGNS
• Fake MJ?/ Ciggarette/ Pill other• Constancy• Can tease apart placebo effects from real
drug effects.
RANGE EFFECTS
• Don’t forget the inability to see an affect of an IV on a DV does not necessarily mean there isn’t one.
• Ceiling effects• Basement effects
ULTIMATE CONTROL
• Control procedures may be tailored for unique experiments
• Ultimate control may never be completely achieved
BEING A REAL PSYCHOLOGICAL SCIENTIST??
• The crux- identification of potential confounds
• AND• RIVAL HYPOTHESIS
RIVAL HYPOTHESIS
• Alternative explanations for experimental results
• Alternative to the research hypothesis
Consider the following
Hampton Court Maze
The radial arm maze for rats
Training of rats on the RAM
• Initial freezing• Adaptation and exploration• Neophobia to food pellet• Adaptation to food• Rapid acquisition of task
RAM memory errors?
• Entrance into previously entered arm• One foot? Two? Half-way? All the way?
Rats become quite good
• Rarely make a memory error• Win-shift strategy
Maybe they take an algorithmic strategy
• Right-right-right etc
To control for strategies..forced choice procedures
Using forced-choice procedures
• No indication of algorithm• Performance still very good
SO…
• What does rat performance in RAM show us?
Rival Hypothesis?
• Memory or sensory guidance?• Intra-maze or extramaze guidance?
• Spatial memory or…..?
Motivation-deficitsSensory-deficitsAttention-deficits