7/30/2019 Chapter5 Validity and Reliability
1/26
RELIABILITY AND VALIDITY
LOUIS COHEN, LAWRENCE MANION
& KEITH MORRISON
7/30/2019 Chapter5 Validity and Reliability
2/26
STRUCTURE OF THE CHAPTER
Defining validity Validity in quantitative research
Validity in qualitative research
Types of validity Triangulation
Ensuring validity
Reliability Reliability in quantitative research
Reliability in qualitative research
7/30/2019 Chapter5 Validity and Reliability
3/26
STRUCTURE OF THE CHAPTER
Validity and reliability in interviews
Validity and reliability in experiments
Validity and reliability in questionnaires
Validity and reliability in observations
Validity and reliability in tests
Validity and reliability in life histories
7/30/2019 Chapter5 Validity and Reliability
4/26
VALIDITY IN QUANTITATIVE AND
QUALITATIVE RESEARCH
Validity in quantitative research often
concerns: objectivity, generalizability,
replicability, predictability, controllability,homothetic statements.
Validity in qualitative research often
concerns: honesty, richness, authenticity,
depth, scope, subjectivity, strength of feeling,
catching uniqueness, idiographic statements.
7/30/2019 Chapter5 Validity and Reliability
5/26
Concurrent Consequential
Construct
Content Criterion-related
Convergent & discriminant
Cross-cultural Cultural validity
Descriptive
Catalytic
Ecological Evaluative
External
Face Internal
Interpretive
Jury Predictive
Systemic
Theoretical
TYPES OF VALIDITY
7/30/2019 Chapter5 Validity and Reliability
6/26
VALIDITY IN QUANTITATIVE RESEARCH
Concurrent
Construct
Content
Criterion-related
Convergent & discriminant
Cross-cultural
Evaluative
External
Face
Internal
Jury
Predictive
Theoretical
7/30/2019 Chapter5 Validity and Reliability
7/26
DIRECTION
OF CAUSALITY
MATURATION
TESTING
THREATS TOVALIDITY AND
RELIABILITY
TYPE 1 AND
TYPE 2
ERRORS
INSTRUMENT-
ATION
OPERATIONAL-
IZATION
REACTIVITY
HISTORY
EXPERIMENTAL
MORTALITY
CONTAMIN-
ATION
7/30/2019 Chapter5 Validity and Reliability
8/26
ESTABLISHING VALIDITY IN
QUALITATIVE RESEARCH Prolonged engagement inthe field
Persistent observation
Triangulation
Leaving an audit trail
Respondent validation
Weighting the evidence (giving priority)
Checking for representativeness
Checking for researcher effects Making contrast/comparisons
Theoretical sampling
Checking the meaning of outliers
Using extreme cases
7/30/2019 Chapter5 Validity and Reliability
9/26
ESTABLISHING VALIDITY IN
QUALITATIVE RESEARCH Ruling out spurious relations
Replicating a finding
Referential adequacy
Following up surprises
Structural relationships
Peer debriefing
Rich and thick description
Looking for possible sources of invalidity Assessing rival explanations
Negative case analysis
Confirmatory data analysis
Effect sizes
7/30/2019 Chapter5 Validity and Reliability
10/26
7/30/2019 Chapter5 Validity and Reliability
11/26
THREATS TO EXTERNAL VALIDITY
IN QUANTITATIVE RESEARCH
Failure to describe independent variables explicitly
Lack of representativeness of available and target
populations
Hawthorne effect
Inadequate operationalizing of dependent variables
Sensitization/reactivity to experimental/research conditions
Interaction effects of extraneous factors and experimental/
research treatments
Invalidity or unreliability of instruments
Ecological validity
Multiple treatment validity
7/30/2019 Chapter5 Validity and Reliability
12/26
THE HAWTHORNE EFFECT
Between 1927 and 1932 researchers carried out
experiments at the Western Electric Companys
Hawthorne plant.
Purposes: To examine the effects of changes of
working conditions on output of workers Sample: Six women, chosen as average workers
Method: Women worked in a test room. Output
measured under different conditions (e.g. no change
change to method of payment introduce tworest periods introduce six rest periods
changes in lighting conditions, early clocking-off,
five-day working week return to initial conditions
Duration: 15 weeks
7/30/2019 Chapter5 Validity and Reliability
13/26
THE HAWTHORNE EFFECT
Results: Output rose steadily during testperiod and after the test period.
Conclusion: Output did not seem to depend
on test conditions. Increased output seemedto be due to the fact that the people had been
involved in the experiment itself, i.e. the act of
research had affected the results. The
results were a research of the research itself. Implications: The act of being involved in
research itself affects the results.
7/30/2019 Chapter5 Validity and Reliability
14/26
THREATS TO EXTERNAL VALIDITY
IN QUALITATIVE RESEARCH Selection effects Setting effects
History effects
Construct effects
7/30/2019 Chapter5 Validity and Reliability
15/26
ENSURING VALIDITY AT THE
DESIGN STAGE
Choose an appropriate time scale;
Ensure adequate resources for the research
Select appropriate methodology Select appropriate instruments
Use an appropriate sample
Ensure reliability Select appropriate foci
Avoid having biased researcher(s)
7/30/2019 Chapter5 Validity and Reliability
16/26
ENSURING VALIDITY AT THE
DATA COLLECTION STAGE
Reduce the Hawthorne effect
Minimize reactivity
Avoid drop-out rates amongst respondents
Take steps to avoid non-return of questionnaires Avoid too long or too short an interval between pre-tests
and post-tests
Ensure inter-rater reliability
Match control and experimental groups Ensure standardized procedures for gathering data
Build on the motivations of respondents
Tailor instruments to situational factors
Address researchercharacteristics
7/30/2019 Chapter5 Validity and Reliability
17/26
ENSURING VALIDITY AT THE
DATA ANALYSIS STAGE
Use respondent validation; Avoid subjective interpretation of data
Reduce the halo effect
Use appropriate statistical treatments
Recognize extraneous factors which may affect data
Avoid poor coding of qualitative data
Avoid making inferences/generalizations beyond the data
Avoid equating correlations and causes Avoid selective use of data
Avoid unfair aggregation of data
Avoid degrading the data;
Avoid Type I and/or Type II errors
7/30/2019 Chapter5 Validity and Reliability
18/26
ENSURING VALIDITY AT THE
DATA REPORTING STAGE
Avoid using data selectively and unrepresentatively
Indicate the context and parameters of the
research
Present the data without misrepresenting the
message
Make claims which are sustainable by the data
Avoid inaccurate or wrong reporting of data Ensure that the research questions are answered
Release research results neither too soon nor too
late
7/30/2019 Chapter5 Validity and Reliability
19/26
RELIABILITY IN QUANTITATIVE
AND QUALITATIVE RESEARCH
Reliability in quantitative research:
consistency (stability), accuracy,predictability, equivalence, replicability,concurrence, descriptive and causalpotential.
Reliability in qualitative research:
accuracy, fairness, dependability,comprehensiveness, respondent validation,checkability, empathy, uniqueness,explanatory and descriptive potential,confirmability.
7/30/2019 Chapter5 Validity and Reliability
20/26
Reliability as stability:
Consistency over time and samples;
Reliability as equivalence:
Equivalent forms of same instrument;
Inter-rater reliability;
Reliability as internal consistency:
Split half reliability (e.g. for test items)
TYPES OF RELIABILITY IN
QUANTITATIVE RESEARCH
7/30/2019 Chapter5 Validity and Reliability
21/26
TRIANGULATION
Methodologies
Instruments
Researchers
Time Location
Theories
Samples Participants
Data
7/30/2019 Chapter5 Validity and Reliability
22/26
SPLIT-HALF RELIABILITY
(Spearman-Brown)Reliability =
r = the actual correlation between the two halvesof the instrument (e.g. 0.85);
Reliability = = = 0.919
r
r
12
85.01
)85.0(2
185
70.1
7/30/2019 Chapter5 Validity and Reliability
23/26
RELIABILITY IN QUALITATIVE
RESEARCH
Credibility
Neutrality
Confirmability Dependability
Consistency
Applicability Trustworthiness
Transferability
7/30/2019 Chapter5 Validity and Reliability
24/26
RELIABILITY AND REPLICATIONIN QUALITATIVE RESEARCH
Repeat: The status position of the researcher
The choice of informants/respondents
The social situations and conditions
The analytic constructs used
The methods of data collection and analysis
Address: Stability of observations
Parallel forms
Inter-rater reliability
Respondent validation
7/30/2019 Chapter5 Validity and Reliability
25/26
IMPROVING RELIABILITY
Minimise external sources of variation; Standardise conditions under which
measurement occurs;
Improve researcher consistency;
Broaden the sample of measurementquestions by:
a) adding similar questions to theinstrument;
b) increasing the number of researchers(triangulation);
c) increasing the number of occasions inan observational study.
Exclude extreme responses (outliers).
7/30/2019 Chapter5 Validity and Reliability
26/26
RELIABILITY AND VALIDITY AT
ALL STAGES Design and methodology
Sampling
Instrumentation
Timing
Data collection
Data analysis
Data reporting