Date post: | 13-Jan-2016 |
Category: |
Documents |
Upload: | paulina-singleton |
View: | 221 times |
Download: | 0 times |
Lecture 6: Reliability and validity of scales (cont)
1. In relation to scales, define the following terms:
- Content validity
- Criterion validity (concurrent and predictive)
- Construct validity (discriminant and convergent)
- Responsiveness
2. Use of proxy respondents
Validity
• Depends on purpose:– screening: discrimination– outcome of treatment: responsiveness,
sensitivity to change– prognosis: predictive validity
Content and face validity
• Judgment of “experts” and/or members of target population
• Does measure adequately sample domain being measured?
• Does it appear to measure what it is intended to measure? (eyeball test)
Criterion validity
• Criterion (“gold” standard)
• Concurrent criterion validity– e.g., screening test vs diagnostic test
• Predictive criterion validity – e.g., cancer staging test vs 5-year survival
Construct validity
• Is the theoretical construct underlying the measure valid?
• Development and testing of hypotheses
• Requires multiple data sources and investigations:– Convergent validity: measure is correlated with
other measures of similar constructs
– discriminant validity: measure is not correlated with measures of different constructs
Responsiveness of measures• Ability to detect clinically important change
over time or differences between treatments
• Requirement of evaluative measures
• Two approaches:– external responsiveness (validation against
change in external criterion)– internal responsiveness: compute effect size of
ratio of change score to measure of variability (different formulae)
Validity of Delirium Index (DI)
• Convergent validity – DI will be correlated with measures of current
health/function:• Current ADL disability (Barthel Index)
• Current severity of illness (clinical judgment and physiological severity score from APACHE)
Validity of Delirium Index (DI)• Discriminant validity:
– DI will be weakly correlated with measures of previous health/function:
• premorbid level of ADL disability
• severity of dementia
• comorbidity (prior)
Validity of DI (cont)
• DI score will predict probability of one-year survival
• DI score will be sensitive to:– changes in medication exposures – changes in environmental factors
Spearman correlation coefficients between Delirium Index and 3 baseline measures of current status
0
0.1
0.2
0.3
0.4
0.5
Barthel Index Clinicalseverity
Physiologicalseverity
Delirium+Dementia (n=165)
Delirium (n=57)
Spearman correlation coefficients between Delirium Index and 3 baseline measures of prior status
0
0.1
0.2
0.3
0.4
0.5
0.6
IADL IQCODE Comorbidity
Delirium+dementia (n=165)
Delirium (n=57)
Delirium severity and survival
• Proportional hazards regression of delirium severity in delirium cohort
• Mean of 1st 2 DI scores
• Results– significant interaction: DI predicted survival in
patients with delirium alone, not in those with dementia
Effects of medications and environmental factors
• Repeated in-hospital measures of DI, medications (medical record), and environmental factors (using checklist)
• Repeated measurements analysis performed to investigate associations of changes in risk factors with change in DI score.
Environmental risk factors for delirium
0 1 2 3 4 5
Cumulative roomchanges (one)
Medical restraint
No clock/watch
Family memberpresent
No reading glasses
Physical restraints
ICU vs medicine
mean DI increase
Source: McCusker et al, JAGS 2001, 49:1327-34
Medications and DI
• Medication changes (increase in # medications, use of anticholinergic medications) were associated with increase in DI score.
Responsiveness of DI
• Internal responsiveness: ability to capture real change over time– effect size (change in mean DI/SD at baseline)– standardized response mean (change in mean
DI/SD of change)
• External responsiveness: are changes in the DI correlated with other measures of change?
Internal responsiveness of the Delirium Index (DI) at 8 weeks
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Effect size Standardized response mean
Delirium+dementia (n=94)
Delirium (n=33)
Validity of DI: Conclusions
• In patients with delirium +/- dementia:– Correlated with measures of current function– Responsive to change over time– Responsive to changes in environmental factors
and medications
• In patients with delirium alone:– Predicts survival
Example: Inflammatory Bowel Disease Questionnaire (IBDQ)
• Disease-specific HRQoL measure
• 30 items covering 4 domains– bowel symptoms– systemic symptoms– emotional function– social function
• How to validate?
Validation of Inflammatory Bowel Disease Questionnaire
• Questionnaires administered to 42 patients with inflammatory bowel disease on 2 occasions, 1 month apart:– IBDQ– Global ratings of function– Global ratings of change by physician and a relative– Disease Activity Index– Emotional function scale from generic HRQoL
instrument
Validation of IBDQ: predictions and results
• Correlation of 0.5 or more between patients’s global rating of change and IBDQ subscale on bowel symptoms (result: 0.42)
• Correlation of 0.3 or more between change in Disease Activity Index and IBDQ subscale on bowel symptoms (result: 0.33)
• Correlation of 0.3 or more between change in Disease Activity Index and IBDQ subscale on systemic symptoms (result: 0.04)
• Correlation of 0.5 or more between change in generic measure of emotional function and IBDQ subscale on emotional function (result: 0.76)