Post on 11-Feb-2016
description
transcript
1
Lecture 7: Evaluation of interventions
• Types of intervention • Introduction to social science terminology and
concepts of intervention study design• Study design
– Experimental– Quasi-experimental – Observational
2
Requirements of health care • Effective
– effectiveness vs efficacy?• Efficient
– minimize use of resources • Equitable
– equity in access, use related to need• Acceptable
– client perception of care
3
Efficacy vs effectiveness(Definitions from Last’s Dictionary of Epidemiology)
• Efficacy (Can it work?) The extent to which a specific intervention procedure, regimen or service produces a beneficial result under ideal conditions. Ideally, the determination of efficacy is based on the results of a randomized controlled trial.
• Effectiveness (Does it work?): The extent to which a specific intervention procedure regimen or service when deployed in the field does what it is intended to do for a defined population. (The main distinction between effectiveness and efficacy is that effectiveness refers to average rather than ideal conditions of use).
4
Types of intervention
• Classified by purpose:– primary prevention (prevention of onset of
disease)– secondary prevention (screening, early
detection, and prompt treatment)– tertiary prevention (of chronic conditions, to
decrease disability and increase quality of life)
5
Types of intervention
• Classified by complexity of technology involved (technology assessment paradigm):– drugs– devices– procedures– systems of care
6
Intervention study or study of an intervention?
• Intervention study (referring to a study design): An investigation involving intentional change in some aspect of the status of the subjects, e.g., introduction of a preventive or therapeutic regimen, or designed to test a hypothesized relationship; usually an experiment such as a randomized controlled trial (Definitions from Last’s Dictionary of Epidemiology)
• Study of an intervention (referring to the study purpose): study of a health care intervention; may be experimental or non-experimental (observational)
7
Level of evaluation
• STRUCTURE: Staff, equipment needed to deliver intervention.
• PROCESS: is the intervention service provided as planned? (Interaction between structure and patient/client)
• OUTCOMES: expected or unexpected results, either positive or negative.
8
Level of evaluation
• In evaluation of intervention, outcomes are of primary interest
• To help interpret the results, measures of structure and process are desirable, e.g.:– adherence to intervention– “dose” of intervention actually received – characteristics of staff who deliver intervention
9
Step 1: intervention objectives
• Specify positive and negative outcomes expected
• Measurable outcomes– Changes in natural history
• death, disease, disability, distress– Behaviors, attitudes (e.g., educational
interventions)
10
Methodological issues in evaluation of interventions
• Two paradigms:– epidemiological (clinical and public health
roots)– social science (sociological roots)
• Two sets of terminology!
11
Internal and external validity of an intervention study
• Internal validity: The degree to which an observed effect can be attributed to an intervention.
• External validity: The degree to which an observed effect that is attributable to an intervention can be generalized to similar populations and settings (generalizability). Note: both internal and external validity are aspects of the validity of a study and should be distinguished from the validity of measurements.
12
Threats to internal validity • History
– extraneous events (e.g. breast cancer screening)• Maturation
– aging (e.g., drug abuse treatment)• Testing
– e.g., effects of pretesting • Instrumentation• Regression (to mean)• Selection• Attrition
13
Threats to external validity
• Is intervention equally effective in different populations, including more naturalistic applications? Usually not - why?:– Methodological
• Interaction of intervention with pre-testing• Reactive effects (to testing) - Hawthorne effects
– Differences in intervention • Characteristics of intervention personnel• Process of implementation
14
Study designs
• Experimental– investigator has complete control over allocation
and timing of intervention – usually randomized
• Quasi-experimental– investigator has no control
• Observational– investigator has no control
15
Diagramming Intervention Evaluation Designs
Campbell and Stanley
• X = program
• O = measurement
• R = randomization
16
Randomized (Experimental) Designs
• Randomized pre-test post-test control group design
R O1 X O2
R O3 O4
• Post-test only control group designR X O1
R O2
17
Quasi-experimental study designs
• Investigator has “some control” over timing or allocation of intervention – Non-randomized or quasi-randomized trials– Non-equivalent control group designs (MAY
OR MAY NOT BE RANDOMIZED):• pre-test and post-test• post-test only• Solomon 4 group
18
Some quasi-experimental designs
Pre-test post-test non-equivalent controlgroup design
O1 X O2O3 O4
Recurrent institutional cycle
X O1 O2 X O3
19
Solomon four-group design
R O1 X O2
R O3 O4
R X O5
R O6
20
Examples of pre-post non-equivalent control group design
• Stanford 5-city study of CHD prevention• Intervention included mass media education
and group interventions for high-risk• 5 cities selected - similar characteristics
– those with shared media market were allocated to intervention
– isolated cities allocated to control group
21
Other designs: recurrent institutional cycle design
• Finnish mental hospital study of dietary intervention to prevent CHD
• 2 hospitals selected, received intervention sequentially
• Useful design if considered unethical to withhold intervention
22
Observational designs
• Investigator has NO control over allocation or timing of intervention: – Cross-sectional (after only)– Separate sample pre- post-test– Time series (trend) designs
– single or multiple
– Cohort studies– Panel studies
23
Example of trend study:Health insurance in Quebec
• 1961: universal hospital insurance– included ER care for accidents
• 1970: universal health insurance (Medicare) – added MD care including hospital outpatient
clinics and ERs
24
Example of trend study:Health insurance in Quebec
• Population surveys before and after• Effects on:
– use of physician services by general population – physician workload– use of emergency rooms– hospitalization and surgery
25
MD visits/person/year by income(household surveys)
0
1
2
3
4
5
6
7
8
All visits <3000 3000- 5000- 9000- 15000+
PrePost
26
MD visits/person/year (household surveys)
0
1
2
3
4
5
6
All visits Office ODP/ER Home
PrePost
27
MD visits/person/year by income(household surveys)
0
1
2
3
4
5
6
7
8
All visits <3000 3000- 5000- 9000- 15000+
PrePost
28
% adults with cough 2+ weeks who consulted MD (household surveys)
0
10
20
30
40
50
60
70
<$5000 $5000- $9,000 Total
PrePost
29
% children (<17) with tonsilitis or sore throat and fever who consulted MD
(household surveys)
0
10
20
30
40
50
60
70
80
<$5000 $5000- $9,000 Total
PrePost
30
% pregnancies with visit in first trimester (household survey)
0
10
20
30
40
50
60
<$5000 $5000- $9,000 Total
PrePost
31
% Tried to contact MD before ED visit; of these, % successful (6 hospital sample)
010203040506070
Trie
d to
con
tact
Spok
e to
MD
Offi
ce/a
nsw
erin
gm
achi
ne
Uns
ucce
ssfu
l
PrePost
32
Time series designs
Time series desgn
O1 02 O3 X O4 O5 O6
Multiple time series design
O1 O 2 O 3 X O 4 O 5 O 6
O7 O8 O9 O10 O11 O12
33
Example of time series study:Tamblyn et al, 2001
• Evaluation of prescription drug cost-sharing among poor and elderly
• Methods:– Trend study: Multiple pre- and post-
measurements– Cohort study:
34Source: Tamblyn et al, JAMA 2001, 285(4): 421-429
35
Source: Tamblyn et al, JAMA 2001, 285(4): 421-429
36
Some Weak Observational Designs
• One-shot case-study
X O
• Static group comparison:
X O1
O3
37
Time-series design: Home care in terminal cancer
• Evaluation of home-hospice programme in Rochester, NY
• Expansion of home-care benefits in 1978• Hypothesis: home-hospice care in last month
of life reduces hospital days and costs• Data sources: Linkage of tumor registry and
health insurance claims databases
38
39
40
Epidemiological observational analytical designs
• Difference in independent and dependent variables:– Studies of risk factors:
• independent variable: risk factor • dependent variable: disease
– Studies of interventions:• independent variable: intervention • dependent variable: outcome
41
Cohort study
• Selection of controls: could they receive either treatment?
• Example: medical vs surgical treatment of CHD
• Sources of bias:– confounding by indication– selection bias– detection bias (etc.)
42
Cohort study
• Cohorts with and without “exposure” (intervention) followed to determine outcomes
• Control cohort - concurrent or historical (confounding by changes over tine in patient population, aspects of treatment other than intervention; measurement of confounders)
43
Example of cohort study• Do HMOs reduce hospitalization in terminal
cancer patients, during 6 months before death?• Administrative databases and tumor registry
from Rochester NY• Cancer deaths in 100 pairs of HMO members
and non-members • Matched by age, cancer site, months from
diagnosis to death
44
45
Case-control study
• Cases (with outcome) compared to controls (without outcome) with regard to (previous) intervention
• Limited to single, categorical outcome• Sources of bias
– Confounding by selection– Confounding by indication– Detection bias– (For screening programs) Separation of screening tests
from tests done after symptoms appear
46
Case-control study: Examples
• Screening programs:– screening Pap test and invasive cervical cancer– screening mammography and breast cancer
deaths– screening sigmoidoscopy and colon cancer
deaths • Vaccine effectiveness (e.g., BCG)• Neonatal intensive care and neonatal deaths
47
Considerations in selection of a study design
• Cost• Feasibility• Ethical issues• Internal validity• External validity • Credibility