Date post: | 19-Jun-2015 |
Category: |
Health & Medicine |
Upload: | carlos-cuello |
View: | 590 times |
Download: | 0 times |
Case study
From evidence to decisions
Frameworks for Clinical Guidelines’ Development
Carlos A. Cuello, MD, PhD(c) Health Research Methodology Program!Department of Clinical Epidemiology and Biostatistics
March 27, 2014
Disclosure
Authors belong to the CE&B department and
to the GRADE / DECIDE working group.
No other conflicts to declare.
Usefulness of the Evidence to DECISION table
for clinical practice guidelines: a case study
with a panel of international experts
Carlos Cuello-Garcia, Jan Brozek, Juan José Yepes-Nuñez, Yuan Zhang, Shreyas Gandhi, Arnav Agarwal, Holger Schünemann
Clinical Epidemiology and Biostatistics McMaster University MacGRADE Center
What are these frameworks?
How can they help?
Why should we use them?
decision
(recommendation)
research
evidence
benefits vs
harms
resource use
patient
values
equity
feasible?
acceptable?
Clinical Guideline
FRAMEWORKS
evidence to decision (EtD)
just 4 columns
& conclusions
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility
JUDGEMENTSRESEARCH EVIDENCE
ADDITIONAL INFORMATION
JUDGEMENTSRESEARCH EVIDENCE
ADDITIONAL INFORMATION
������������� ����������
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility
JUDGEMENTSRESEARCH EVIDENCE
ADDITIONAL INFORMATION
������������������������������������ ������������
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility
�����
���������������� ���
JUDGEMENTSRESEARCH EVIDENCE
ADDITIONAL INFORMATION
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility
��������������
JUDGEMENTSRESEARCH EVIDENCE
ADDITIONAL INFORMATION
�� �����������
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility
CONCLUSIONS
Balance of consequences
Decision /recommendation
Justification
Implementation considerations
Monitoring
Evaluation
Research priorities
���������� …
PURPOSE OF THE FRAMEWORK
PROS CONS
Inform decision makers’ judgements about the pros and
cons of each option (intervention) that is considered.
The important factors that determine a decision (criteria) are considered.
Structure discussion and manage disagreements
Make the basis for decisions transparent to guideline users
Provide a concise summary of the best available research evidence to inform judgements about each
criterion
A case study . . .
Guideline for Allergic Diseases Prevention
INTERNATIONAL PANEL
guidelines on allergy prevention
• Probiotics
• Prebiotics
• Vitamin D
guidelines on allergy prevention
• Probiotics
• Prebiotics
• Vitamin D
Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Generic EtD framework 1
Evidence to decision framework
Question 1: Should probiotics vs. no probiotics be used in pregnant women?
Population:!pregnant!women!
Option:!probiotics!Comparison:!no!probiotics!Setting:3outpatient!Perspective:3individual!patient
Background: The$intestinal$microbiome$could$play$an$important$role$in$the$immune$system$maturation,$and$it$has$been$suggested$that$early6life$probiotic$administration,$whether$directly$to$the$infant$or$in$their$mothers$breast$milk,$may$reduce$the$risk$of$allergies$in$childhood.$The$objective$of$this$question$is$to$evaluate$the$impact$of$probiotics$administered$to$the$expecting$mothers$on$their$infant.
Subgroup considerations: subpopulation of women at high risk for allergy in a child
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION
PR
OB
LEM
Is the problem a priority?
No Probably no Uncertain Probably yes Yes
X
Allergic diseases represent a spectrum of health conditions and a worldwide burden in different populations. (1)
Are a large number of people affected?
No Probably no Uncertain Probably yes Yes
X
As many as 40% of the worldwide population is affected by any type of allergy. In infants prevalence depends highly on the allergic status of their parents, being approximately of 10% in those without an allergic parent or sibling, versus 20% to 30% in those with the atopic background in their relatives. (2)
Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Generic EtD framework 2
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS
VA
LUE
S
Is there important uncertainty or variability about how much people value the main outcomes?
Important uncertainty
or variability
Possibly important
uncertainty or
variability
Probably no
important uncertainty
or variability
No important
uncertainty or
variability
No known undesirable outcomes
X
Detailed judgements
The relative importance or values of the main outcomes of interest:
Outcome Relative importance Certainty of the evidence Eczema critical low
Asthma/wheezing critical low
Food allergy critical low
Adverse effects critical low
We judged that the outcomes eczema, asthma and food allergy are critical for people. The adverse outcomes are probably of high importance and the burden of taking daily pills is limited. Some immunocompromised women might not accept the risk.
BE
NE
FIT
S &
HA
RM
S O
F T
HE
OP
TIO
NS
What is the overall certainty of the evidence of effectiveness?
No included studies Very low Low Moderate High
X
Summary of findings:
Outcome With [intervention]
Without [intervention]
Difference (per 100) (95%CI)
Relative effect (RR)
(95%CI)
Certainty of the
evidence (GRADE)
Eczema (follow-up 1 to 5 years)
365/1520 (24%)
484/1515 (31.9%)
9 fewer per 100 (from 4 fewer to 13
fewer)
RR 0.72 (0.6 to 0.86)
⊕⊝⊝⊝ VERY LOW
Asthma/wheezing (follow-up 2 to 7 years)
143/992 (14.4%)
139/982 (14.2%)
0 fewer per 100 (from 3 fewer to 3
more)
RR 0.97 (0.77 to 1.22)
⊕⊕⊝⊝ LOW
Food allergy (follow-up 1 to 2 years)
36/279 (12.9%)
41/284 (14.4%)
1 more per 100 (from 3 fewer to 8
more)
RR 1.08 (0.73 to 1.59)
⊕⊝⊝⊝ VERY LOW
Adverse effects 101/394 (25.6%)
88/397 (22.2%)
3 more per 100 (from 4 fewer to 12
more)
RR 1.13 (0.82 to 1.52)
⊕⊝⊝⊝ VERY LOW
Link to detailed evidence profile Subgroup considerations: Link(s) to summary of findings and judgments for subgroups
The data are indirect for all outcomes because they are primarily derived from studies that looked at mixed exposure in women during pregnancy and breastfeeding and of infants after birth. Only 1 RCT assessed the effect on eczema in pregnant women only: RR 0.88 (0.63 to 1.22); RD 5 fewer per 100 (from 14 fewer to 9 more) 5 RCTs included pregnant women + later breastfeeding mothers: RR 0.5 (0.4 to 0.63); RD 21 fewer per 100 (from 15 fewer to 25 fewer) 5 RCTs included pregnant women + infants after birth (follow-up 1 to 5 years): RR 0.87 (0.72 to 1.04), RD 4 fewer per 100 (from 8 fewer to 1 more) 3 RCTs included pregnant women + subsequently breastfeeding + infants (follow-up 3 to 4 years): RR 0.78 (0.49 to 1.24); RD 7 fewer per 100 (from 17 fewer to 8 more) No effects were observed on asthma/wheezing and food allergy.
How substantial are the desirable anticipated effects?
Don’t know
Not important
Somewhat important
Moderately important
Very important
Varies
X
X
Detailed judgements
There was some disagreement among panel members whether the effect is somewhat or moderately important.
How substantial are the undesirable anticipated effects?
Don’t know
Very important
Moderately important
Somewhat important
Not important
Varies
X
Detailed judgements
No serious adverse effects, and no difference in mild adverse effects between the groups.
Do the desirable effects outweigh the undesirable effects?
No Probably No
Don’t know Probably Yes
Yes Varies
X
Detailed judgements
Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Generic EtD framework 2
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS
VA
LUE
S
Is there important uncertainty or variability about how much people value the main outcomes?
Important uncertainty
or variability
Possibly important
uncertainty or
variability
Probably no
important uncertainty
or variability
No important
uncertainty or
variability
No known undesirable outcomes
X
Detailed judgements
The relative importance or values of the main outcomes of interest:
Outcome Relative importance Certainty of the evidence Eczema critical low
Asthma/wheezing critical low
Food allergy critical low
Adverse effects critical low
We judged that the outcomes eczema, asthma and food allergy are critical for people. The adverse outcomes are probably of high importance and the burden of taking daily pills is limited. Some immunocompromised women might not accept the risk.
BE
NE
FIT
S &
HA
RM
S O
F T
HE
OP
TIO
NS
What is the overall certainty of the evidence of effectiveness?
No included studies Very low Low Moderate High
X
Summary of findings:
Outcome With [intervention]
Without [intervention]
Difference (per 100) (95%CI)
Relative effect (RR)
(95%CI)
Certainty of the
evidence (GRADE)
Eczema (follow-up 1 to 5 years)
365/1520 (24%)
484/1515 (31.9%)
9 fewer per 100 (from 4 fewer to 13
fewer)
RR 0.72 (0.6 to 0.86)
⊕⊝⊝⊝ VERY LOW
Asthma/wheezing (follow-up 2 to 7 years)
143/992 (14.4%)
139/982 (14.2%)
0 fewer per 100 (from 3 fewer to 3
more)
RR 0.97 (0.77 to 1.22)
⊕⊕⊝⊝ LOW
Food allergy (follow-up 1 to 2 years)
36/279 (12.9%)
41/284 (14.4%)
1 more per 100 (from 3 fewer to 8
more)
RR 1.08 (0.73 to 1.59)
⊕⊝⊝⊝ VERY LOW
Adverse effects 101/394 (25.6%)
88/397 (22.2%)
3 more per 100 (from 4 fewer to 12
more)
RR 1.13 (0.82 to 1.52)
⊕⊝⊝⊝ VERY LOW
Link to detailed evidence profile Subgroup considerations: Link(s) to summary of findings and judgments for subgroups
The data are indirect for all outcomes because they are primarily derived from studies that looked at mixed exposure in women during pregnancy and breastfeeding and of infants after birth. Only 1 RCT assessed the effect on eczema in pregnant women only: RR 0.88 (0.63 to 1.22); RD 5 fewer per 100 (from 14 fewer to 9 more) 5 RCTs included pregnant women + later breastfeeding mothers: RR 0.5 (0.4 to 0.63); RD 21 fewer per 100 (from 15 fewer to 25 fewer) 5 RCTs included pregnant women + infants after birth (follow-up 1 to 5 years): RR 0.87 (0.72 to 1.04), RD 4 fewer per 100 (from 8 fewer to 1 more) 3 RCTs included pregnant women + subsequently breastfeeding + infants (follow-up 3 to 4 years): RR 0.78 (0.49 to 1.24); RD 7 fewer per 100 (from 17 fewer to 8 more) No effects were observed on asthma/wheezing and food allergy.
How substantial are the desirable anticipated effects?
Don’t know
Not important
Somewhat important
Moderately important
Very important
Varies
X
X
Detailed judgements
There was some disagreement among panel members whether the effect is somewhat or moderately important.
How substantial are the undesirable anticipated effects?
Don’t know
Very important
Moderately important
Somewhat important
Not important
Varies
X
Detailed judgements
No serious adverse effects, and no difference in mild adverse effects between the groups.
Do the desirable effects outweigh the undesirable effects?
No Probably No
Don’t know Probably Yes
Yes Varies
X
Detailed judgements
Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Generic EtD framework 3
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS
RE
SO
UR
CE
US
E
How large are the resource requirements?
Large costs
Moderate costs
Small Moderate savings
Large savings
Varies
X
Detailed judgements
Prices are likely to vary substantially depending on the setting. This may be a particularly important consideration in low and middle-income countries. A level and type of insurance may play a substantial role as well. From a health systems point of view it might also be cost effective given that probiotic would be used for 9 months and cost of treatment of eczema may be distributed across many years.
Extremely limited research evidence (internet searches of drug prices)
Bifidobacterium bifidum (cost per person per year US$) Dose: 1 pill
each day
Lactobacillus gg (cost per person per
year US$) 1 pill each day
North-America
Average $181.16 $341.6
South-America
Average $174.3 $286
Europe
Average $167.86 $251.56
Fewer office visits would occur as a result of eczema if the effects on eczema were true.
How large is the incremental cost relative to the net benefit?
Very large ICER
Large ICER
Moderate ICER
Small ICER
Savings Varies
Detailed judgements
No research evidence
If eczema was reduced the intervention might be cost-effective given fewer office visits (between $17,400 and $34,100 to treat 100 people for 1 year or ¾ of that for 9 months) preventing 9 cases of eczema. In most studies probiotics were used in the last trimester of pregnancy, which, if used this same way, might reduce the cost per pregnant woman.
EQ
UIT
Y What would
be the impact on health inequities?
Increased Probably increased
Uncertain Probably reduced
Reduced Varies
X
Detailed judgements
No research evidence
In some settings it may be important to consider equity as the access may depend on socioeconomic status of the country or setting where coverage will depend on policymakers.
AC
CE
PT
AB
ILIT
Y
Is the option acceptable to key stakeholders?
No Probably No
Uncertain Probably Yes
Yes Varies
X
Detailed judgements
No research evidence
Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Generic EtD framework 4
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS
FE
AS
IBIL
ITY
Is the option feasible to implement?
No Probably No
Uncertain Probably Yes
Yes Varies
X
Detailed judgements
No research evidence
Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
Generic EtD framework 5
Recommendation Should probiotics vs. no probiotics be used in pregnant women (exposing their children in utero)?
Overall balance of consequences Undesirable consequences clearly outweigh desirable
consequences
Undesirable consequences probably outweigh desirable
consequences
The balance of desirable and undesirable consequences indicates
they are very similar*
Desirable consequences probably outweigh undesirable
consequences
Desirable consequences
clearly outweigh undesirable
consequences
! ! ! X !
We recommend against the option or for the alternative
We suggest not to use the option or to use the alternative
We suggest using the option We recommend the option
! ! X !
Panel decisions 3 panel members with potential COI recused themselves from participating in formulating the recommendation. Consensus was obtained from the rest of the team.
Recommendation (text) The guideline panel suggests using probiotics in pregnant women at high risk for allergy in their children (conditional recommendation, very low quality evidence).
Remarks and justification Most studies commenced probiotics in the last trimester of pregnancy. The very low quality evidence for adverse effects indicates that our confidence in the absence of increased adverse effects is low. Future research is needed (see definitions of very low quality) e.g., generalizing to immune-compromised children
Subgroup considerations Women with high risk of allergy in their children Women with average risk of allergy in their children
Implementation considerations This recommendation is based on trials investigating the probiotics or mixtures of probiotic listed below. We have not found a difference between these different probiotics, but that does not mean there is no difference
Monitoring and evaluation considerations
Research priorities Develop instruments for evaluating the risk of allergy in children as the family history predicts only about 30% of the population risk. There is some evidence that first child is at higher risk for allergy than subsequent children. Long-term follow-up of long-term effects. No direct evidence for the use of probiotics in formula – this should be evaluated in future research and is an unmet need.
Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
SURVEY
24 panel members
13 RESPONDED
11 ACCEPTED TO COMPLETE THE
SURVEY
1. The EtD table helped in the ORGANIZATION of the development process (i.e., for getting a recommendation from a body of evidence)
Strongly disagree
DisagreeSomewhat disagree
neutralSomewhat
agreeAgree Strongly agree
2. The EtD is a structured process that is BETTER than an informal/unstructured discussion
Strongly disagree
DisagreeSomewhat disagree
neutralSomewhat
agreeAgree Strongly agree
3. The EtD helped you UNDERSTAND the process of getting a recommendation from a body of evidence
Strongly disagree
DisagreeSomewhat disagree
neutralSomewhat
agreeAgree Strongly agree
4. The EtD made you think about concepts like patient values & preferences, costs & risks of the intervention, equity, acceptance, and feasibility of the intervention
Strongly disagree
DisagreeSomewhat disagree
neutralSomewhat
agreeAgree Strongly agree
– Anonymous panel member
“It made the process smooth, easy to understand and significantly reduced the time it would have otherwise taken to complete”
What are these frameworks?
How can they help?
Why should we use them?