Standard wording for formulating evidence
conclusions and implications for recommendations
Rode Kruis
Emmy De Buck
Manager Centre for Evidence-Based Practice
Belgian Red Cross-Flanders
I certify that, to the best of my knowledge, no aspect of my current personal or professional situation might reasonably be expected to affect significantly my views on the subject on which I am presenting, other than the following:
Financial relationships:
Employment at the Centre for Evidence-Based Practice of Belgian Red Cross-Flanders
Disclosure of interests
Centre for Evidence-Based Practice (CEBaP) of Belgian Red Cross-Flanders:
provides evidence-based support to all programmes and activities of Belgian Red Cross-Flanders, by developing:
− Practice guidelines
− Systematic reviews
Staff: 4 methodologists, PhD’s
Introduction
In our centre several methodologists (reviewers) work in parallel to develop evidence reviews in a timely way
The evidence conclusions go from the methodologists to other collaborators (colleagues of operational services, experts)
Important to standardize the evidence synthesis and to provide standard wording of the evidence conclusions/statements
Rationale
Information from guideline developers:
− NICE1: “A short evidence statement should be presented alongside the evidence profile, summarising the key features of the evidence on clinical and cost effectiveness”
− SIGN2: no narrative evidence conclusions
− Finnish Medical Society Duodecim3: wording varies depending on the level of evidence
Wording of evidence conclusions (1)
1 http://publications.nice.org.uk/the-guidelines- manual-pmg6/reviewing-the-evidence2 http://www.sign.ac.uk/methodology/index.html3http://2011.colloquium.cochrane.org/sites/2011.colloquium.cochrane.org/files/uploads/users/ u721/Wording%20of%20evidence%20and%20recommendations%20in%20EBM%20Guidelines.doc
Information from systematic review developers:
− Wide variation among authors in the Cochrane Library in reporting results and conclusions1
− The Cochrane Handbook, Chapter 9, Section 9.12:• “A narrative assessment of the evidence can be challenging...”
• “If a descriptive paragraph is provided for the results from each study, this should be done consistently, including the same elements of information for each study.”
• “Organizing the studies into groupings or clusters is encouraged (e.g. by intervention type, population groups, setting etc).”
Conclusion: almost no guidance about wording of evidence conclusions
Wording of evidence conclusions (2)
1 Van Tulder et al. 2003. SPINE 28(12): 290–12992 http://handbook.cochrane.org/chapter_9/9_1_introduction.htm
Target population for reading the evidence conclusions:
− colleagues who write draft recommendations− experts who validate the draft recommendations
Evidence conclusions should contain:
− number and type of studies− intervention, comparison− outcome− level of evidence− direction of effect
Criteria for evidence conclusions (1)
Wording of evidence conclusions should reflect:
− if results were statistically significant or not
− the quality of the evidence
− the difference between “evidence of no effect” and “no evidence of effect” 1
Evidence conclusions should be organized according to the intervention, outcome,…
Categories of evidence conclusions:− Evidence and limited evidence
− No evidence
− Conflicting evidence
Criteria for evidence conclusions (2)
1 Altman and Bland 1995. BMJ 311: 485
Y
Is there any evidence available?
Is there any imprecision?
N
Is the evidence conflicting?
N
Is the body of evidence of low or very low quality?
“No evidence”
Y
Y
N
“Conflicting evidence”
“Limited evidence”
Y “Limited evidence”
N
“Evidence”
Is the intervention considered as common sense?
Y
N
START
Is it animportant practical point
of which the expert panel reaches a consensus and nobody
is likely to question it?
Y
N
Is the balance made by the expert panel positive?
YRecommendation
N No Recommendation
Is the balance made by the expert panel positive?
Y
N
Is the balance made by the expert panel positive?
Y
N
Good Practice Point
Literature review Consensusmeeting with experts
Recommendation
Recommendation
No Recommendation
No Recommendation
No Good Practice Point
Flowchart
Category of evidence Wording of evidence conclusion
No evidence No relevant studies were identified using the above search strategy and criteria.
Conflicting evidence There is conflicting evidence from # experimental studies and/or # observational studies…
(Limited) evidence Wording depends on:
-statistical significance
-level of evidence (LOE)
-imprecision
See extended table
Standard wording
P-value LOE Imprecision Wording
< 0.05 A/B No There is evidence from # experimental studies and/or # observational studies in favour of [intervention] (<Author> <year>, <Author> <year>, etc.).
It was shown that <intervention> resulted in a statistically significant increase/decrease of <outcome>, compared to <comparison> (<Author> <year>).
Evidence is of high/moderate quality.
Yes There is limited evidence ... It was shown that ...
Evidence is of moderate quality and results cannot be considered precise due to limited sample size, lack of data and/or large variability of results.
C/D No There is limited evidence ... It was shown that ...Evidence is of low/very low quality.
Yes There is limited evidence ... It was shown that ...
Evidence is of low/very low quality and results cannot be considered precise due to ...
Standard wording
P-value LOE Imprecision Wording
> 0.05 A/B No There is evidence from # experimental studies and/or # observational studies, neither in favour of the intervention nor the control (<Author> <year>, <Author> <year>, etc.).It was shown that <intervention> did not result in a statistically significant difference of <outcome>, compared to <comparison> (<Author> <year>). Evidence is of high/moderate quality.
Yes There is limited evidence ...A statistically significant increase/decrease of <outcome>, using <intervention> compared to <comparison>, could not be demonstrated (<Author> <year>). Evidence is of moderate quality and results of this study/these studies are imprecise due to limited sample size, lack of data and/or large variability of results.
C/D No There is limited evidence ... It was shown that <intervention> did not result ...Evidence is of low/very low quality.
Yes There is limited evidence ... A statistically significant increase/decrease ... could not be demonstrated ...Evidence is of low/very low quality and results of this study/these studies are imprecise due to ...
Only in case of p<0.05, LOE A/B and no imprecision, a draft recommendation for or against the intervention should be included in the guideline
In the other cases: a draft recommendation may be included in the guideline
The final recommendation will be based on the judgement of the multidisciplinary expert panel, taking into account:
− the scientific evidence
− preferences of the target group
− expertise and practical experience
Implications for recommendations
PICO: Is rice water (I) effective compared to standard Oral Rehydration Solution (ORS) (C) to improve diarrhoea (O) in adults or children with diarrhoea (P)?
Included studies: 2 RCTs: Metha 1986, Fakhir 1990
Example
High [A] Downgrading due to
Limitations of study design
-1 Lack of blinding/blinding unclear; lack of allocation concealment in 1 study
Imprecision -1 Limited sample sizeInconsistency 0Indirectness 0Publication bias 0QUALITY (GRADE) Low [C]
Example
Outcome Comparison Effect Size #studies # parti-cipants
Ref
Stool frequency between 1 and 3 per day
Rice water versus glucose ORS
Day 1: Not statistically significant:4/100 vs 0/100; RR: 9.00, 95% CI [0.49, 165.00] (p=0.14)Day 2: Statistically significant:16/100 vs 6/100; RR: 2.67, 95% CI [1.09, 6.54] (p=0.03)Day 3: Statistically significant:46/100 vs 27/100; RR: 1.70, 95% CI [1.16, 2.51] (p=0.007)Day 4: Statistically significant:68/100 vs 42/100; RR: 1.62, 95% CI [1.24, 2.11] (p=0.0004)
1, 100 vs 100
Metha, 1986
Small stool volume Day 1: Not statistically significant:2/100 vs 0/100; RR: 5.00, 95% CI [0.24, 102.85] (p=0.30)Day 2: Not statistically significant:40/100 vs 30/100; RR: 1.33, 95% CI [0.91, 1.96] (p=0.14)Day 3: Not statistically significant:56/100 vs 42/100; RR: 1.33, 95% CI [1.00, 1.78] (p=0.05)Day 4: Statistically significant:76/100 vs 44/100; RR: 1.73, 95% CI [1.35, 2.21] (p<0.0001)
Duration of purging in the hospital (hours)
Statistically significant:60.2±2.6 vs 78.6±4.6MD: -18.40, 95%CI ; [-20.60;-16.20] (p=0.00001)
1, 20 vs 23
Fakhir, 1990
P-value LOE Imprecision Wording
< 0.05 A/B No There is evidence from # experimental studies and/or # observational studies in favour of [intervention] (<Author> <year>, <Author> <year>, etc.).
It was shown that <intervention> resulted in a statistically significant increase/decrease of <outcome>, compared to <comparison> (<Author> <year>).
Evidence is of high/moderate quality.
Yes There is limited evidence ... It was shown that ...
Evidence is of moderate quality and results cannot be considered precise due to limited sample size, lack of data and/or large variability of results.
C/D No There is limited evidence ... It was shown that ...Evidence is of low/very low quality.
Yes There is limited evidence ... It was shown that ...
Evidence is of low/very low quality and results cannot be considered precise due to ...
Example
PICO: Is rice water (I) effective compared to standard Oral Rehydration Solution (ORS) (C) to improve diarrhoea (O) in adults or children with diarrhoea (P)?
Evidence conclusion:
There is limited evidence from 2 experimental studies in favour of rice water (Metha 1986, Fakhir 1990).
It was shown that rice water resulted in a statistically significant decrease of duration of purging, stool frequency on day 2 to 4 and stool volume on day 4 compared to using standard ORS.
Evidence is of low quality and results cannot be considered precise due to a limited sample size.
Example
The use of standard wording in our evidence conclusions resulted in:
− standardized training of new staff members
− guidelines with uniform evidence statements
− a better interpretation of the available evidence
− more transparency in our methodology
Recommendation for guideline developers:
− use standard wording of evidence conclusions, in order to improve consistency within one guideline and between guidelines
Take home message
Thanks to my colleagues Tessa, Nele and Hans for the fruitful discussions concerning this subject.
Acknowledgements
Thank you for your attention ! Questions ?
Contact: [email protected] [email protected]
Centre for Evidence-Based Practice