16
Centre for Evidence Based Intervention Evidence from Review Groups
Centre for Evidence Based Intervention
Evidence from Review Groups
Objectives
Identify policies & procedures on…
Determining publication of empty reviews
Practice of handling excluded studies across Cochrane
and other providers
Informing practice based on empty reviews and
excluded studies
Methods
Survey with closed and open-ended questions
53 Cochrane Collaboration Review Groups
WHO, JBI, What Works Clearinghouse and SCIE
Coded & analyzed responses
Why might a review be empty?
“because it is a new or recently introduced intervention
which has yet to gather strength of evidence”
“the intervention does not lend itself to randomised
controlled trials”
“poor question?”
……..
Cochrane Groups publishing empty reviews?
N=53
Rationale for publishing Empty Reviews
Page 6
N=46
Other providers
WHO - doesn't have a policy on empty reviews. In a way it isn't relevant - they would search for the best available evidence, but would need to make a recommendation regardless of what they find.
SCIE - As the evidence base in social care is very limited, we may often find that a review with inclusion criteria specifying high quality controlled (and/or randomised) studies of specific interventions would be ‘empty’ – hence we would not commission a review.
WWCH - review focus is changed depending on the evidence base, thus empty reviews are not generally produced.
Views about including empty reviews Include to instigate further research
“Our aim with empty reviews is to raise the profile of the question and hopefully generate research leading to high quality evidence which we can then report quickly in an updated review.”
Include if good case made “There may be benefit in amending the TRF form to include
the line: ‘If you think this review might be empty, please give your reasons why you feel it is important to do the review.’”
Reasons for not publishing empty reviews
(Some groups have minimum # of studies stipulation - e.g., 2-3)Not worth time and resources
“a colossal waste of everyone’s time and energy”
Influence on group’s impact factorMay be misleading - Empty reviews (< 3 studies) can put positive supportive evidence from initial, low quality studies behind treatmentBUT, question can still matter
“may be exceptions where review questions appear of great public health relevance”
Handling excluded studies
Standard policy? Permitted in review?
Conditions for providing evidence based on excluded studies N=31
Further considerations for whether to allow excluded studies Excluded based on intervention or outcomes?
RCTs feasible/ethical for topic area?
If not, lower grade studies may be acceptable
Guidelines may be needed for application of evidence from
excluded studies
“We dissuade authors from providing evidence from excluded studies
because of inconsistency, which would lead to provision of evidence
that is arbitrary and unsystematic.”
Guidance for ‘Implications for practice’ section
For ‘Yes’ responses:
(n=7)
Extra guidance but on case-by-case
basis (3)
General advice (2)
Refer to other empty reviews, EPICOT,
& extra advice (1)
Group Methodologists give guidance (1)N=53
Aim to update or withdraw empty
reviews faster than others?
For ‘Yes’ responses:
(n=4)
No reason given (2)
If there are recently published RCTs
(2)N=53
Views about priority on updating empty reviews
No, topic-based priority only “Priority is only given on the basis of importance of the review topic, not on
whether there are included studies or not.”
Only if new study emerges “Aim to update/review them around the same time as other reviews , but if
we became aware of a study relevant to an empty review, it would probably prompt a more rapid update.”
