Running on Empty · frustration in locating a review only to find that it has no included studies...

Running on Empty: The Cochrane Empty Reviews Project report of

findings and consensus group feedback

Paul Montgomery, University of Oxford Joanne Yaffe, University of Utah

Sally Hopewell, UK Cochrane Centre Lindsay Shepard, University of Utah

The Empty Reviews Project is funded by the Cochrane Opportunities Fund

Empty Reviews Project Group

Consensus Meeting Group

Julian Higgins Cochrane Handbook Cambridge, UK

David Moher Consensus Meetings/Guideline Design Ontario, Canada

David Tovey Cochrane Editorial Unit London, UK

Geraldine MacDonald DPLP Group – High Nos. of Empty Reviews Belfast, NI

Samantha Faulkner ENT Group – Policy on Empty Reviews Oxford, UK

Catherine McIlwain Consumers Oxford, UK

Iain Chalmers Author of Empty Reviews Oxford, UK

Sasha Shepperd EPOC Group – Author Oxford, UK

Apologies

Claire Williams Heart Group – Low No. of Empty Reviews London, UK

Barney Reeves Non Randomised Studies Methods Group Bristol, UK

Empty Reviews Project Core Team

Jo Yaffe Utah, USA

Paul Montgomery Oxford, UK

Sally Hopewell Oxford, UK

Lindsay Shepard Utah, USA

The Cochrane Empty Reviews Project

[email protected]

Summary of Session

• Presentation

– Introduction to the issue of empty reviews in The CDSR

–Update on the project to date

–Report on the consensus meeting

• Feedback and interactive discussion to move the project forward with your views


[email protected]

Definition

• An “empty review” is one that reports no studies eligible for inclusion.


[email protected]

How Do Systematic Reviews Come to Be Empty?

• Lack of high quality studies to address question posed and/or

–A very narrow population

–A very focused intervention

– Specific outcomes

• Many plan empty reviews to justify new trials in response to funding policies


[email protected]

Current Position

• As of 15 August 2010, 376 (8.7%) of Cochrane reviews contained no included studies

• Within The CDSR, the number of empty reviews is increasing and is inconsistent across Review Groups


[email protected]

Date of First Publication of Empty Reviews as at 15 August 2010

Introduction

• Of the 376 (8.7%) empty Cochrane reviews,

– 25.3% did not list excluded studies,

– 90.2% did not list ongoing studies,

– 96.0% did not list studies waiting assessment, and

– 23.4% did not list any studies other than what was discussed in their background section

• Proportions of empty reviews differed between groups, from 0% to 26.9%

• CRG editors differed somewhat in their perspectives on empty reviews and on editorial guidance provided to authors of these reviews


[email protected]

Proportion of Empty Reviews across CRGs (as of 15 August 2010)

0.0

5.0

10.0

15.0

20.0

25.0

30.0

Acu

te R

esp

irat

ory

Air

way

s

An

aest

he

sia

Bac

k

Bo

ne

/Jo

int/

Mu

scle

Bre

ast

Can

cer

Ch

ildh

oo

d C

an

cer

Co

lore

ctal

Can

cer

Co

nsu

me

rs

Cys

Fib

/Ge

n D

is

De

me

nti

a

De

p/A

nx/

Ne

uro

sis

De

vel/

Psy

ch/L

ear

n

Dru

gs/A

lco

ho

l

Ear/

No

se/T

hro

at

Effe

ctiv

e P

ract

/Org

Epile

psy

Eye

s/V

isio

n

Fert

ility

Re

g

GY

N C

ance

r

Blo

od

Ca

nce

r

He

art

He

pat

o-B

iliar

y

HIV

/AID

S

Hyp

ert

en

sio

n

Inco

nti

ne

nce

Infe

ctio

us

Dis

eas

e

Bo

we

l Dis

ord

ers

Inju

rie

s

Lun

g C

ance

r

Me

nst

rual

/Su

bfe

rt

Me

tab

/En

do

crin

e

Me

tho

ds

Mo

vem

en

t D

is

Mu

ltip

le S

cle

rosi

s

Mu

scu

losk

ele

tal

Ne

on

atal

Ne

uro

mu

scu

lar

Ora

l He

alth

Pai

n/P

allia

tive

Pe

rip

he

ral V

ascu

lar

Pre

gnan

cy/B

irth

Pro

stat

e/U

rol C

anc

Pu

blic

He

alth

Re

nal

Sch

izo

ph

ren

ia

STD

s

Skin

Stro

ke

Tob

acco

Up

pe

r G

I/P

ancr

eas

Wo

un

ds

Importance

• Empty reviews are important as they:

– Tell us who is undertaking the review and thus interested in the topic

–Highlight major research gaps

– Indicate the state of the evidence at a point in time

– Justify further research and/or funding

–Highlight potential harms


[email protected]

Frustrations

• Policy-makers and practitioners report frustration in locating a review only to find that it has no included studies and is of limited utility for clinical decisions

• Lang, Edwards & Flieszer (2007) suggested that authors of empty reviews note observations based on excluded studies so that decision-makers are not left empty handed


[email protected]

Also

• In the absence of included studies some authors use evidence from excluded studies for reporting implications for practice: – basing their recommendations on the “available

evidence” not necessarily the best “evidence”

• At present there is little guidance for reporting excluded studies in the Cochrane Handbook for Systematic Reviews of Interventions – There is no guidance for reporting of empty

reviews The Cochrane Empty Reviews Project

[email protected]

Aims of the Empty Reviews Project

• To examine the way in which empty Cochrane reviews are reported

• To review current guidelines for reporting of empty reviews

• To develop suggestions for guidance for the conduct and reporting of empty reviews


[email protected]

Consider Current Guidance in the Cochrane Handbook

With thanks to Julian Higgins MRC Biostatistics Unit, Cambridge

Co-Editor, Cochrane Handbook for Systematic Reviews of Interventions


[email protected]


[email protected]


[email protected]


[email protected]


[email protected]


[email protected]


[email protected]

Note 1/2

• Cochrane reviews should be scientific reviews and as such must not draw conclusions based on data that are not part of the review – Such evidence suffers from all the problems of narrative reviews that

we strive so hard to avoid

• If no reliable evidence is found, this needs to be reported clearly, and not middied with unreliable evidence

• If it is realised after the fact that reliable evidence was excluded, the reviewers got their eligibility criteria wrong in the first place – You can’t fix that by post hoc cherry picking


[email protected]

Note 2/2

• We do need to think about the relevance of reviews to decision makers, who often want the best available evidence rather than the available good evidence

– Perhaps a paradigm shift is necessary, and many are working on this (e.g. NRSMG)

– But to revolve the discussion around “empty reviews” is not helpful


[email protected]

Reporting of Empty Reviews

• Lang et al. (2007)

– Suggest guidelines needed

– Propose that reviewers could note observations from ineligible articles and abstracts

• Green et al. (2007)

– Note that specific structure could be helpful

– Argue that basing conclusions on studies which do not meet inclusion criteria increases the risk of bias


[email protected]

How Do CRGs (and Others) Handle Empty Reviews?

• Identify policies & procedures on

–Determining publication of empty reviews

–Practice of handling excluded studies across Cochrane and other providers

– Informing practice based on empty reviews and excluded studies


[email protected]

Methods

• Survey with closed and open-ended questions

• 53 Cochrane Collaboration Review Groups

• WHO, JBI, What Works Clearinghouse and SCIE

• Coded & analyzed responses


[email protected]

Cochrane Groups Publishing Empty Reviews

N=53


[email protected]

Rationale for Publishing Empty Reviews

% o

f

responses

N=46


[email protected]

Other Providers

• WHO

– No policy on empty reviews

– They search for the best available evidence, and then make a recommendation based on what they find

• SCIE

– As the evidence base in social care is very limited, their policy is similar

• WWCH

– Review focus is changed depending on the evidence base, thus empty reviews are not generally produced


[email protected]

CRG Views about Including Empty Reviews

• Include to instigate further research – “Our aim with empty reviews is to raise the profile of the

question and hopefully generate research leading to high quality evidence which we can then report quickly in an updated review.”

• Include if good case made – “There may be benefit in amending the TRF form to include

the line: ‘If you think this review might be empty, please give your reasons why you feel it is important to do the review.’”


[email protected]

CRG Reasons for Not Publishing Empty Reviews

•Not worth time and resources – “a colossal waste of everyone’s time and energy”

•Influence on group’s impact factor •May be misleading

–Empty reviews (< 3 studies) can put positive supportive evidence from initial, low quality studies behind treatment

•Some groups have minimum # of studies stipulation - e.g., 2-3

•BUT, question can still matter – “may be exceptions where review questions appear of

great public health relevance”


[email protected]

Handling Excluded Studies

Standard Policy?

No 75%

Un-known

4%

Yes 21%

Permitted in Review?

No 31%

Un-known 11%

Yes 58%


[email protected]

Conditions for Providing Evidence Based on Excluded Studies

% o

f re

spon

ses N=31


[email protected]

Further Considerations for Whether to Allow Excluded Studies

• Excluded based on intervention or outcomes?

• RCTs feasible/ethical for topic area? – If not, lower grade studies may be acceptable

• Guidelines may be needed for application of evidence from excluded studies – “We dissuade authors from providing evidence from

excluded studies because of inconsistency, which would lead to provision of evidence that is arbitrary and unsystematic.”


[email protected]

Guidance for ‘Implications for practice’ Section

For ‘Yes’ responses (n=7):

• Extra guidance but on case-by-case basis (3)

• General advice (2)

• Refer to other empty reviews, EPICOT, & extra advice (1)

• Group Methodologists give guidance (1)

N=53


[email protected]

Views about Priority on Updating Empty Reviews

• No, topic-based priority only: – “Priority is only given on the basis of importance of the

review topic, not on whether there are included studies or not.”

• Only if new study emerges: – “Aim to update/review them around the same time as

other reviews , but if we became aware of a study relevant to an empty review, it would probably prompt a more rapid update.”


[email protected]

Content Analysis of Implications for Practice in Empty Reviews


[email protected]

What We Wanted to Know

• Despite the lack of high quality, or any, evidence included in empty reviews, how do authors report implications for practice?


[email protected]

Objectives of the Content Analysis

To explore the following:

• How authors frame discussions of Implications for Practice?

• Do authors incorporate information from other research? If so, how and where?

• Do authors make recommendations for practice?


[email protected]

Conceptual Definition of Recommendation

• We defined recommendations for practice as statements which instruct readers, including consumers, practitioners, or policy makers to do something


[email protected]

One Sentence Wonders

• 117 (31.1%) – "None identified from this systematic review.”

• Mention lack of evidence and review topic: – “No randomized controlled trials, which assessed the

effectiveness or safety of oral immunoglobulin preparations for the treatment of rotavirus diarrhoea in hospitalized low birthweight infants were found.”

• Absence of evidence is not evidence of absence: – "No relevant trials appear to have been published so this

review concludes that there is no evidence of effect of leukotriene antagonists in bronchiectasis rather than evidence of no effect.”


[email protected]

Vague Wording

• 26 (6.9%) - e.g. – “It is likely that clinicians will continue with their current

practice, using clinical judgement and prescribing patterns to dictate treatment because there is no RCT-based evidence to help guide their choice of drug. It is difficult to know whether current practice is justified outside of a well designed, conducted and reported RCT. Currently policymakers have no RCT-based evidence upon which to base guidelines for HIVAN. They are likely to continue to rely on opinion and habit when making their recommendations. Funders of studies may wish to make this important subgroup of people a priority for future research.”


[email protected]

Failure to Cite

• 50% of implications for practice sections make reference to evidence, only 16% of these cite its source – “The implications for fluid management in acute

respiratory infections have not been studied in any RCTs to date. There is currently no evidence for or against increased fluids in acute respiratory infections. Non-experimental (observational) data suggests that there may be a risk of symptomatic hyponatraemia due to increased antidiuretic hormone secretion in lower respiratory tract infections, particularly in children. The incidence in the primary care setting and the clinical significance of this observational data needs to be determined with further research, conducted as randomised controlled trials.”


[email protected]

Targeted Discussions "1. For people with severe mental illness

At present, the choice between dedicated supported housing schemes and outreach services is based on a combination of personal preference, professional judgement and availability of resources. Decisions of this nature should be made with the full understanding that no one intervention has been shown to be more effective than another in making a difference to symptoms, future use of services, quality of life or other measures of importance. Furthermore, the efficacy of supported housing remains untested. Participating in trials that test the effectiveness of such services should be encouraged.

2. For practitioners

In the absence of evidence of their relative efficacy, decisions on the provision of alternative forms of accommodation and continued support for people with mental illness can only be based on a combination of professional judgement, patient preference and availability. This should be made clear to the patient or client who has to make this important decision. Practitioners may wish to actively support or participate in trials to test the effectiveness of supported housing schemes for people with severe mental illness.

3. For managers and policymakers

Policies in favour of dedicated supported housing schemes should be viewed with some caution and should not be implemented without plans for evaluation using rigorous methods or should be delayed pending further evidence of their effectiveness. Forming alliances with researchers within this field may result in a fruitful collaboration that would not only inform local policies on this issue, but would also provide much needed evidence base on its effectiveness."


[email protected]

Implications for Practice with Recommendations

• 59 of 376 empty reviews (15.7%) appeared to contain recommendations

• Grouped as:

– generally supportive,

– recommending alternatives, or

– recommending either restricted use or non-use


[email protected]

Generally Favorable Recommendations

• Represent 23 of 59 (39.0%) empty reviews which appear to make recommendations

– 14 of these appear to have been based largely on other research cited in the background

– 7 seem to not be based on evidence, but on guidelines, theory, or absence of evidence of harm

– 2 seem to be based on excluded studies


[email protected]

Example of Generally Favorable Recommendation

• “There is a need to raise awareness of the likely risks of waterpipe tobacco smoking worldwide. Particular attention should be devoted to examining empirically the potentially false perceptions about levels of dependence, disease and mortality associated with waterpipe tobacco smoking.

• Evidence-based information about waterpipe should be developed and disseminated in order to deglamourise and denormalise its use.

• Young people, especially women and girls in the Eastern Mediterranean Region, may be at higher risk of waterpipe use, and may need particular attention from healthcare workers.

• In the absence of policy research on waterpipe, and given what we know about its potential harmful effects, there is a need to include this tobacco use method in all tobacco control policy initiatives (e.g. bans on advertisement and sales to minors, taxation, warning labels, smokefree indoor air).

• Treatments of proven efficacy should be provided to help waterpipe smokers quit.”


[email protected]

Recommendations to Use Something Else

• 8 (13.6%) of 59 empty reviews making recommendations

• Appear to be based on existing guidelines, practice standards, other research, or nothing readily apparent


[email protected]

Example of Recommendation to Use Something Else

• “As there were no randomised or quasi-randomised controlled trials identified, we cannot draw any conclusions on the benefits (or otherwise) of regular administration of vitamin A in people with cystic fibrosis. Until further data are available, country or region specific guidelines (e.g. UK CF Trust Nutrition Guidelines (CF Trust 2002)) on the use and monitoring of vitamin A in people with cystic fibrosis should be followed.”


[email protected]

Recommendations for Restricted Use

• 13 (22.0%) of 59 empty reviews making recommendations

• Limits on use include use in some circumstances (1), possible use in some circumstances (1), minimal use (1), or use only in the context of a controlled trial (10)

• Appear to be based largely on evidence of harm, possibility of harm, or no readily apparent basis


[email protected]

Examples of Recommendations for Restricted Use

• Restricted Use

“There is no evidence so far that subjective barriers reduce wandering, and the possibility of harm (particularly psychological distress) cannot be excluded. If used, then subjective barriers should form part of a diverse approach to problem wandering, which may include the identification and definition of the problem in the individual, preventative activities such as exercise classes or occupational therapies, and improved communication between carer and wanderer.”

• “Restrict to Trials”

“…2. For clinicians In the absence of reliable evidence, clinicians prescribing calcium channel blockers for people with tardive dyskinesia must balance the possible benefits against the potential adverse effects of the treatment. Calcium-channel blockers lower people’s blood pressure and may even cause symptoms of tardive dyskinesia to increase. These drugs should really only be used in a situation where their effects are closely monitored, i.e. within a randomised experimental design that can be written up and disseminated in order to inform practice.”


[email protected]

Recommendations to Not Use

• 15 (25.4%) of 59 empty reviews making a recommendation

• Appear to be based on lack of evidence, evidence of harm, or ongoing research where trials had been aborted


[email protected]

Example of Recommendation to Not Use

• “Given the lack of evidence comparing hydralazine versus placebo for primary (essential) hypertension, hydralazine should not be recommended as monotherapy. Clinicians who wish to recommend hydralazine for their patients should understand that while hydralazine may reduce blood pressure (based on non-randomized cross-over trials), to the best of our knowledge, there are no known clinical studies which have associated the use of hydralazine with a reduction in all cause mortality, myocardial infarction, or stroke. In addition, despite poor reporting of treatment-emergent adverse effects, clinicians must weigh the risks of potential serious side effects associated with the use of hydralazine, such as reflex tachycardia, hemolytic anemia, glomerulonephritis, vasculitis, as well as lupus-like syndrome against the potential benefits of blood pressure reduction with no proven beneficial effect on adverse cardiovascular outcomes.”


[email protected]

Summary

• Implications for Practice sections differ widely across empty reviews

• Some Implications for Practice sections are very brief, while others are quite lengthy

• Some Implications for Practice sections are difficult to interpret or vaguely worded

• Many Implications for Practice sections appear to refer to research without a formal citation

• Whilst the majority of Implications for Practice sections do not make recommendations, some do


[email protected]

Some Good Practices Observed

• Clear statement on lack of included evidence

• Explanation of what that means, i.e., “absence of evidence is not evidence of absence”

• Appropriate comments on harms

• Description of limitations of excluded data

• Reference to related reviews (e.g., similar intervention/different population) with caveats to generalisability

• Consideration of economic issues


[email protected]

Where Has This Project Got to So Far?

1. Paper submitted outlining the issues surrounding empty reviews in The CDSR and the quantitative data presented here

2. Papers in preparation outlining the handling of empty reviews, how they are regarded and the qualitative data from CRGs and other systematic review producers

3. Website to collect views on the issues regarding need for guidance

4. Consensus meeting in June with stakeholders; feedback needed before proceeding further


[email protected]

Feedback and Guidance for Empty Reviews


[email protected]

Questions for Discussion

• How do we define an empty review?

– A systematic review that results in no included/eligible studies

• Empty reviews are suitable for/when :

– Important questions for practice or policy

– Trials are feasible

– Though may need to amend the protocol as part of an iterative process


[email protected]

Questions for Discussion

• Do we need guidance for conducting and reporting empty reviews?

– Yes, in these circumstances:

1- Reviews done and ready to report

2- Reviews in progress

3- Anticipating low evidence reviews

4- Updates

• Our Focus is #1, perhaps, with implications for #2-4


[email protected]

Completed Empty Reviews

Should guidance be generic or organized by review question (e.g., parachute vs. RCTs unfound vs. side effects vs. trials inconceivable) or CRG?

• Guidance should be general to the whole collaboration


[email protected]

Which Sections of Empty Reviews Require Guidance?

Abstract

Main Results

• Brief sentence – no evidence

Author’s Conclusion


• May highlight possible severe/prevalent adverse effects


[email protected]


Plain Language Summary


• Explain why empty, if appropriate, referring to eligibility criteria

• May highlight possible severe/prevalent adverse effects

Summary of Findings

• Include even if empty


[email protected]


Background

Objectives

Methods

Criteria for considering studies for this review

Search methods for identification of studies

Data collection and analysis

• Omit data collection/analysis sections

• Reference protocol


[email protected]


Results

Description of studies • Flow Diagram

Results of the search Included studies

• Brief sentence – no evidence On-going studies Excluded studies

• Usual text • Do not report excluded results here

Risk of bias in included studies Effects of interventions


[email protected]


Discussion • Follow handbook subheadings • Provide examples • May discuss excluded study results, but –Under heading –Reference –Discuss reason for exclusion –Bookend with caveats –Be brief


[email protected]


Author’s conclusions Implications for practice • Follow handbook – no recommendations

– Add examples to Handbook • Brief sentence – no evidence • Other evidence is not to be reported here • General statements referring to local practice guidelines • May flag severe/prevalent adverse effects

– Cite evidence

Implications for research • Signal for update (on-going, awaiting, etc.) • General statement – research needed/follow Handbook


[email protected]

Writing and Dissemination Strategies

• Standard Handbook applies

– Possible special addenda for empty reviews, section dependent (draft 2-3 pages to integrate)

• David Tovey – editor training

• Colloquia workshops, editors’ meeting, etc.

• Website, webinars, etc.

• Other forums?

• Stand alone empty review reporting document?


[email protected]

Preliminary Considerations of Empty Reviews at Title Registration, at

Protocol, in Progress, and at Update


[email protected]

Should Contingency Plans for Empty Reviews Be Defined in the Review TRF/Protocol?

• At TRF – (Decision tree)

1. Trial feasible? Y/N

2. Aware of at least one possibly includable trial? Y/N

3. Important to review even if empty?

• At Protocol – (Decision tree continued)

– Provide a contingency plan in case no eligible evidence?


[email protected]

Should empty reviews be flagged in the Cochrane Library?

• No


[email protected]

What are the views regarding the updating of empty reviews?

• Same general principles of updating apply


[email protected]

Any other points to consider?


[email protected]

The Cochrane Empty Reviews Project Funded by The Cochrane

Opportunities Fund

Paul Montgomery, University of Oxford Sally Hopewell, UK Cochrane Centre Joanne Yaffe, University of Utah Lindsay Shepard, University of Utah

[email protected] http://www.empty-reviews.org

mailto:[email protected]

http://www.empty-reviews.org/



Date post:	10-Oct-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Running on Empty · frustration in locating a review only to find that it has no included studies...

Documents