“MEETING THE CHALLENGE OF MAINTAINING COCHRANE REVIEWS” Author: Dr David Tovey FRCGP Editor in...

“MEETING THE CHALLENGE OF MAINTAINING COCHRANE REVIEWS”Author: Dr David Tovey FRCGPEditor in Chief, The Cochrane Library

AHRQ 2012 Annual Conference

2

Plan

Introduction Previous Cochrane project “Fit for purpose” project Future challenges and opportunities for

co-operation Questions

Sally Hopewell, Yemisi Takwoingi, Alex Sutton, Rachel Marshall and Bazian Ltd.

This project was funded by the NIHR Cochrane – NHS Engagement Award Scheme (project number 10/4000/01). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health.

Acknowledgements


4

Acknowledgements

Cochrane Musculoskeletal Group Cochrane Pain, Palliative and Supportive Care

Group Cochrane Infectious Disease Group Cochrane Wounds Group Cochrane Neonatal Group Cochrane Airways Group Karla Soares-Weiser


5

Introduction: Protecting Archie Cochrane’s vision

“It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials”


6

Introduction: Protecting Archie Cochrane’s vision

“It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials”

Credible& Safe


7

Introduction: the size of the challenge


8



9



10


“The boulder in my rucksack”


11

Previous Cochrane projects: the updating officer project

8 reviews selected

On average these reviews each took 6.4 months (range 3-11 months) to update from receipt of the search strategy to submission for editorial review.

The main challenges: lack of familiarity with individual Review Groups’ methods out of date methodology within the existing reviews lack of subsequent author commitment lack of wish for ongoing commitment

Would imply the need for a small army of updating officers!!


12

“Fit for purpose”project

Three objectives of the project: To work with external stakeholders to

develop a customisable prioritisation tool – prioritisation by topic

To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status

To explore whether targeted consultancy could facilitate updating and identify process efficiencies


13

“Fit for purpose”project





Methods

Project conducted by Bazian, co-applicants in the NHS engagement award.

Stakeholders in the NHS (clinicians, consumers, commissioners etc.) formed a panel.

The panel had two meetings, and communicated via email before and after meetings.

Results: prioritisation criteria in the tool

The tool was tested on 19 Cochrane Reviews, which found: There is a level of judgement when assigning scores. A certain level of knowledge of the field is helpful when

assigning scores Needs to be customised (notable absentee parameters..)

Recommended that the results of using the tool are discussed with external stakeholders including patient and carer representatives.

Results: testing the tool


17

“Fit for purpose” project





We refined and amalgamated two complementary methodologies for prioritising systematic review updates: a qualitative tool based on a broad range of updating signals

(Loudon 2008). formal statistical methods which assess when the inclusion of

new studies is likely to change a review’s conclusions (Sutton 2009).

Methods

Output: Decision tool

Step 1: Is the clinical question answered or no longer relevant?

Step 2: Are there any new factors to consider?

Steps 3 & 4: Are there new studies? Are the conclusions likely to change?

Statistical prediction tool

About metarank

Based on minimal information on the new evidence assumes an update strategy is in place such

that number of new studies and their sample sizes are known

‘Signals’ of the need to update implemented as a STATA user-written function

Performs simulation of several meta-analyses, each with one or more new studies of different sizes

Metarank output Summary of results for each meta-analysis includes

details of the original meta-analysis and the signals detected

Table with all reviews in the dataset ranked in order of priority by a given signal or the total of all signals triggered

Includes descriptive information for the collection e.g. average number of trials and participants in the

collection of meta-analyses number of trials in the largest meta-analysis largest number of participants in a meta-analysis

SYSTEMATIC REVIEW UPDATE SIGNALS REVIEW ID: 11 Fixed effect Mantel-Haenszel model Outcome measure = relative risk Estimates from meta-analysis of previous studies: ------------------------------------------------------------------ Effect size = 0.69 95% confidence interval = (0.58, 0.83) I-squared = 66.17% p = 0.0001 Egger's regression test for publication bias, p= 0.7666 Simulation Results: ------------------------- alpha= 0.05 Power = 0.10%, 95% confidence interval (0.00, 0.30) Limits of clinical equivalence not stated Power of magnitude of treatment effect = 0.50% Average ratio of standard error of updated meta-analysis to previous meta-analysis = 0.59 Average ratio of total weight of new studies to that of previous studies in the updated meta-analysis = 1.97 Shojania et al signals: ----------------------------- 1. A new study is 3 or more times larger than a previous study: No Size of largest previous study = 554, largest new study = 886 and ratio = 1.60 2. Largest new study (n=886) larger than previous largest study (n=554): Yes 3. Ratio of the total number of participants in all studies to the previous total (=2.87) > 1.5: Yes Total number of participants in previous studies = 901 Total number of participants in all studies = 2589 4. Ratio of the total number of studies to the previous total (=1.80) > 1.5: Yes Number of previous studies = 5 Total number of studies = 9 Signals triggered = 3 Barrowman's n: -------------------- Barrowman's n is not applicable because previous meta-analysis was statistically significant (p=0.0001).

Advantages of metarank

Assessment of the likelihood of any definable criteria changing in a meta-analysis

Ongoing studies can be taken into account aid in predicting potential ‘shelf life’ of a

review in the light of new or accruing evidence

To provide information with respect to further research needed such as the number of new trials and the number of participants useful information to end users of reviews in

assessing the stability or validity of a review

Limitations of metarank

Relies on availability of some information on new trials

Simplistic approach used in deriving the predictive distribution for new studies

Other issues external to a review may need to be taken into account such as the rate at which new evidence on a topic

evolves public heath significance etc.

No criteria developed to establish when enough evidence has accrued on a given topic and review that it is deemed decisive and not worth conducting further primary research

The decision tool provides a set of criteria that can be used to assess whether to update a Cochrane Review.

The tool can be applied to a single Cochrane Review or can be used to prioritise a suite of reviews (e.g. those from an individual Cochrane Review Group)

Decision tool: summary


30

Results: testing the tool

One Cochrane Review Group so far Cautiously positive -

“helpful structure to assess each review’s eligibility for updating”“a transparent way to explain decisions around updating to stakeholders”

Time consuming but might improve with experience

Some assumptions around the statistical tool “ too crude”

Need to have wider experience and use


31

“Fit for purpose” project

Three objectives of the project: To identify those Cochrane Reviews that

NHS stakeholders regard as the most important to update – prioritisation by topic




32

Aims

We aimed to provide short and focused periods of assistance, to incentivise authors, and to address barriers to updating.

The intended approach was to retain both the responsibility for producing the update within the authors, and editorial support for within the CRGs.


33

Methods

One-year project (July 2010 to July 2011) Involved 6 CRGs Each CRG offered 37 days’ of assistance for updating

1 to 6 Cochrane Reviews (between November 2010 and April 2011)

Tasks performed by reviewers from inside and outside Cochrane without specific topic knowledge


34

Tasks offered to CRGs and authors


35

Results: assessment of tasks


36

Results: CRG and author feedback

Evaluation form: all CRGs responded, most authors responded (1 on holiday, 1 no response)

Scale 1-5 (1 greatly improved, 5 greatly worsened)

All authors and managing editors who responded to the question on rolling out an updating service for The Cochrane Collaboration were in favour of an updating service (2 CRGs did not respond)

Authors CRGs

Author motivation

1.6 1.8

Speed of update

1.6 2.5

Quality of update

1.8 2.75

Overall impression of assistance

1.44 1.5


37

Results: status of updates

We worked on 14 Cochrane Review updates Mean time spent: 35.5 hours (median 26.71

hours, range 4.5 to 109.75 hours) Lots of activity but.... In October 2011, no updates yet published


38

Future challenges:

Whose priorities? Expectations and the increasing

professionalisation of systematic reviews The rise and rise of methods advances How to balance updates versus new

reviews? Authorship of updates


39

Conclusion

Updating is critical to patient safety and credibility

Increasing recognition of need to prioritise (but this isn’t as easy as it seems)

Transparency to user/reader is critical

Different approaches but no “one size fits all” solution


40

Conclusion: How can we work more effectively together?

Sharing intelligence “Keep up” initiative Shared surveillance

Sharing the workload Sharing data “Wiki” approaches

Novel approaches Use of data mining and semantic technologies Limited search and “Summary updates”


41

thank you for [email protected]


42

Authorship of updates

Scenario:A review is updated that originally included 11 studies

and now has 17. The text includes 35% the same text as the previous version, across all sections. In addition a summary of findings table has been added.The author team has changed completely and none of the previous authors have contributed to the update.

QuestionsHow should the work of the previous authors be

credited?


43

ICMJE guidance on contributorship

Authorship credit should be based on 1. Substantial contributions to conception and

design, acquisition of data, or analysis and interpretation of data;

2. Drafting the article or revising it critically for important intellectual content; and

3. Final approval of the version to be published.

Authors should meet conditions 1, 2, and 3.


44

Authorship of updates

A. As authors of the updated reviewB. Lead author included on the updateC. Should be negotiated on an individual

basisD. Original authors not included as authors

but acknowledged within the reviewE. Some other approach

Date post:	19-Jan-2016
Category:	Documents
Upload:	sabrina-hudson
View:	215 times
Download:	0 times

“MEETING THE CHALLENGE OF MAINTAINING COCHRANE REVIEWS” Author: Dr David Tovey FRCGP Editor in...

Documents