uOttawa.cauOttawa.ca

Deconstructing Rapid Reviews

• An Exploration of Knowledge, Traits and Attitudes

Presented by: Shannon Kelly, CADTH Symposium 2015 – Saskatoon, SK

uOttawa.ca

Faculty of MedicineSchool of Epidemiology, Public Health and Preventive Medicine

uOttawa.ca

Objectives

• To present results from 3 projects on rapid reviews conducted as part of MSc. Epidemiology at uOttawa

I. Defining rapid reviews – a consensus approach

II. Attitudes and perceptions towards rapid reviews

III. Defining rapid reviews – a pragmatic approach

2

uOttawa.ca

RAPID REVIEWS?Why do we care about

uOttawa.ca

What is a rapid review?

“…a streamlined approach to synthesizing evidence in a timely manner - typically for the purpose of informing emergent decisions faced by decision makers in health care settings.”

~ Khangura, 2012

 Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Systematic Reviews. 2012;1:10.

uOttawa.ca

Rapid reviews influence decision-making in Canada

• Knowledge users in Canada (everywhere) need support to make timely, evidence-informed health care decisions

• Systematic reviews and HTA often take too long!

• Delay important policy decisions

• Use less-than-best evidence

• Rapid reviews help fill this void

• Balance rigour, relevance, timelines

…We still need to know so much more…

uOttawa.ca

What do we know?

Ultra-rapid reviews, first test results

uOttawa.ca

What don’t we know?

“While rapid review approaches are being used by HTA producers across the world in a variety of ways an agreed-upon definition describing its constitution and methods is lacking” (Khangura, 2014)“While the concept

of rapid evidence synthesis, or rapid review, is not novel, it remains a poorly understood and yet ill-defined set of disparate methodologies supported by a paucity of published, available scientific literature”(Khangura, 2014)

“..evidence is only slowly emerging as to which steps in the systematic review process may be altered by increased speed” (Schünemann, 2015)

“In order to consider offering rapid reviews, one first has to be able to define what they are. Previous reviews on the topic of rapid reviews describe them as ‘ill-defined,’ ‘not well-defined,’ lacking a single definition, or ‘varying widely in terms of the language used to describe them.’” (AHRQ, Hartling, 2015)

“…a literature search by the authors does not elucidate any clear or final definition of what a RR is and how exactly the methodology of a RR differs from a full SR.” (Harker, 2012)

“There is little empirical evidence comparing the continuum of products among rapid reviews and full systematic reviews, or analysing the diverse methods used in rapid reviews. It is important not only to establish transparent methodologies for rapid reviews, but also to understand the implications of what is lost in terms of rigour, bias, and results when methods associated with full systematic review are streamlined.” (Ganaan, 2010)

uOttawa.ca

Needed a research program to improve knowledge on rapid reviews

An improved understanding of rapid reviews will help us better meet the needs of decision-makers

uOttawa.ca

DEFINING RAPID REVIEWSA consensus approach

uOttawa.ca

Objective

• The study will employ a modified Delphi method to survey of experts in the field of evidence synthesis and rapid reviews to discover what they consider the key defining characteristics of rapid reviews to be.

Note: For our purposes, defining characteristics are:

‘one of a number of essential features by which a rapid reviews can be recognized’.

uOttawa.ca

Method: The Delphi Technique

Definition: a procedure that structures a communication process among a group of experts.(Linstone & Turoff, 1975)

Characteristics of the Delphi technique: Staged process, assessment of agreement or consensus Assessment of the group’s view; An opportunity for informants to revise their views; Opportunity to react to and assess differing view points; Anonymity of informants.

Panelists are selected based on their expertise in the subject matter; not random.

uOttawa.ca

Participants

Experts targeted for participation were:

1. Subject-matter experts or experienced researchers with knowledge of a variety of evidence synthesis methods and practical experience with rapid reviews;

2. Authors with publications relevant to rapid reviews; and,

3. Delegates presenting pertinent work at recent meetings, workshops, conferences or symposia.

• International Cochrane Colloquium, Health Technology Assessment international (HTAi), CADTH Symposium, Cochrane Canada Symposium).  

uOttawa.ca

The Modified Delphi Process

STUDY SCOPING AND EXPERT PANEL DEFINITION

ROUND 1: WEB-BASED SURVEY ON KEY

RR THEMES

(FluidSurveys)

ROUND 2:DRAFT CHARACTERISTIC

STATEMENTS

(email)

ROUND 3:REVISED STATEMENTS, EXPERT REVISION OF

AGREEMENT

(email)

FINAL 11 CHARACTERISTICSTATEMENTS

PUBLICATIONPENDING

Review agreement, refine statements

120 POTENTIAL EXPERTS

Pilot Review, summarize,

analyze, draft statements66 EXPERTS 44 EXPERTS

26 EXPERTSCONSENSUS ON 10 OF 11 STATEMENTS

Consensus level = 70%, Approved by TOHREB

Review agreeme

nt

Summarize results

uOttawa.ca

Results: Key Characteristics (1)

• Conducted in less time than a systematic review;

• Have a protocol (at least describing objectives, scope, PICO, approach);

• Tailor the explicit, reproducible methods conventionally used in a systematic review in some manner to expedite the process to meet the needs of a decision-maker;

• Use a spectrum of approaches to complete an evidence synthesis related to a defined research question(s) using the most systematic or rigourous methods the limited time frame will allow.

Rapid reviews:

uOttawa.ca

RR time to completion

Delphi Expert Opinion

Previous RR Samples GANAAN

2010HARKER

2012INAHTA scan

2012POLISENA

2015AHRQ2015

TRICCO (RR SYMP.

CADTH)

Minimum 3 wk >3 mo 1 wk 1 wk 5 min <1mo

Maximum 6 mo < 24 mo 12 mo 12 mo 8 mo 12 mo

Rapid Review

Systematic Review

0 10 20 30 40 50 60

Time (in weeks)

Khangura et al. (2012): SRs take 6 months to 2 years, and RRs ≤ 5 weeks.

Delphi panel: Median time for SR is 36 weeks, while median for RR is 9 weeks.

Min = minute; d = days, wk = week; mo = month; yr = year

uOttawa.ca

Results: Key Characteristics (2)

• Balance the risk of the yet undetermined impact of workflow adaptations to the validity of the review and its results, with the needs of the knowledge user;

• Report methods and findings transparently, with appropriate detail required to

– answer the research question;– meet the requirements of the knowledge user and

allow assessment of confidence in the results; – inform the audience for which the review is intended;– while meeting delivery time line agreed upon in

advance.Consensus (72 - 100%)

uOttawa.ca

Still up for debate…• Tension: risk of bias/critical appraisal;

– individual study or review level

– distinct and opposing camps

• Achieved consensus on no consensus – RRs as a unique method;

• Time is still relative…;• Protocol format, publication/registration;• Peer review;• Unpublished (grey) literature;• Not all rapid reviews meet this definition? Overlap with

some key features of SR, HTA.

Statements based only on the experience of the expert panel…must be balanced with published literature.

uOttawa.ca

RAPID REVIEWSExploring attitudes and perceptions towards

uOttawa.ca

Method: Q methodology

A research technique that permits the systematic study of subjectivity (opinions, beliefs and preferences).

INTERPRETATION

FACTOR ANALYSIS

RANK ORDER OF THE STATEMENTS: Q SORT

DEVELOPMENT OF REPRESENTATIVE STATEMENTS: Q SET

DEFINING STUDY PARTICIPANTS: P SET

COLLECTION OF OPINION STATEMENTS: CONCOURSE

RESEARCH QUESTION

*Adapted from Amin, 2000

uOttawa.ca

Conceptual Q sort layout

We replicated this process using online Q methodology tools:

uOttawa.ca

• When time allows, a comprehensive systematic review of all available evidence should always be conducted.

• Further research comparing the methods and results of rapid reviews and systematic reviews is required before I decide how I feel about rapid reviews.

• Rapid reviews mean different things to different people.

• Rapid reviews should only precede a more comprehensive and rigorous systematic review.

• The opportunity cost of a comprehensive SR or HTA is too high and it is more advantageous to conduct rapid reviews when timeliness is a factor.

• All evidence synthesis products, including rapid reviews, SRs, or HTAs, can be conducted very well or very poorly.

• Rapid reviews are comparable to SRs except they are done in a more timely fashion.

• Rapid reviews are ‘quick and dirty' systematic reviews.

• Rapid reviews need to be tailored to the specific needs of the knowledge user.

• Rapid reviews meet the needs of knowledge users.

• It is always appropriate to conduct a rapid review.

• It is important to have minimum standards for the reporting of rapid reviews (e.g., a PRISMA-RR).

• Standardization of rapid review methods may conflict with the needs of knowledge users

• A good quality review of evidence is determined by the methods used, not by the speed at which it is completed.

• It is difficult to tell a rapid review from a systematic review unless very specific nomenclature is used in the title or description of methods.

• A rapid review cannot be a systematic review.

Sample statements

uOttawa.ca

Summary

• 11 participants performed the online Q sort

• 50 statements representinga range of viewpoints andand subjective opinion on RRs

• Following factor analysis3 factors extracted

Geographic Location n (%)

Canada 11 (100)

Age: n (%)

18 to 35 2 (18.2)

36 to 50 4 (36.4)

50 or above 5 (45.5)

Sex: n (%)

Female 8 (72.7)

Education: n (%)

Doctorate 7 (63.6)

Masters 3 (27.3)

Undergraduate 1 (9.1)

Consider Themselves: n (%)

Researcher/Producer 8 (72.7)

Knowledge User 2 (18.2)

Neither 1 (9.1)

Have ever been the author of a rapid review

n (%)

Yes 7 (63.6)

No 4 (36.4)

Unsure 0 (0)

Have used a rapid review to aid in a policy or decision-making?

n (%)

Yes 7 (63.6)

No 3 (27.3)

Unsure 1 (0.9)

uOttawa.ca

Title

uOttawa.ca

Title

uOttawa.ca

Factor 3. “Pragmatism over principle”

uOttawa.ca

Consensus Statements

Agree6. Rapid reviews mean different things to different people.

Disagree7. Rapid reviews should only precede a more

comprehensive and rigorous systematic review.34. It is appropriate to endeavor to define a single, unique

methodology for rapid reviews.42. Any review of evidence that takes longer than 3 months

to produce is not a rapid review.43. Any review of evidence that takes longer than 1 month to

produce is not a rapid review.

uOttawa.ca

DEFINING RAPID REVIEWSA pragmatic approach

uOttawa.ca

Summary of results

• Review of RRs published in journals or as reports by various agencies and organizations worldwide;

– Supplemented with INAHTA scan to query HTA agencies.

• Screened 1,800+ database records and grey literature results - Single reviewer, checking by a second;

• Included RR samples were examined for completion time, nomenclature, study characteristics, approach/tailoring and other relevant features, including whether the results are contextualized or framed with policy implications.

uOttawa.ca

In general

• Findings matched previous work on RR methods:– “ One species, many breeds” Heterogeneity of approaches,

nomenclature – supports the evidence continuum concept;– May find more than one approach within a single

organization (e.g., CADTH);– Methods not well-reported; Self-identifying RRs with no way

to assess what was tailored.• In addition:

– RRs are getting published;– Need to focus on contextualization of results, policy

implications for end-user;– Starting to see RR protocols published (n=4) or registered

in PROSPERO (n=3).

uOttawa.ca

Summary• Addressed RR knowledge gaps with 3

separate but interrelated studies;

• Experts have identified defining characteristics that may help differentiate RRs from other evidence synthesis approaches;

• Agree to disagree in some areas;

• Three distinct viewpoints on RRs were ascertained during a Q methodology study of subjective opinion;

• A selection of RR samples show that there is room for improvement (without compromising on the needs of end-users).

uOttawa.ca

Questions? Thoughts?

A special thank you to Dr. Tammy Clifford and Dr. David Moher, to CADTH for the initial funding for this student project, and

everyone who has participated in surveys, piloted questionnaires, shared data and supported this research with their own time