Cochrane Rapid Response: Accelerating Knowledge Translation for Decision-Makers

Chantelle GarrittyOttawa Hospital Research Institute

and Catherine Gallagher

George Mason University

Overview

Knowledge translationEvidence synthesis and knowledge translation: inseparableDecision-making dilemmas

Requirement of high rigor evidenceLack of timeMultiple restrictionsFallacies

Cochrane’s approach Rapid Response: Borrowed heavily from KTA Canada

Impact on translationDecision-making outcomes

Rapid response v. Traditional systematic reviewTradesImpactsEvaluation

Rapid reviews in context…and as a precursor to translation

Evidence needs arise… • Both organically and purposively• Within a system with static and dynamic operational parameters• At any level: Locally, nationally and globally• Always within limited resource structures

Rapid reviews should…• Make the very best use of limited resources• Provide a gauge of the distance between status quo and optimum interventions or systems•Have clear implications for practice and for outcomes• Lead to implementation and assessment of impact

Rapid reviews align perfectly with “habits of high value health care organizations”…• Specification and planning – criteria based decision making for clinical and operations•Deliberate design and management of resources for all systems and for sub-population•Measurement and oversight•Self-study (Bohmer, 2011 NEJM)

Here we discuss tools and methods to…• Enhance, automate, and accelerate processes• Limit biases of narrative input, increase transparency, and facilitate investment• Place the rapid review in its larger context of clinical and systems utility• More specifically , understand the relationship between acceleration and dissemination

*Recognize the limitations of current depth and breadth of evidence pool –Bohmer, 2011 NEJM

Part I: Knowledge translation

What is it?

CIHR (2000) “the exchange, synthesis and ethically sound application of knowledge – within a complex system of

interactions among researchers and users – to accelerate the capture of the benefits of research for the public through improved health, more effective services and products, and a strengthened health care system”

Emerged alongside Evidence-based Practice Movement:

“process of moving what we learned through research to the actual applications…

….in a variety of practice settings and circumstances…...the interest in KT appears to coincide with the growing

engagement in the EBP approach in which…

…practitioners make practice decisions based on the integration of the research evidence with clinical expertise and the patient’s unique values and circumstances.”

Straus, Richardson, Glasziou, & Haynes, 2005

Coinciding with knowledge translation movement

Increased:• rigor and standardization of all elements of the scientific process• burden of proof of utility of investigation• investigation into communication methods (dissemination sciences)• open access tools to summarize knowledge base, manage references • move publicly funded research to the public• stakeholder involvement• joint decision making

It is impossible to separate knowledge translation from knowledge generation and synthesis

Why we need evidence synthesis and knowledge translation

1. To make sense of knowledge proliferation– 10,000 new trials each year in MEDLINE– 350,000 new trials each year in Cochrane

2. Public is not benefiting from evidence

30-40% of patients do not receive care according to present scientific evidence20-25% of care provided is not needed or potentially harmful

Grol & Grimshaw (2003) Lancet

3. Current publication practices do not speak language or format of decision makers (both societal and individual)

Why we need evidence synthesis and knowledge translation

4. Calibrate traditional academic model to meet practice needs

Investigator interest versus Social utilityTimeliness versus ExhaustivenessSimplicity of design versus Complexity of social worldGeneralizability versus Local acceptabilityAcademic freedom versus Costs to publicIncreases in standardization versus Loss of natural variationQuantity of publications versus Quality of

reconciliation

What does it look like?

There are many models of the KT process

Commonalities across models include:1. Early and consistent stakeholder engagement2. Standardization of reporting requirements to maximize benefits of research efforts3. Concern with content delivery to public4. Iterative processes

The Ottawa Model of Research Use

Decision-maker dilemmas impacting:evidence synthesis, knowledge translation and implementation

Legislative and funding require use of high rigor evidence for programs and practiceNew burden on practitioners to acquire, interpret, and apply evidenceBeyond skill and resourcesEvidence pool is often underdeveloped

Additional obstacles to evidence incorporation, chief among them:Timepolicy time does not coincide with research timeResourcesfundingavailable qualified labor poolvaluesCultureclinical, organizational, experiential, consensus on values that impact the reception of evidence

Common fallacies in evidence incorporationLocality effect “findings weren’t generate in my district with my people”Experience “we know it works”

Overcoming decision-maker dilemmas

Knowledge translation is designed to reduce the burden to decision-makersBut it is only as good as the preceding steps…

While it is recognized that each step from question formation through dissemination is integral to knowledge translation, we have less progress in the earlier phases

In a word:

Early phases of the knowledge generation and synthesis process must always consider:a. the desired dissemination, translation and implementation goals andb. decision-maker obstacles

Evidence synthesis and knowledge translation are inseparable

Question Refinement Synthesis DecisionTranslation

and Implementation

Socio-political context

Remainder….How do we adapt earlier phases so that knowledge translation reaches the right

people, at the right time, with the right message.?For Cochrane that means can and how can we adapt Cochrane Review methodologies

to increase the likelihood of successful dissemination?

In a word: is there is a viable alternative to the gold standard systematic review

The systematic review dilemma

1. The methodologies that make systematic reviews authoritative take time

…in years… not months, weeks, days or hours

2. Decision-makers often have trouble with evidence importation

…global evidence viewed skeptically…preference for local samples, data, evaluations

3. This leaves knowledge translation and decision-makers with unattractive options to proceed

….without high rigor information, …with a non-representative

sample of high rigor evidence, ….with low rigor evidence.

Cochrane’s Response

Adapts rapid review methodsRelies on KTA’s work on rapid evidence reviewsCreates new protocol that defines processes from question

development to implementation

IDEALLY

The rapid review is a mixed-method, interdisciplinary examination of evidence that directly addresses the “pull” needs of decision makers.

Findings of rapid reviews provide a compass to understand current practice and to set the course for outcome improvements in a specific environment.

Rapid reviews are tailored, fast, and implicitly designed to address implementation decisions.

Error, biases, and wasted time are threats to the value of a rapid review.

INCORPORATIONS TO IMPROVE EFFICIENCIES and IMPACT:

Knowledge management frameworksDiagnosing true knowledge gapsPrioritization criteria

Mixed methods and fieldsSurvey methods research and cognitive interviewingDecision makingUnderstanding of risk and probabilitySimulation research Implementation and evaluation researchPublic health and systems research

Business Lean knowledge workTeamingWork place 360°evaluations

Through our 2-year CIHR KTA research grant, we began to dive into the

realm of ‘rapid reviews.’

Modifying KTA to Cochrane Culture

Local Partnership – KTA Champlain LHIN needed assistance in developing

knowledge capacity Wanted energies directed to “push” activities;

primarily knowledge intelligence services as they required expedient evidence-based answers to help direct policy, implementation, and practice decisions

“Evidence Summaries” – a form of rapid review – was developed and was iteratively refined

A series of evidence summaries (n=19) were produced (~4-6wks each) in response to clinical and health services questions developed with LHIN managers and stakeholders

Scoping Exercise of Rapid Reviews (Ganann et al 2010)

Findings from 70 studies that authors looked at:

•RRs employ a variety of methodologies;

• Variety of different labels (rapid reviews, HTAs; evidence summaries; ultra rapid reviews)

• Seem to vary in depth of description of methods used to abbreviate the process

• Few discussed limitations (what was lost) or what bias was potentially introduced by using RR methods

No universally accepted definition of rapid review

• Closest we’ve come to a definition:– Rapid review (RR) is a type of literature review

produced using accelerated and streamlined systematic review (SR) methods• Using various means to short circuit the

system

KTA Project with the Champlain LHIN

Rapid Review Methods Essentially developed an abbreviated & accelerated

version of current SR methods Borrowing from Cochrane but having to make

certain concessions compared to a traditional SR to accommodate expedited turnaround time (4-6 weeks)

Out of which emerged an 8-step rapid review process developed iteratively across development of 19 ‘Evidence Summaries’ within a 24-month span

Rapid reviews: From start to finishit was an 8-stage approach…

1. Needs assessment (with stakeholders)2. Question development and refinement (PICO/TS)3. Protocol development and approval4. Systematic literature search (with limits)5. Screening and selection of studies (with limits)6. Narrative synthesis of included studies (no meta-analysis)7. Report production phase8. Ongoing follow-up with end users

Objective: high rigour, transparency, and usability

Rapid Reviews vs. Traditional Systematic Review

General Differences- Condensed timeline - Limits set on inclusion criteria of the

search including how grey literature is dealt with

- Post-hoc rethinking of eligibility criteria depending on volume and applicability of evidence

- Heavy reliance on systematic review evidence vs. Inclusion of primary studies (e.g., RCTs or observational studies)

General Similarities- Retains core values of being

transparent to facilitate replication- Adoption of standards to assess

quality of included studies- Share preference for highest

quality studies

Overview of our reports:• Conducted a total of 19 rapid evidence summaries to

date• 13 – focused on clinical initiatives

– 9 across the field of obstetrics/gynaecology• Better Outcomes Registry & Network Ontario (BORN Ontario) –

an initiative to increase capacity to plan and deliver improved maternal-child services

– 6 – focused on health systems/ health services initiatives• Effectiveness of Antimicrobial Stewardship Programs in Hospitals

– Requests came from various stakeholders (n=9)

• Snapshot of Rapid Review report

KTA Interviews – Knowledge Users’ Impressions of Rapid Reviews Approach

Stakeholders(CHAMPLAIN LHIN, BORN ONTARIO, OTTAWA HOSPITAL RESEARCH INSTITUTE

(OHRI), THE OTTAWA HOSPITAL (TOH))• Scientific Director of Performance Measurement – TOH• Chief Information Officer, Champlain LHIN • Former CEO, Champlain LHIN• Clinician, transport nurse, educator, and lactation consultant, BORN ONTARIO• Scientific Manager, BORN ONTARIO• Division Chief, Endocrinology and Metabolism, TOH• Clinical Investigator (MD), Clinical Epidemiology, OHRI, TOH• Senior Health Planning & Engagement Specialist at Champlain LHIN

Knowledge Users’ FeedbackUtility• Structure/ format of summaries is important – eyes

being drawn to key messages (capture reader’s attention)• Thought to be incredibly useful to have evidence at your

fingertips and summarized as it was• Format was user-friendly for non-researcher decisions makers

and clinicians• Format increased level of comfort with data for non-clinicians,

and validated that decisions are made based in part on the evidence provided

Knowledge Users’ Feedback-2• Notion that rapid reviews assists with

having wide group of stakeholders from various backgrounds to get on the same page in terms of understanding what the latest evidence is and what it means (levels out playing field in decision-making so process less prone to group think scenarios)

• Directly influenced practice change in local delivery of care (Example: summaries are being used to move forward on the regional plan on delivering diabetes care)

Knowledge Users’ Feedback-3

• Approach has helped build relationships between those creating the summaries and the implementers - key ingredient to ensure ongoing dialogue

• Notion that the evidence is coming from an “objective, third party” lens – and not from one particular angle/clinical slant, made the findings of the rapid reviews seem more credible

• Rapid reviews were used to help set the context for LHIN discussions, and were key to planning

• Also, used to lay the groundwork as part of a broader environmental scans; ultimately helping with prioritizing of LHIN projects

Knowledge Users’ Feedback-4Barriers• Policy and practice implementation difficult – taking general

evidence and applying it locally found to be challenging• System issues provide obstacles (for example, such as sites where

they only have OR times for 1-day/ week which then controls when surgeries can happen –even though that might not match with when it should be done)

• Unfavourable attitudes towards evidence requires behaviour change efforts (interventions difficult to implement)

• Taking the scientific evidence and translating it into a user-friendly format for a variety of end-users (dissemination science applied)

KTA Dissemination

• Reviews are disseminated on our website (www.ohri.ca/kta), to end-users directly, and when possible, through publication

KTA Publications

Most highly cited article in J of SRs

since published in February 2012;

5237 views/downloads

KTA Teaching & Training

• To date, we have delivered several workshops

Local Program DevelopmentThe Ottawa Hospital Technology Assessment Program (TOHTAP)• Hospital-based knowledge support

service with focus on rapid reviews• Knowledge synthesis + local data +

economical evaluation• Inaugural year with possible long-

term funding commitment (OHRI, TOH Performance Management & Patient Safety groups)

Cochrane Rapid Response....

International Program DevelopmentCochrane Innovations• Newly established program by

Cochrane Collaboration with primary goal of providing a mechanism by which Collaboration can respond to requests for commissioned reviews

Cochrane Response

• Rapid reviews are one type of commissioned review, for which the Cochrane version is referred to as “Cochrane Response”

• Currently, small global consortium has been assembled to develop materials to assist in the conduct of Cochrane Responses – to date relied heavily on our KTA approach

• Authors will be experienced Cochrane authors and active members of the Cochrane Response Consortium

• Each requires at least two reviewers, one information retrieval specialist, content expertise, and the necessary input of the client

• Process adds in 24-hour peer review periods with access to input from across the many content and methods experts that make up the collaboration (Cochrane’s 55 Review Groups, 14 Fields, and 12 Methods Groups)

• Each Response is placed on an 8-week schedule and has intermediate deliverables scheduled throughout this period

• There is an intention to publish these rapid reviews through the Cochrane Library, making them available for other decision-makers

Cochrane Response

11-stage process developed based on Cochrane principles, and a review timeline of 8-weeks

Cochrane Response Process

1. Topic Refinement 2. Contracts 3. Problem

Statement4. Protocol

Development

5. Literature Search

6. Screening & Study Selection

7. Data Mining & Extraction 8. Synthesis

9. Report Contruction

10. Contingency Phase

11. Additional Products or

Services

In SummaryIDEALLY

The rapid review is a mixed-method, interdisciplinary examination of evidence that directly addresses the “pull” needs of decision makers.

Findings of rapid reviews provide a compass to understand current practice and to set the course for outcome improvements in a specific environment.

Rapid reviews are tailored, fast, and implicitly designed to address implementation decisions.

Error, biases, and wasted time are threats to the value of a rapid review.

INCORPORATIONS TO IMPROVE EFFICIENCIES and IMPACT:

Knowledge management frameworksDiagnosing true knowledge gapsPrioritization criteria

Mixed methods and fieldsSurvey methods research and cognitive interviewingDecision makingUnderstanding of risk and probabilitySimulation research Implementation and evaluation researchPublic health and systems research

Business Lean knowledge workTeamingWork place 360°evaluations

Evidence and decisions: variables at play

Decision maker characteristics: Goals, Position, Agency, Beliefs, Experiences

Type of decision:•Funding•Service•Refinement or change of practice

Processes used to refine PICO/ST

Resources for evidence synthesis:• Qualified labor•Training integrity•time•funding•Generalizability•Local context

Distance between evidence and current practice

Access and Management of

stakeholder groups

Consensus on utility of question

Question Refinement Synthesis Decision Translation and Implementation

Socio-political context

Diversity of Stakeholder group

Competing priorities:Introduce funding

Do nothing

Translation activities windows missed

US Representative: VA 11th District

Connelly

Cochrane Collaboration

Cochrane Innovations

Targeted patient population within Northern Virginia’s 11th District

US CONGRESS

Ottawa Hospital Research Institute:

uOttawa EPCKnowledge to Action Canada

Extensions of sustainable relationship into Virginia’s 8th and 10th Districts

Congressman Moran

VA 8th District

Congressman Wolf

VA 10th District

INOVA Health Systems

G e o r g e M a s o n U n i v e r s i t y

Chief Medical Officer CMO’s hospitals and Care Centers

I N O V A H E A L T H S Y S T E M S

Cochrane Policy College

Dept of Medicine

Core EDD team

AHRQEffective care

programEPCs, DEcIDE

Rigor requirements needed for all aspects of knowledge generation, rapid review, and dissemination

Empirical assessment of outcome improvementDo those receiving intervention perform better

Retain knowledgeChange practiceShare information

Are practices influencedAre systems refinedIs money saved

How do moderators at each stage affect reception of knowledge

What happens when you manipulate conditionsAssert funding unto schemeIntroduce alternative therapySkip one or more of the steps

What is the best format of findings display (see Dr. Probability next)

Encourage use of limited resources for high utility questions

Does partnership and ongoing delivery encourage change over inserting external evidence?

Decision making trainingDo decision makers understand evidence?Are they able to use it in light of competing priorities? Can they use evidence to ensure greatest clinical, fiscal, social utility?Did change happen?Are outcomes betterAre decisions

Next steps: parsing the variables and assessing their impact so that knowledge translation is truly calibrated for:Type of decisionType of consumerCurrent contextual factors (local vs. global)

Manipulate parametersConduct random assignment to knowledge productsConduct implementation and outcome evaluation

Mixed methods

But what about the medicalisation of

1000 people?