GRADE Workshop: Agenda Introduction. Guideline development ... · 2 GRADE Workshop –Ispra–11-12...

1

Disc laimer: The contents of this presentation are the views of the author and do not necessarily represent an offic ial position of the European C ommission. © European Union, 2013

Determinants of quality of evidence:

What can lower the evidence? (I)

Jesús López Alcalde

Public Health Policy Support Unit

Institute for Health and Consumer Protection

(JRC-IHCP)

Joint Research Centre

The European Commission’s

in-house science service

2 GRADE Workshop – Ispra– 11-12 December 2013

GRADE Workshop: Agenda

9:00 - 10:15 Introduction. Guideline development process and the GRADE approach

10:15-11:00 Types of questions. Framing a question: PICO question. Exercise

11:00-11:15 Coffee

11:15-12:00 Choosing outcomes. Relative importance of outcomes. Exercise

12:00-12:45 Study designs. Exercise

12:45-13:30 Lunch

13:30-14:30 Search of the literature. Exercise

14:30-15:00 Determinants of quality of evidence: What can lower the evidence? (I)

15:00-15:15 Coffee

15:15-17:00 Determinants of quality of evidence: What can lower the evidence? (II) Exercise

9:00-11:15 Determinants of quality of evidence: What can lower the evidence? (III)Determinants of quality of evidence: What can upgrade evidence?

11:15-11:30 Coffee

11:30-13:00 Going from the evidence to the recommendation. Exercise

13:00-13:45 Lunch

13:45-16:00 Using the Guideline Development Tool (GDT) software. ExerciseFeedback and conclusions

Recommendation

Evidence synthesis

P

I

C

O

Outcome

Outcome

Outcome

Outcome

Critical

Important

Critical

NotSummary of findings &

estimate of effect for

each outcome

Grade overall

quality of evidence

across outcomes based on

lowest quality

of critical outcomes

Randomization raises

initial quality

RCTs: high

Observational: low

1. Risk of bias

2. Inconsistency

3. Indirectness

4. Imprecision

5. Publication biasGra

de d

own

Gra

de u

p 1. Large effect

2. Dose response

3. Opposing bias &

Confounders

Very low

Low

Moderate

High

Grade recommendations

(Evidence to Recommendation)

• For or against (direction) ↓↑• Strong or conditional/weak (strength)

By considering balance of consequences

(evidence to recommendations):

� Quality of evidence

� Balance benefits/harms

� Values and preferences (equity)

� Resource use (cost, feasibility)

� Acceptability

Formulate Recommendations (↓↑ | ⊕…)

“The panel recommends that ….should...”

“The panel suggests that ….should...”

“The panel suggests to not ...”

“The panel recommends to not...”

Transparency, clear, actionable

Guideline

Slide kindly offered by the GRADE Working Group


• Common misunderstandings

• GRADE considers two definitions of ‘‘quality of evidence’’ (QoE)

• GRADE specifies four levels for the quality of evidence of each outcome

across studies

• GRADE’s approach to rating the quality of evidence of each outcome

across studies

Outline


Evidence ≠ Expert opinion

Assesing the Quality of the Evidence ≠ Making a recommendation

Common misunderstandings:


• Their experience with patients

• Their understanding of biology and mechanism of the problem

• Their knowledge of preclinical and early clinical research

• Their knowledge of the results of clinical research (randomised clinical

trials and observational studies)

Expert opinion is not evidence

Opinion

Please, answer: What is the basis of the opinion of experts?

2


Opinion

Guideline developers should always:

• Engage experts to define the PICO question

• Engage the experts to interpret the evidence

• Uncover and make clear the evidence that underlies the experts’ opinions

• Rate the quality of that evidence

How to deal with expert opinion in guidelines:

Quality of that evidence

Evidence

Uncover

Assess


Example: Expert opinion is not evidence

JCE 64 (2011) 401-406

Expert opinion Components

Tight control of sugar in blood will reduce the long-term risk of developingkidney disease […]

‘‘I institute tight control on every patient - I believe

they all deserve the best possible treatment- so I

have a lot of experience with this.

I have many patients who have been with me for a decade, or even several decades, and who take their

medicine faithfully and have great blood sugars. These patients also have very few complications. On

the other hand, I have a lot of patients who have

terrible control and develop complications early on.

Also, there are a lot of studies showing that tightcontrol reduces the risk of complications.’’

Personal experience

Clinical research


Evidence ≠ Expert opinion

Assesing the Quality of the Evidence ≠ Making a recommendation

Common misunderstandings:


GRADE explicitly separates two steps:

• To assess the quality of the evidence

• To make a recommendation and grade its

strength

Quality of evidence ≠ Strength of recommendation

Recommendation

Quality of the evidence

JCE 64 (2011) 401-406

A particular quality of evidence does notnecessarily imply a particular strength ofrecommendation

Balance benefits/harms

Values and preferences

Resource use (cost)

Equity Feasibility

Acceptability

11 GRADE Workshop – Ispra– 11-12 December 2013 12 GRADE Workshop – Ispra– 11-12 December 2013



• GRADE specifies four levels for the quality of evidence

• GRADE’s approach to rating the quality of evidence for each outcome

across studies

Outline

3


GRADE considers two definitions of ‘‘quality of evidence’’ (QoE)

As part of a systematic review

As part of making a recommendation in guidelines

Our level of confidence that theestimate of the effect is…

… correct(close to the truth)

… adequate tosupport a particular recommendation


Systematic reviews seldom include processes required for makingrigorous recommendations:

• Authors of systematic reviews are, generally, not in a position to weigh the

trade-offs between the desirable and undesirable consequences of adhering to

a recommendation.

Relevant stakeholders included in are in a better position to make thesejudgments.

Reason for two different definitions of QoE in GRADE




• GRADE specifies four levels for the quality of evidence for eachoutcome across studies

• GRADE’s approach to rating the quality of evidence for each outcome

across studies

Outline


Level of evidence

Our level of confidence in the effect estimate

Explanation

High We have high confidenceThe true effect lies close to the

estimate of the effect

ModerateWe have moderateconfidence

The true effect is likely to be close to the estimate of the

effect, but there is a possibility

that it is substantially different

Low We have limited confidenceThe true effect may be substantially different from

the estimate of the effect

Very lowWe have very littleconfidence

The true effect is likely to be substantially different from

the estimate of effect

GRADE: 4 levels for the QoE for each outcome across studies




• GRADE specifies four levels for the quality of evidence for each outcome

across studies

• GRADE’s approach to rating the quality of evidence for eachoutcome across studies

Outline

Evidence synthesis

P

I

C

O

Outcome

Outcome

Outcome

Outcome

Critical

Important

Critical

Not

Randomization raises

initial quality

RCTs: high

Observational: low

1. Risk of bias

2. Inconsistency

3. Indirectness

4. Imprecision

5. Publication biasGra

de d

own

Gra

de u

p 1. Large effect

2. Dose response

3. Opposing bias &

Confounders

Very low

Low

Moderate

High

Slide modified from the GRADE Working Group

• GRADE’s approach to rating

• the quality of evidence for each outcome across studies

4


Example: Quality of the evidence for an outcome across studies

JCE 66 (2013) 151e157

Ann Surg Oncol 2007;14:1825e34.


GRADE: rating the QoE for each outcome across studies

Identify thestudy design

Randomisedtrials

Observationalstudies

Initial quality of a body of

evidence

High ++++

Low ++

Lower if Higher ifFinal quality of a body of

evidence

High ++++

Moderate +++

Low ++

Very low +

Risk of Bias:-1 Serious

-2 Very serious

Indirectness:-1 Serious

-2 Very serious

Inconsistency-1 Serious

-2 Very serious

Imprecision-1 Serious

-2 Very serious

Publication bias-1 Likely

-2 Very likely

Large effect+1 Large

+2 Very large

Dose response+1 Evidence of

a gradient

All plausible residual confounding:

+1 would reduce a

demonstrated effect+1 would suggest a

spurious effect if no effect was observed


Risk of bias


GRADE Workshop: Agenda

9:00 - 10:15 Introduction. Guideline development process and the GRADE approach

10:15-11:00 Types of questions. Framing a question: PICO question. Exercise

11:00-11:15 Coffee

11:15-12:00 Choosing outcomes. Relative importance of outcomes. Exercise

12:00-12:45 Study designs. Exercise

12:45-13:30 Lunch

13:30-14:30 Search of the literature. Exercise

14:30-15:00 Determinants of quality of evidence: What can lower the evidence? (I)

15:00-15:15 Coffee

15:15-17:00 Determinants of quality of evidence: What can lower the evidence? (II) Exercise

9:00-11:15 Determinants of quality of evidence: What can lower the evidence? (III)Determinants of quality of evidence: What can upgrade evidence?

11:15-11:30 Coffee

11:30-13:00 Going from the evidence to the recommendation. Exercise

13:00-13:45 Lunch

13:45-16:00 Using the Guideline Development Tool (GDT) software. ExerciseFeedback and conclusions


GRADE: rating the QoE for each outcome across studies


Randomisedtrials



evidence

High ++++

Low ++


evidence

High ++++

Moderate +++

Low ++

Very low +


-2 Very serious


-2 Very serious


-2 Very serious


-2 Very serious


-2 Very likely


+2 Very large


a gradient


+1 would reduce a



Outline• What is bias?

• Cochrane ‘Risk of bias’ tool for randomized controlled trials

• Assessing risk of bias: CHOC-ATT trial

• Exercise

5

In a study, what is bias?Systematic error or deviation from the truth in the results of a study




• Exercise

7 evidence based factors to address:1. Random sequence generation

2. Allocation concealment

3. Blinding of participants and personnel

4. Blinding of outcome assessment

5. Incomplete outcome data

6. Selective reporting

7. Other bias

Consult the Cochrane Handbook

Cochrane ‘Risk of bias’ tool for randomized clinical trials

Review authors’ judgement:

Support for judgement

Risk of bias summary Risk of bias graph

6




• Exercise

Feeling sleepy?

Attention lagging?

Maybe you

need…

Sle

ep

ing

kid

in N

SC

by

ro

flto

sh w

ww

.fli

ckr.

com

/ph

oto

s/2

11

31

32

7@

N0

0/2

10

07

63

00

9

Any questions?

Exercise

JRC xxxxx – © European Union, 2013Disc laimer: The contents of this presentation are the views of the author and do not necessarily represent an offic ial position of the European C ommission.

Sm

all

Ch

oco

late

Fo

un

tain

by

Ale

xan

de

r K

lin

k h

ttp

://c

om

mo

ns.

wik

ime

dia

.org

/wik

i/Fi

le:S

ma

ll_

Ch

oco

late

_Fo

un

tain

.JP

G

CHOC-ATT Trial

Does CHOColate improve ATTention and

reduce sleepiness during workshops?


JRC is planning the CHOC-ATT trial

Aim of the trial:

To find out if eating chocolate (which contains sugar and caffeine) can:

• improve attention during workshops; and

• reduce sleepiness during workshops

You will be recruited as participants in the trial

Scenario


Question 1: How to investigate whether chocolate has any effect?

Create 2 groups, one receiving chocolate and the other receiving nothing

Step 1: Generating an allocation sequence (I)

7


Question 2: How to divide the participants in the trial into two groups?

What do you think about the following methods:

Non-random:• Left side/right side of the room

Quasi-random:• Alternancy

Stratifying by known confounders:• Probably sleepy people balanced

between groups

Step 1: Generating an allocation sequence (II)

PROBLEM:

There may be some important

factors that are not balanced

between the groups, and might

affect the outcome of the trial.

Question 3:

How to generate two groups with all

the relevant factors that might

affect the outcome of the trial

balanced between the groups?

Random sequence


Everyone stands an equal chance of being in each group, which balances all

known and unknown confounders between two groups

Random sequence

Control group: Doing nothing

Intervention group


We need randomised clinical trials!!!

Put a RCT in your life!

http://www.randomiseme.org/


Question 4: How to generate a random sequence?

• Computer-generated sequence

• Random number table

• Drawing lots or numbers out of a hat

• Rolling dice

Step 1: Generating an allocation sequence (III)


Example: method to assign random numbers to the participants

Use an Excel spreadsheet

8


Important message

Follow the path dictated by chance!!!!


The randomisation schedule is now ready for use by recruiters in the trial.

I will stick the schedule to the wall in order to follow it.

Step 2: Allocation concealment

QUESTION 5: Consequences of having this list on the wall in the recruitment office?

ANSWER: It may affect the recruitment

QUESTION 6: How to overcome this problem?

ANSWER: Allocation concealment

Protect the random sequence (and thus prevent any interference or

manipulation of the sequence by the trial recruiter or participants)


Central randomisation serviceSuch as a pharmacy, a central office contacted by phone or email, or an

automated computer system.

Sequentially numbered, sealed, opaque envelopes or drug containers

Question 7: How allocation concealment might be achieved?

Question 8: Do you identify any problems with this method to conceal the allocation?

Sequentially numbered?

Sealed envelopes?

Opaque envelopes?

Let’s implement the sequence

Sources of biasRandom sequence generation

Allocation concealmentSelection

Performance

Detection

Attrition

Reporting

Target population

Allocation

Intervention group Control group

Outcome

assessment

Outcome

assessment

Publication of study outcomes

Selection bias: Systematic differences

between baseline characteristics of the

groups that are compared.


Please, open your envelopes. Who have number 1 (Chocolate)?

Please, eat the chocolate

Perfect, both the investigators and participants know which participants

received chocolate and which did not.

Step 4: Blinding

ANSWER:

Performance bias:Example: participants who did not receive chocolate might buy chocolate or

coffee to stay alert. This might genuinely affect the results of the trial.

Detection bias:Example: participants might be more likely to report negative outcomes if

they are unhappy because they did not receive any chocolate.

Question 9: Why this could be a problem?

Sources of bias

Selection

Performance

Detection

Attrition

Reporting

Target population

Allocation


Outcome

assessment

Outcome

assessment


Systematic differences between both groups in the care that is provided, or

in exposure to factors other than the interventions of interest

Systematic differences between groups in how the outcome is evaluated

9


ANSWER: Performance bias: Blinding of study participants to prevent they know which intervention are

receiving

Blinding of study personnel to prevent they know which intervention they

are administering

Detection bias: Blinding outcome assessors to prevent they know which intervention was

received

Question 10: How to overcome performance and detection bias? Sources of biasRandom sequence generation


Performance

Detection

Attrition

Reporting

Target population

Allocation


Outcome

assessment

Outcome

assessment


Blinding of

participants, personnel

Blinding of outcome

assessment


Investigators are absent when envelopes are opened and the intervention

distributed

Participants in the control group should receive a placebo that is

indistinguishable from chocolate in terms of taste, look, smell, packaging,

etc.

Objective outcome measures could be used that are not affected by blinding

Question 10: How blinding could be implemented in the CHOC-AT trial?


Systematic differences between the groups of the study regarding the

presence of incomplete outcome data.

There are two reasons for incomplete outcome data in clinical trials:

• Exclusions: some participants are omitted from reports of analyses,

despite outcome data being available.

• Attrition: outcome data are not available

Attrition bias

Control group: Doing nothing

Intervention group

?

?

?



Performance

Detection

Attrition

Reporting

Target population

Allocation


Outcome

assessment

Outcome

assessment


Blinding of


Blinding of outcome

assessment

Incomplete outcome

data

10


Considerations:

• How much data is missing from each group?

• Why is it missing?

• How were the data analysed?

Incomplete outcome bias

Selective outcome

reporting

Target population

Allocation


1.- Attention

2.- Sleepiness

1.- Attention

2.- Sleepiness

Cholesterol levels

Question 11: What do you think about this?

We have conducted the CHOC-ATT trial

Study aim: to find out if eating chocolate:• Improve attention; and • Reduce sleepiness

Cholesterol levels


The selection of a subset of the original variables recorded, on the basis of

the results, for inclusion in publication of trials (Hutton 2000)

The particular concern:

• Statistically non-significant results might be selectively withheld from

publication

Selective outcome reporting

Question 12: How frequent is selective outcome reporting?

When comparing publications with protocols, 62% of trials had at least

one primary outcome that was changed, introduced or omitted

Chan AW et al. JAMA 2004;291:2457-65



Performance

Detection

Attrition

Reporting

Target population

Allocation


Outcome

assessment

Outcome

assessment


Blinding of


Blinding of outcome

assessment

Incomplete outcome

data

Selective outcome

Reporting

Acknowledgements

• Higgins JPT, Altman DG, Sterne JAC (editors). Chapter 8: Assessing risk

of bias in included studies. In: Higgins JPT, Green S (editors).

Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0

[updated March 2011]. The Cochrane Collaboration, 2011. Available from

www.cochrane-handbook.org

• Materials and exercise adapted from the Cochrane Trainer's Network

Any questions?

Exercise


11

Disc laimer: The contents of this presentation are the views of the author and do not necessarily represent an offic ial position of the European C ommission. © European Union, 2013

What can lower the evidence? (II)

Indirectness

Jesús López Alcalde

Public Health Policy Support Unit

Institute for Health and Consumer Protection

(JRC-IHCP)

Joint Research Centre

The European Commission’s

in-house science service


What can modify the evidence?


Randomisedtrials



evidence

High ++++

Low ++


evidence

High ++++

Moderate +++

Low ++

Very low +


-2 Very serious


-2 Very serious


-2 Very serious


-2 Very serious


-2 Very likely


+2 Very large


a gradient


+1 would reduce a




1. Indirect comparison

2. Indirect population, intervention or outcome

There are two types of indirectness:


Indirect comparison

A BversusThe comparison we aim to answer:

The comparisons we find: A

B

versus C

Cversus

Statin A

Statin BversusThe comparison we aim to answer:

The comparisons we find: Statin A

Statin B

versus Placebo

Placeboversus


1. Indirect comparison

2. Indirect population, intervention or outcome

There are two types of indirectness:


The PICO question being addressed by the guideline panel is different from

the available evidence regarding the population, intervention, or an

outcome

Indirect population, intervention, or outcome

12

Randomized controlled trials: low external validity

Women

Pediatrics

Foto obtained from Wikimedia CommonsFoto obtained from my camera 68 GRADE Workshop – Ispra– 11-12 December 2013

Please, consider "everyday patients", not "ideal patients"

66% of everyday depresive patientswould be excluded from RCts


Considering the importance of a surrogate outcome:

1st: Rate the importance of the patient-importantoutcome for which the surrogate is a substitute

2nd: Rate down the quality of evidence forindirectness of outcome


Quality of evidence may decrease when:

• Substantial differences exist between the P, the I, or the O measured in

the studies and those under consideration in a guideline.

• Head-to-head comparisons are lacking.

Key points

Any questions?

Exercise


Date post:	02-Jun-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

GRADE Workshop: Agenda Introduction. Guideline development ... · 2 GRADE Workshop –Ispra–11-12...

Documents