Why Grade Why Grade Recommendations?Recommendations?

• strong recommendationsstrong recommendations– strong methods strong methods – large precise effect large precise effect – few down sides of therapyfew down sides of therapy

• weak recommendationsweak recommendations– weak methodsweak methods– imprecise estimateimprecise estimate– small effectsmall effect– substantial down sidessubstantial down sides

Why Grade Why Grade Recommendations?Recommendations?

• focus on the evidencefocus on the evidence

• alternate practitioner behavioralternate practitioner behavior– apply uniformly orapply uniformly or– examine evidence themselvesexamine evidence themselves– consider patient circumstancesconsider patient circumstances– explore with the patientexplore with the patient

What Are We Grading?What Are We Grading?

• methodological quality of evidencemethodological quality of evidence– likelihood of biaslikelihood of bias

• strength of recommendationsstrength of recommendations– must do to might domust do to might do

Grade of Evidence for Grade of Evidence for Specific Question Specific Question

• too vague: alendronate in osteoporosistoo vague: alendronate in osteoporosis

• patients – post-menopausal womenpatients – post-menopausal women

• intervention intervention – daily alendronate, dose 10 to 20 mg.daily alendronate, dose 10 to 20 mg.

• outcome – non-vertebral fracturesoutcome – non-vertebral fractures

What Influences Grade?What Influences Grade?

• study designstudy design– basicbasic– detailed design and executiondetailed design and execution

• consistencyconsistency

• directnessdirectness

• reporting biasreporting bias

Methodological QualityMethodological Quality

• study designstudy design– randomizationrandomization– quasi-randomizationquasi-randomization– observational studyobservational study

• detailed design and executiondetailed design and execution– concealmentconcealment– balance in known prognostic factorsbalance in known prognostic factors– intention to treat principle observedintention to treat principle observed– blindingblinding– completeness of follow-upcompleteness of follow-up

Methodologic QualityMethodologic Quality

• consistency of resultsconsistency of results

• if inconsistency, look for explanationif inconsistency, look for explanation– patients, intervention, outcome, patients, intervention, outcome,

methodsmethods

• no clear thresholdno clear threshold– size of effect, confidence intervals, size of effect, confidence intervals,

statistical significancestatistical significance

Relative Risk of Conversion to Sinus RhythmAmiodarone vs Placebo or Digoxin or CCB

Favours Control Favours Amiodarone

Cowan 1986 1.11 (0.78 to 1.58)Noc 1990 18.0 (1.17 to 276)

Capucci 1992 0.77 (0.37 to 1.62)Cochrane 1994 1.15 (0.91 to 1.44)Donovan 1995 1.05 (0.69 to 1.60)

Hou 1995 1.29 (0.97 to 1.72)Kondili 1995 1.33 (0.71 to 2.47)Galve 1996 1.13 (0.84 to 1.52)

Kontoyannis 1998 1.42 (1.08 to 1.85)Bellandi 1999 1.41 (1.15 to 1.72)

Cotter 1999 1.43 (1.15 to 1.80)Kochiadakis 1999 1.46 (1.19 to 1.78)

Bianconi 2000 2.04 (0.19 to 22.00)Galperin 2000 33.70 (2.08 to 546)

Hohnloser 2000 3.13 (1.50 to 6.70)Joseph 2000 1.32 (0.95 to 1.80)Natale 2000 5.12 (2.60 to 10.00)

Peukurinen 2000 2.45 (1.49 to 4.02)Vardas 2000 2.01 (1.55 to 2.60)Villani 2000 4.75 (1.60 to 14.00)

Cybulski 2001 1.87 (1.37 to 2.55)0.1 1 10 100

n = 83

n = 95

n = 203

n = 85

n = 120

n = 34

n = 24

n = 40

n = 30

n = 64

n = 39

n = 42

n = 100

n = 42n = 120

n = 100

n = 204

n = 75

n = 62

n = 208

n = 160

Relative Risk of Conversion to Sinus RhythmAmiodarone vs Placebo or Digoxin or CCB

Favours Control Favours Amiodarone

Bianconi 2000 2.04 (0.19 to 22.00)Galperin 2000 33.70 (2.08 to 546)

Hohnloser 2000 3.13 (1.50 to 6.70)Natale 2000 5.12 (2.60 to 10.00)Villani 2000 4.75 (1.60 to 14.00)

Pooled Estimate 4.33 (2.76 to 6.77)

Cowan 1986 1.11 (0.78 to 1.58)Noc 1990 18.0 (1.17 to 276)

Capucci 1992 0.77 (0.37 to 1.62)Cochrane 1994 1.15 (0.91 to 1.44)Donovan 1995 1.05 (0.69 to 1.60)

Hou 1995 1.29 (0.97 to 1.72)Kondili 1995 1.33 (0.71 to 2.47)Galve 1996 1.13 (0.84 to 1.52)

Kontoyannis 1998 1.42 (1.08 to 1.85)Bellandi 1999 1.41 (1.15 to 1.72)

Cotter 1999 1.43 (1.15 to 1.80)Kochiadakis 1999 1.46 (1.19 to 1.78)

Joseph 2000 1.32 (0.95 to 1.80)Peukurinen 2000 2.45 (1.49 to 4.02)

Vardas 2000 2.01 (1.55 to 2.60)Cybulski 2001 1.87 (1.37 to 2.55)

Pooled Estimate 1.40 (1.25 to 1.57)0.1 1 10 100

AF Duration > 48 hrs

AF Duration =/< 48 hrs

n = 83n = 95

n = 203n = 85

n = 120

n = 34

n = 24n = 40

n = 30n = 64n = 39

n = 42n = 100n = 42n = 120n = 100n = 204n = 75

n = 62

n = 208n = 160

Directness of EvidenceDirectness of Evidence

• indirect treatment comparisonsindirect treatment comparisons– interested in A versus Binterested in A versus B– have A versus C and B versus Chave A versus C and B versus C

• alendronate vs risedronatealendronate vs risedronate– both versus placebo, no head-to-headboth versus placebo, no head-to-head

Four Levels of Four Levels of “Directness”“Directness”

• patients meet trials’ eligibility criteria patients meet trials’ eligibility criteria

• not included, but no reason to questionnot included, but no reason to question– slight age difference, comorbidity, raceslight age difference, comorbidity, race

• some question, bottom line applicablesome question, bottom line applicable– valvular atrial fibrillationvalvular atrial fibrillation

• serious question about biologyserious question about biology– heart failure trials applicability to aortic stenosisheart failure trials applicability to aortic stenosis

Levels of DirectnessLevels of Directness

• interventionsinterventions– same drugs and dosessame drugs and doses– similar drugs and dosessimilar drugs and doses– same class and biologysame class and biology– questionable class and biologyquestionable class and biology

• outcomesoutcomes– same outcomessame outcomes– similar (duration, quality of life)similar (duration, quality of life)– less breathlessness for role functionless breathlessness for role function– laboratory exercise capacity for q of lifelaboratory exercise capacity for q of life

Magnitude, Precision, Magnitude, Precision, Reporting BiasReporting Bias

• magnitude not generally part of qualitymagnitude not generally part of quality– but very large magnitude can upgradebut very large magnitude can upgrade

• precision not generally part of qualityprecision not generally part of quality– but sparse data can lower qualitybut sparse data can lower quality

• reporting biasreporting bias– high likelihood can lower qualityhigh likelihood can lower quality

Grading SystemGrading System

• high qualityhigh quality well done RCTwell done RCT

• intermediateintermediate quasi-RCTquasi-RCT

• lowlow well done observationalwell done observational

• insufficient insufficient anything elseanything else

Moving DownMoving Down

• study execution– serious flaws can lower by one level– fatal flaws can lower by two levels

• consistency– important inconsistency can lower by one level

• directness of evidence– some uncertainty re relevance lower by one level– major uncertainty re relevance lower by two levels

• selection bias – strong suspicion lower by 1 level

Moving UpMoving Up

• very strong association, up 2 levels– insulin in diabetic ketoacidosis

• strong, consistent association with no plausible confounders, up 2 levels – fluoride for preventing cavities

• strong association can move up 1 level– ? HRT ?

Risk/Benefit tradeoffRisk/Benefit tradeoff• aspirin after myocardial infarctionaspirin after myocardial infarction

– meta-analysis of high quality RCTsmeta-analysis of high quality RCTs– clear positive treatment effectclear positive treatment effect– unequivocal recommendation to treat, unequivocal recommendation to treat,

uniform practiceuniform practice

• accelerated TPA vs streptokinase after accelerated TPA vs streptokinase after MIMI– very large single trialvery large single trial– clear positive effectclear positive effect– uncertainty whether to treat, varying uncertainty whether to treat, varying

practicepractice

Risk/Benefit tradeoffRisk/Benefit tradeoff

• Aspirin after myocardial infarctionAspirin after myocardial infarction– 25% reduction in relative risk25% reduction in relative risk– side effects minimal, cost minimalside effects minimal, cost minimal– benefit obviously much greater than benefit obviously much greater than

risk/costrisk/cost

• TPA vs streptokinase after MITPA vs streptokinase after MI– 12% reduction in relative risk12% reduction in relative risk– increased rate of intracranial hemorrhageincreased rate of intracranial hemorrhage– large increase in costlarge increase in cost– benefit vs risk/cost a judgment callbenefit vs risk/cost a judgment call

Strength of Strength of RecommendationsRecommendations

Aspirin after MI – do itAspirin after MI – do it

TPA rather than SK in MI TPA rather than SK in MI -- probably do it-- probably do it

-- probably don’t do it-- probably don’t do it

Grade of Grade of RecommendationsRecommendations

•do itdo it•probably do itprobably do it• toss-uptoss-up•probably don’t do itprobably don’t do it•don’t do itdon’t do it

Judgment: Benefits vs Risks/CostsJudgment: Benefits vs Risks/Costs

• seriousness of outcomeseriousness of outcome• magnitude of effectmagnitude of effect• precision of treatment effectprecision of treatment effect• risk of target eventrisk of target event• risk of adverse eventsrisk of adverse events• cost of therapycost of therapy• valuesvalues

Grades TranslationsGrades Translations• do it - 90% would choose itdo it - 90% would choose it• possibly do it – 60% to 90% would possibly do it – 60% to 90% would

choosechoose• toss-up – 40% to 60% would choosetoss-up – 40% to 60% would choose• possibly don’t – 60% to 90% would possibly don’t – 60% to 90% would

declinedecline• don’t - 90% would declinedon’t - 90% would decline

ConclusionConclusion• challenges in gradingchallenges in grading

– balancing simplicity and complexitybalancing simplicity and complexity– judgement always requiredjudgement always required

• consistent grading system requiredconsistent grading system required

• must consider study design, execution, consistency, must consider study design, execution, consistency, directness, reporting biasdirectness, reporting bias– magnitude, precision, only when extrememagnitude, precision, only when extreme

• balance of benefits and risks/costbalance of benefits and risks/cost– magnitude of effects; precision of effects; values and magnitude of effects; precision of effects; values and

preferencespreferences