Two questions in grading Two questions in grading recommendationsrecommendations

• Are you sure?Are you sure?– Yes: Grade 1Yes: Grade 1– No: Grade 2No: Grade 2

• What is the methodological quality of the What is the methodological quality of the underlying evidenceunderlying evidence– High quality: Grade AHigh quality: Grade A– Intermediate quality: Grade BIntermediate quality: Grade B– Poor quality: Grade CPoor quality: Grade C

What is the methodological quality What is the methodological quality of the underlying evidence?of the underlying evidence?

• High quality evidence: Grade AHigh quality evidence: Grade A- Randomized trials, few limitations- Randomized trials, few limitations

• Intermediate quality evidence: Grade BIntermediate quality evidence: Grade B– Randomized trials with important limitationsRandomized trials with important limitations

• varying results (heterogeneity)varying results (heterogeneity)• major methodological flawsmajor methodological flaws• total sample size in all studies combined under 100total sample size in all studies combined under 100

• Poor quality: Grade CPoor quality: Grade C– Observational studiesObservational studies

B: Randomized trials, inconsistent resultsB: Randomized trials, inconsistent results

C: Observational studiesC: Observational studies

• Benefits vs Risks/costs TradeoffBenefits vs Risks/costs TradeoffI: Clear that benefits do/don’t outweight risks/costI: Clear that benefits do/don’t outweight risks/cost

II: Benefit vs risk/cost tradeoff not clearII: Benefit vs risk/cost tradeoff not clear

Generalizing resultGeneralizing result

• ASA in unstable angina 50% RRRASA in unstable angina 50% RRR– trials of patients up to 80 years oldtrials of patients up to 80 years old– no trials in those over 80 -- still Grade A, or C?no trials in those over 80 -- still Grade A, or C?

• Warfarin in atrial fibrillationWarfarin in atrial fibrillation– lots of trials in non-valvular atrial fibrillationlots of trials in non-valvular atrial fibrillation– no trials in valvular a fib -- still Grade A, or C?no trials in valvular a fib -- still Grade A, or C?

• IV heparin for pregnant women with DVTIV heparin for pregnant women with DVT– lots of trials in non-pregnantlots of trials in non-pregnant– no trials in pregnant -- still Grade A, or Cno trials in pregnant -- still Grade A, or C

What do we mean by “are you sure?What do we mean by “are you sure?

• 1st: Is there uniformity in assessment of risk/benefit 1st: Is there uniformity in assessment of risk/benefit in your consensus group and in the communityin your consensus group and in the community– If If yesyes, probably Grade 1, probably Grade 1– If If nono, probably Grade 2, probably Grade 2

• 1st: Is the risk/benefit clear1st: Is the risk/benefit clear– Grade 1: Benefit clearly greater than risk Grade 1: Benefit clearly greater than risk oror risk risk

clearly greater than benefitclearly greater than benefit– Grade 2: Risk/benefit uncertainGrade 2: Risk/benefit uncertain

What do we mean by “are you sure”What do we mean by “are you sure”• Consider patient valuesConsider patient values

• Example: Different values of stroke/bleedingExample: Different values of stroke/bleeding

• 3rd: Would (almost) all your patients make the same choice?3rd: Would (almost) all your patients make the same choice?– YesYes: Grade 1: Grade 1

– NoNo: Grade 2: Grade 2

• 4th: Would a decision aid be useful and worthwhile?4th: Would a decision aid be useful and worthwhile?– No, no needNo, no need: Grade 1: Grade 1

– Yes, neededYes, needed: Grade 2: Grade 2

What do we mean by “are you sure?What do we mean by “are you sure?

• 5th: Directive to clinicians5th: Directive to clinicians– Grade 1: just do itGrade 1: just do it– Grade 2: think about itGrade 2: think about it

• your own judgment of strength of evidenceyour own judgment of strength of evidence

• your own judgment of risk/benefityour own judgment of risk/benefit

• talk to your patients, their values may impacttalk to your patients, their values may impact

Risk/Benefit clearRisk/Benefit clear

• Aspirin with acute myocardial infarctionAspirin with acute myocardial infarction– 25% reduction in relative risk, narrow confidence interval25% reduction in relative risk, narrow confidence interval– side effects trivial, cost negligibleside effects trivial, cost negligible– benefit obviously much greater than risk/cost, 1(A)benefit obviously much greater than risk/cost, 1(A)

• Thrombolysis in MI symptoms with only ST changesThrombolysis in MI symptoms with only ST changes– no difference from placebo, narrow confidence intervalno difference from placebo, narrow confidence interval– small risk of intracranial hemorrhagesmall risk of intracranial hemorrhage– risk obviously greater than possible benefit, 1(A)risk obviously greater than possible benefit, 1(A)

Judgment: Benefits vs Risks/CostsJudgment: Benefits vs Risks/Costs

• Seriousness of outcomeSeriousness of outcome• Magnitude of effectMagnitude of effect• Precision of treatment effectPrecision of treatment effect• Risk of target eventRisk of target event• Risk of serious adverse eventsRisk of serious adverse events• Cost of therapyCost of therapy• ValuesValues

Cost and Magnitude of EffectCost and Magnitude of Effect

• Clopidigrel vs ASA for atheroembolismClopidigrel vs ASA for atheroembolism– 8.7% RRR relative to ASA8.7% RRR relative to ASA– 5.83% to 5.32% in MI, ischemic 5.83% to 5.32% in MI, ischemic

stoke and deathstoke and death– NNT 200, cost $1,052 vs. $21NNT 200, cost $1,052 vs. $21– some will feel benefits not some will feel benefits not

worth extra costs and worth extra costs and therefore 2(A) therefore 2(A)

Cost and toxicityCost and toxicity

• TPA versus streptokinaseTPA versus streptokinase– RCT shows 15% RRR with TPARCT shows 15% RRR with TPA

• TPA larger costTPA larger cost

• TPA increased risk of intracranial hemorrhageTPA increased risk of intracranial hemorrhage

• Varying practice, unclear risk/benefitVarying practice, unclear risk/benefit– Grade 2 (B)Grade 2 (B)

Imprecision of treatment effectImprecision of treatment effect

• Should dipyridamole be added to aspirin after MI?Should dipyridamole be added to aspirin after MI?

• 1998 single RCT1998 single RCT– 85 deaths in 810 ASA alone, 87 in 810 ASA and dipyridamole85 deaths in 810 ASA alone, 87 in 810 ASA and dipyridamole– RR with ASA 0.98 (95% CI 0.70 to 1.26) RR with ASA 0.98 (95% CI 0.70 to 1.26)

• Recommendation: don’t use dipyridamoleRecommendation: don’t use dipyridamole

• Clearly Grade A; ? 1 or 2Clearly Grade A; ? 1 or 2

• Consensus criterion: Grade 1(A)Consensus criterion: Grade 1(A)

Precision of estimatePrecision of estimate• RR with ASA 0.98 (95% CI 0.70 to 1.26)RR with ASA 0.98 (95% CI 0.70 to 1.26)

– ASA may reduce risk relative to combination by 30%ASA may reduce risk relative to combination by 30%– combination may reduce risk relative to ASA by 26%combination may reduce risk relative to ASA by 26%

• Are we Are we suresure dipyridamole doesn’t add - No dipyridamole doesn’t add - No• Patient: I’ll take any low cost low toxicity medication that Patient: I’ll take any low cost low toxicity medication that

MIGHT helpMIGHT help

• Risk/benefit or patient value criteria: 2 (A)Risk/benefit or patient value criteria: 2 (A)• How to use confidence intervalHow to use confidence interval

– look at boundaries, is decision same at either end?look at boundaries, is decision same at either end?

Judgement: benefits versus risks/costsJudgement: benefits versus risks/costsSeriousness of outcomeSeriousness of outcome Death vs post-phlebitic syndromeDeath vs post-phlebitic syndrome

Magnitude of effectMagnitude of effect 68% RRR warfarin in a fib, vs68% RRR warfarin in a fib, vs

9% RRR with clopidigrel in CAD9% RRR with clopidigrel in CAD

Precision of treatment effectPrecision of treatment effect warfarin in a fib vs. ASA in a fibwarfarin in a fib vs. ASA in a fib

Risk of target eventRisk of target event warfarin in high vs low risk a fibwarfarin in high vs low risk a fib

Risk of serious adverse eventRisk of serious adverse event coumadin versus aspirincoumadin versus aspirin

CostsCosts ASA vs. clopidigrelASA vs. clopidigrel

Values (every decision)Values (every decision) high value on avoiding stroke: high value on avoiding stroke: TPA; clopidigrel; warfarinTPA; clopidigrel; warfarin

1 A recommendation1 A recommendation

• Patients with atrial fibrillation and additional risk Patients with atrial fibrillation and additional risk factors for arterial embolism without excessive factors for arterial embolism without excessive bleeding risk should receive warfarinbleeding risk should receive warfarin

• strong recommendation, can apply to most patients in most circumstances with no reservations

1 B recommendation1 B recommendation

• Clinicians should not administer Clinicians should not administer magnesium sulfate to patients with acute magnesium sulfate to patients with acute myocardial infarctionmyocardial infarction– meta-analysis of smaller RCTs +ve, meta-analysis of smaller RCTs +ve,

large RCT -ve large RCT -ve

• Strong recommendation, likely to apply to most patients

1 C recommendation1 C recommendation

• Patients with acute peripheral arterial Patients with acute peripheral arterial thrombi or emboli should be systematically thrombi or emboli should be systematically heparinizedheparinized– No RCTs, strong biological rationaleNo RCTs, strong biological rationale

• Intermediate strength recommendation, may change when stronger evidence available

2 A recommendation2 A recommendation

• Men over 50 without established CAD, but with one or Men over 50 without established CAD, but with one or more additional risk factors for CAD should take daily ASAmore additional risk factors for CAD should take daily ASA– RCT shows lower risk of MI but may be higher risk of cerebral RCT shows lower risk of MI but may be higher risk of cerebral

bleedbleed– both risks very low, individual values may determine decisionboth risks very low, individual values may determine decision

• Intermediate strength recommendation, best action may differ depending on circumstances or patients’ or societal values

2 B recommendation2 B recommendation

• Intra-arterial thrombolytic therapy may be used as an Intra-arterial thrombolytic therapy may be used as an alternative to surgery in patients with acute peripheral alternative to surgery in patients with acute peripheral arterial thrombi or emboli arterial thrombi or emboli – 2 RCTs, show comparable results, 1 surgery clearly better2 RCTs, show comparable results, 1 surgery clearly better

• Weak recommendation, alternative approaches likely to be better for some patients or circumstances

2 C recommendation2 C recommendation

• Pregnant women with previous venous Pregnant women with previous venous thrombosis associated with a transient risk factor thrombosis associated with a transient risk factor should receive surveillance only during should receive surveillance only during pregnancy and heparin and warfarin post-partumpregnancy and heparin and warfarin post-partum– incidence of thrombosis and magnitude of benefit incidence of thrombosis and magnitude of benefit

with therapy unestablishedwith therapy unestablished

• Very weak recommendation, alternatives equally reasonable

Are we producing guidelines?Are we producing guidelines?

• Yes!Yes!– Recommendations from authoritative body intended to influence clinical practiceRecommendations from authoritative body intended to influence clinical practice

• Shanneyfelt, JAMA;1999;281:1900Shanneyfelt, JAMA;1999;281:1900– 279 guidelines published 1985 to 1997279 guidelines published 1985 to 1997– adherence to standards for evidence summary 34%adherence to standards for evidence summary 34%– adherence to standards for making recommendations 6%adherence to standards for making recommendations 6%

• ConclusionsConclusions– we should do better we should do better oror– everyone else doing equally badly, we don’t need to worryeveryone else doing equally badly, we don’t need to worry

Systematic review of evidenceSystematic review of evidence

• Formal statement of eligibility criteria for each questionFormal statement of eligibility criteria for each question– patients, interventions, outcomes, methodologypatients, interventions, outcomes, methodology

• Systematic search for evidenceSystematic search for evidence

• Explicit process of data abstractionExplicit process of data abstraction

• Pooling of resultsPooling of results– wherever appropriatewherever appropriate

– systematic approachsystematic approach

From evidence to recommendationsFrom evidence to recommendations• Value judgments implicit in every recommendationValue judgments implicit in every recommendation

• Whose valuesWhose values– Ours?Ours?– Society?Society?– Patients?Patients?

• PossibilitiesPossibilities– explicit elicitation of values explicit elicitation of values – include people with different values/perspectivesinclude people with different values/perspectives

• patient; primary care doctorpatient; primary care doctor