Post on 03-Jan-2016
description
transcript
Critical Appraisal of Systematic Reviews
Douglas Newberry
Systematic Reviews —
or How to make a Monkey out of EBM without hardly
trying!
Systematic Reviews:Objectives:
• Appraise a systematic review for validity
• Discuss Meta Analysis / use Odds Ratios
• Obtain Number Needed to Treat (NNT) from Odds Ratios
• Consider clinical implications of a Systematic Review {including when to bin it instead!}
We can see further than our forbearers because we stand on
the shoulders of Giants{and have better spectacles}
• these ideas are cribbed unashamedly from friends, books & previous courses
Systematic Reviews:What are your Objectives:
What do you want to cover?
Please interject with helpful questions!
Did I really want a systematic review?
(but please do not pretend)
• admit your ignorance — expert review or consensus guidelines > broad introduction, cover many areas (class C evidence).
• if the question is important > formulate it!
• Systematic review > narrow but rigorous focus.
Systematic Reviews — Where do I start:
• Start with your 4 (or 3) part clinical question!
• Is a systematic review a sensible approach?
• Does THIS systematic review address MY question?
• Is it a systematic review at all?
Is it a systematic review? does it:• define a four part (answerable) clinical
question?
• combine Randomized Controlled Trials (RCT’s)?
• describe PRE-DEFINED search methods?
• PRE-DEFINED inclusion criteria?
• PRE-DEFINED methodological exclusion criteria?
Sceptical View? Take it with a grain of salt:
• transparent declaration of funding of work?
• Drug Company sponsorship of Reviews vs. Methodological quality>Cochrane review!
• who employs the authors?
• open discussion of existing controversy & commercial gain?
• Don’t waste salt on your food, keep it for your reading!
Meta analysis — combine what with what?
• Low Molecular Weight Heparin (LMWH) in hip surgery — begin before or after the operation?
• meta analysis of placebo controlled RCT’s of heparin in hip surgery >>
• pre-op & post-op LMWH vs. placebo
• post-op LMWH Vs placebo
• pre-operative >> less intra-op bleeding??
Can we believe it ?
• bias free search & inclusion criteria?
• appraisal of methodology of primary studies?
• consistent results from all primary studies?
– if not, are the differences sensibly explained?
• are the conclusions supported by the data?
If we believe it — does it apply to our patient?
• Is our patient (or population) so different from those in the primary studies that the results may not apply?
• consider differences in:
– time — many things change.
– culture — both treatments and values of outcomes can be different
– stage of illness or prevalence can effect results.
We believe it ! but—>> does it matter?
• Is the benefit worthwhile to our patient?
• Ask the patient about cultural values.
• Think about Relative Risk Reduction vs. Absolute Risk to our patient.
• Potential benefit is the Absolute risk avoided in our patient = Absolute Risk Reduction (ARR)!
Absolute Risk—> The risk our patient is facing!
• How likely is our patient to die (or have the outcome of interest) without intervention? = Control Event Rate (CER)
• consider this individual patient’s risk factors to estimate Patient Expected Event Rate = PEER.
• Absolute Risk usually increases with age.
• Improvement measured as Absolute Risk Reduction (ARR)
Relative Risk Reduction:• Usually reported in studies.
• Ratio of the improvement of outcome over outcome without intervention (Rx):
• {Control Event Rate (CER) — Experimental Event Rate (EER)} / CER
• i.e. {CER-EER}/CER
• often independent of prevalence!
• often similar at different ages!
Our patient wants an absolute Risk Reduction (ARR):
• is a 40% reduction in Cardiac Risk worth taking pills daily for 10 years?? >vote!
• if I have a 30% risk of MI or death {30 out of 100 people like me will suffer MI or death} in next 10 years > 40% RRR >> only 18 out of 100 will have MI or death. ARR = 12 out of 100! >>I like that!
• BUT if I have a 1% risk in 10 years, 40% less is a 0.6% risk >> hardly different!
Number Needed to Treat (NNT) (very trendy but tricky):
• only defined for specific prevalence-Patient’s Expected Event Rate=PEER!
• only defined for a specific intervention!• only defined for a specific outcome!
– eg. Pravastatin™ 40 mg nocte x10 years, in a 65 year old male, ex-smoker with high BP and Diabetes, to reduce MI or Death.
• NNT is the inverse of Absolute Risk Reduction: i.e. NNT = 1/ARR
Number Needed to Treat (NNT) for previous example:
• 12 fewer MI or death in 10 years per 100 persons treated: ARR=12/100
• NNT = 1/(12/100)=100/12= 8.3
• But the same Relative Risk Reduction (RRR) of 40% with a low prevalence:
• 0.4 fewer MI/death per 100 treated, ARR=0.4/100.
• NNT = 1/(0.4/100) = 100/0.4 = 250!
Why Odds Ratios? > compare results of different studies.
• consider 2x2 table:
• RRR is (a-b/a) — but you can only go in rows within same study!
• Odds ratio is (a/c)/(b/d) = ad / bc — the individual ratios are in columns, and therefore are independent of the prevalence which is different in different studies.
• must use odds ratios to combine RCT’s
Odds Ratio (OR) to NNT — is the improvement worth the trouble?
• 1>OR>0, lower the OR = better the treatment (Rx) >> lower NNT.
• for any OR, NNT is lowest when PEER=0.5
• estimate the PEER (patient’s risk)
• apply the OR to get patient's NNT.
Convert PEER & OR to NNT:
Odds Ratio (OR)Control CER 0.9 0.7 0.5
Event 0.1 110 36 21Rate(CER) 0.5 38 11 6{applyPEER 0.9 101 27 12here}
Formula used in the table:
NNT=1- {PEER * (1-OR)}
(1-PEER)*(PEER)*(1-OR)
Table induced nausea!• lower OR >> lower NNT
• Patient needs to be at risk (non-trivial PEER) in order for risk reduction to be worth the effort.
• for any OR, NNT lowest when PEER=0.5
• more effective treatment > lower NNT
• BUT are your patient’s values satisfied by the intervention and its sequelae?
Subgroup analysis: Sceptical unless:• the subgroups make biological and clinical
sense?
• the differences are both clinically & statistically significant?
• was a-priori hypothesis (before this data)?
• other evidence supports these sub-groups?
• few (OK) or many (nix) sub-group analyses?
Any Questions?
Summary 1: Set your goals.• define your 4 (or 3) part question.
• do you want a true systematic review?
• does this narrow review address my question?
• PRE-DEFINED search, inclusion, exclusion!
Summary 2: Be Sceptical!• look for bias, conflict of interest.
• critical appraisal of primary studies?
• consistent results? if not, why not?
• does our patient fit the groups studied?
• does it matter to our patient?
Summary 3: Risks that matter.• Absolute risk > estimate the Patient
Expected Event Rate (PEER)
• obtain Relative Risk Reduction (RRR) or Odds Ratio (OR) from a Meta-analysis
• plug into a table to estimate Number Needed to Treat (NNT)
Summary 4: Sceptical & common sense!
• beware of post-hoc sub-group analysis, especially if multiple.
• step back and consider if the systematic review really related to our patient’s situation (PEER), culture and expectations?
• do not loose sight of common sense!
Coffee Now!
• Small Groups Afterwards