How to Interpret Research Evidence EBM Workshop September.2007 Aaron Tejani...

How to Interpret How to Interpret Research EvidenceResearch Evidence

EBM WorkshopEBM WorkshopSeptember.2007September.2007

Aaron TejaniAaron [email protected]@fraserhealth.ca

DeclarationDeclaration

Paid by:Paid by: Fraser Health 80%Fraser Health 80% Therapeutics Initiative, UBC 20%Therapeutics Initiative, UBC 20%

No perceived or actual conflict of interest with No perceived or actual conflict of interest with the pharmaceutical industry in the last 4 yearsthe pharmaceutical industry in the last 4 years

What do these guys have in What do these guys have in common?common?

Questions?Questions?

The wise man doesn’t give the right answers, The wise man doesn’t give the right answers, he poses the right questionshe poses the right questions

- Claude Levi-- Claude Levi-StraussStrauss

Identifying Misleading ClaimsIdentifying Misleading Claims

Cautionary Tales in the Clinical Interpretation Cautionary Tales in the Clinical Interpretation of Therapeutic Trial Reports.of Therapeutic Trial Reports. Scott et al. IntMedJ2005;35:611-21Scott et al. IntMedJ2005;35:611-21

Cautionary tales in the interpretation of systematic reviews of therapy trials Scott et al. IntMedJ 2006;36:587–599

Users’ Guide to Detecting Misleading Claims in Users’ Guide to Detecting Misleading Claims in Clinical Trial Reports.Clinical Trial Reports. Montori VM, Jaeschke R et al. BMJ2004;329:1093-96Montori VM, Jaeschke R et al. BMJ2004;329:1093-96

Evidence Based MedicineEvidence Based Medicine

DefinitionDefinition The integration of best research evidence with clinical The integration of best research evidence with clinical

expertise and patient values. expertise and patient values. When these three elements are integrated, clinicians and patients When these three elements are integrated, clinicians and patients

form a diagnostic and therapeutic alliance with optimized clinical form a diagnostic and therapeutic alliance with optimized clinical outcomes and quality of life.outcomes and quality of life.

David Sackett and David Sackett and colleaguescolleagues

EBM is NOT purely academic or financial exerciseEBM is NOT purely academic or financial exercise Its implementation has major clinical implications that can Its implementation has major clinical implications that can

save lives and prevent harmsave lives and prevent harm

Critical AppraisalCritical Appraisal

DefinitionDefinition A method of assessing and interpreting the A method of assessing and interpreting the

evidence by systematically considering itsevidence by systematically considering its ValidityValidity ResultsResults RelevanceRelevance

Essential part of evidence-based clinical Essential part of evidence-based clinical practicepractice Required to determine BEST evidenceRequired to determine BEST evidence

Different Forms of EvidenceDifferent Forms of Evidence

Systematic reviewSystematic review A rigorous, systematic process to identify, synthesis and A rigorous, systematic process to identify, synthesis and

evaluate the available literatureevaluate the available literature Can be used to change practice by implementing the best Can be used to change practice by implementing the best

available literatureavailable literature

Meta-analysisMeta-analysis It is an extension of a well done systematic review, which It is an extension of a well done systematic review, which

provides a quantitative estimate of the net benefit provides a quantitative estimate of the net benefit aggregated over the included studiesaggregated over the included studies


Clinical TrialsClinical Trials Randomized controlled trial (RCT)Randomized controlled trial (RCT)

Minimize biasMinimize bias

CohortCohort Useful for topics with known health risksUseful for topics with known health risks

Harms of smokingHarms of smoking Effects of drugs in pregnancyEffects of drugs in pregnancy


Case series or case controlCase series or case control Useful for identifying areas that require further Useful for identifying areas that require further

investigationinvestigation Help identify adverse effectsHelp identify adverse effects

Expert opinionExpert opinion When other forms of data are not availableWhen other forms of data are not available

Hierarchies of EvidenceHierarchies of Evidence

I-1 I-1 Systematic review of several double-blind Systematic review of several double-blind randomized control trialsrandomized control trials

I-2 I-2 One or more large double-blind randomized One or more large double-blind randomized control trialscontrol trials

II-1 II-1 One or more well conducted cohort One or more well conducted cohort studiesstudies

II-2 II-2 One or more well-conducted case-control One or more well-conducted case-control studiesstudies

III III Expert committee sitting in review, peer Expert committee sitting in review, peer leader opinionleader opinion

IV IV Personal experiencePersonal experience

**There are many different types of grading systems

The NumbersThe Numbers

TI letter #16 1996

Real Clinical ScenariosReal Clinical Scenarios

TI letter #16 1996

Absolute Risk Reduction (ARR)Absolute Risk Reduction (ARR)

Absolute Risk Reduction illustratesAbsolute Risk Reduction illustrates The actual decrease from control to treatment in The actual decrease from control to treatment in

terms of effectterms of effect The absolute changeThe absolute change

ExampleExample Absolute risk reduction for the use of beta blockers Absolute risk reduction for the use of beta blockers

post MIpost MI

3.9%3.9%

Absolute Risk ReductionAbsolute Risk Reduction

Calculating Absolute Risk Reduction Calculating Absolute Risk Reduction ARR =X-YARR =X-Y

x=control event ratex=control event rate

y=treatment event ratey=treatment event rate Example:Example:

Treatment cancer rate=4%Treatment cancer rate=4% Control cancer rate=8%Control cancer rate=8% ARR=8-4=4% in cancer with treatment compared to ARR=8-4=4% in cancer with treatment compared to

controlcontrol

Number Needed to Treat (NNT)Number Needed to Treat (NNT)

Tool to place results into humanistic termsTool to place results into humanistic terms Number needed to treat (NNT)Number needed to treat (NNT)

Benefit of therapyBenefit of therapy Number needed to harm (NNH)Number needed to harm (NNH)

Harm of therapy Harm of therapy

Number Needed to TreatNumber Needed to Treat

Calculation of Number Needed to Treat (NNT)Calculation of Number Needed to Treat (NNT)NNT= 100/ARR(%)NNT= 100/ARR(%)

Previous exmaplePrevious exmapleARR=4%, NNT=100/4=25ARR=4%, NNT=100/4=25Treat 25 people to prevent one cancerTreat 25 people to prevent one cancer

Calculation of Number Needed To HarmCalculation of Number Needed To Harm NNH= 100/ARI(%)NNH= 100/ARI(%)

NNHNNH

ExampleExample Treatment GI bleed rate=10%Treatment GI bleed rate=10% Control GI bleed rate=5%Control GI bleed rate=5% ARI=10-5=5% increased risk with treatment for a ARI=10-5=5% increased risk with treatment for a

GI bleedGI bleed NNH=100/5=20NNH=100/5=20 1 GI bleed will occur for every 20 people treated1 GI bleed will occur for every 20 people treated

ImportantImportant

Only calculate ARR/ARI/NNT/NNH if the Only calculate ARR/ARI/NNT/NNH if the result is statistically significant!!result is statistically significant!!

Relative Risk Reduction (RR)Relative Risk Reduction (RR)

DefinitionDefinition Difference between the control and treatment Difference between the control and treatment

usually in terms of “reducing the chances”usually in terms of “reducing the chances”

Relative Risk ReductionRelative Risk Reduction

Calculating Relative risk (RR) Calculating Relative risk (RR) X is control groupX is control group Y is treatment groupY is treatment group

RR = Y/XRR = Y/X

Calculating Relative Risk Reduction (RRR)Calculating Relative Risk Reduction (RRR) X is controlX is control Y is treatmentY is treatment

RRR = RRR = CER-EER/CERCER-EER/CERPrevious example=8-4/4=50% relative reduction in cancer Previous example=8-4/4=50% relative reduction in cancer

with treatmentwith treatment

Types of OutcomesTypes of Outcomes

Dichotomous/categoricalDichotomous/categorical Yes or NoYes or No Possible to calculate absolute risk and NNTPossible to calculate absolute risk and NNT

ContinuousContinuous Blood pressureBlood pressure Rating scalesRating scales In order to calculate AR and NNT/NNH, a In order to calculate AR and NNT/NNH, a

clinically relevant change must be clearly defined clinically relevant change must be clearly defined a prioria priori

Confidence IntervalsConfidence Intervals If the trial is repeated an infinite number of times, the If the trial is repeated an infinite number of times, the

results will fall within this range 95% of the time results will fall within this range 95% of the time If p=0.05If p=0.05

95% certain that difference found is within the stated 95% certain that difference found is within the stated range, 5% likelihood it is due to chancerange, 5% likelihood it is due to chance if p=0.05 if p=0.05

Helps to determine how precise the results areHelps to determine how precise the results are Narrow versus wide confidence intervalNarrow versus wide confidence interval Point estimatePoint estimate

CI Humour BreakCI Humour Break

No, you can’t leave the roomNo, you can’t leave the room

P-ValuesP-Values

Determines if results are true or could be due to Determines if results are true or could be due to chance (Type 1 error, alpha)chance (Type 1 error, alpha) P value of 0.05 means that there is a 5% probability that the P value of 0.05 means that there is a 5% probability that the

results are due to chanceresults are due to chance P-value of 0.01 means that there is a 1% probability that P-value of 0.01 means that there is a 1% probability that

the results are due to chancethe results are due to chance Two tail or one tail testsTwo tail or one tail tests

Specifies direction of differenceSpecifies direction of difference i.e 2-sided, can see differences in positive or negative directioni.e 2-sided, can see differences in positive or negative direction

P value needs to be adjusted for multiple comparisonsP value needs to be adjusted for multiple comparisons

PowerPower

PowerPower Need to have enough people in the study to have Need to have enough people in the study to have

enough power to determine if a difference actually enough power to determine if a difference actually occurred occurred

If no difference seen, then need to consider the If no difference seen, then need to consider the sample size/power calculationsample size/power calculation

If a difference is seen, power is not an issueIf a difference is seen, power is not an issue

AlphaAlpha

Choosing your AlphaChoosing your Alpha What you are willing to acceptWhat you are willing to accept

5% or 1% probability that difference is due to chance5% or 1% probability that difference is due to chance Once you pick your alpha you CANNOT change it post Once you pick your alpha you CANNOT change it post

hochoc Alpha relates to p-value and is used to calculate Alpha relates to p-value and is used to calculate

sample sizesample size

Risk, Odds and HazardsRisk, Odds and Hazards

Hazards

OddsRisk

Risk Ratio (relative risk)Risk Ratio (relative risk)

If 24 skiersIf 24 skiers 6 fall6 fall The risk of falling is 25% (6/24)The risk of falling is 25% (6/24)

OddsOdds

If 24 skiersIf 24 skiers 6 fall6 fall 18 do not fall18 do not fall Odds is 6/18 or 1/3Odds is 6/18 or 1/3 The chances of falling were 3 to 1 againstThe chances of falling were 3 to 1 against 3 times more likely not to fall than to fall3 times more likely not to fall than to fall

Odds RatioOdds Ratio

Odds of one treatment versus odds of the other Odds of one treatment versus odds of the other treatmenttreatment If 24 skiersIf 24 skiers 6 fall6 fall If 24 snowboardersIf 24 snowboarders 12 fall12 fall

Odds ratio 0.25/0.5Odds ratio 0.25/0.5 50%50% Half as likely to fall if you are skiing as compared to Half as likely to fall if you are skiing as compared to

snowboardingsnowboarding

Continuous OutcomesContinuous Outcomes

E.g. blood pressure, rating scales, etcE.g. blood pressure, rating scales, etc Important pointsImportant points

Make sure scale is validMake sure scale is valid Need to know what a “clinically relevant” change Need to know what a “clinically relevant” change

in the scale isin the scale is Need to understand what the scale is measuringNeed to understand what the scale is measuring

i.e linear, non-linear, Likert-type scalei.e linear, non-linear, Likert-type scale

Don’t get hung up with STATS!!Don’t get hung up with STATS!!

The appropriate statistical test is not as The appropriate statistical test is not as important as methods and outcomesimportant as methods and outcomes

DO NOT PANIC DO NOT PANIC More important aspects of critical appraisalMore important aspects of critical appraisal

Clearly defined methodsClearly defined methods Reasonable and clinically significant outcomes and Reasonable and clinically significant outcomes and

measurementsmeasurements

RandomizationRandomization

Define RandomizationDefine Randomization

A method based on chance alone by which A method based on chance alone by which study participants are assigned to a treatment study participants are assigned to a treatment group. Randomization minimizes the group. Randomization minimizes the differences among groups by equally differences among groups by equally distributing people with particular distributing people with particular characteristics among all the trial arms. characteristics among all the trial arms. www.medterms.com/script/main/art.asp?www.medterms.com/script/main/art.asp?

articlekey=38700articlekey=38700

What Characteristics Are Equally What Characteristics Are Equally Distributed?Distributed?

The Benefits of RandomizationThe Benefits of Randomization

Everyone has an equal chance of getting Everyone has an equal chance of getting assigned treatment or controlassigned treatment or control

MOST IMPORTANT:MOST IMPORTANT: Groups are divided equally for known and Groups are divided equally for known and

unknown characteristicsunknown characteristics This ensures…This ensures…

Differences in outcome are likely only due to Differences in outcome are likely only due to differences in assigned treatmentdifferences in assigned treatment

If randomization was deemed successfulIf randomization was deemed successful

User’s Guides’ StatementUser’s Guides’ Statement

““The beauty of randomization is that it assures, The beauty of randomization is that it assures, if sample size is sufficiently large, that both if sample size is sufficiently large, that both known and unknown determinants of outcome known and unknown determinants of outcome are evenly distributed between treatment and are evenly distributed between treatment and control groups. “control groups. “

http://www.cche.net/usersguides/therapy.asp

Was Randomization Effective?Was Randomization Effective? Look at Baseline characteristics to see if they were Look at Baseline characteristics to see if they were

balancedbalanced NOTE: This does not account for unknown characteristics NOTE: This does not account for unknown characteristics

but if numbers are large and known’s are balancedbut if numbers are large and known’s are balanced Can assume unknown’s are as wellCan assume unknown’s are as well

p-values when comparing baseline characteristicsp-values when comparing baseline characteristics Don’t worry about themDon’t worry about them If you think the differences may have an effect then they If you think the differences may have an effect then they

mightmight Doesn’t matter is they are “statistically significant differences”Doesn’t matter is they are “statistically significant differences”

User’s Guides’ StatementUser’s Guides’ Statement

““The issue here is not whether there are The issue here is not whether there are statistically significant differences in known statistically significant differences in known prognostic factors between treatment groups prognostic factors between treatment groups (in a randomized trial one knows in advance (in a randomized trial one knows in advance that any differences that did occur happened that any differences that did occur happened by chance) but rather the magnitude of these by chance) but rather the magnitude of these differences.”differences.”

http://www.cche.net/usersguides/therapy.asp

NAC for Prevention of RF in NAC for Prevention of RF in Cardiac SurgeryCardiac Surgery

If baseline characteristics are If baseline characteristics are different…different…

All is not lostAll is not lost Authors can do analyses adjusting for Authors can do analyses adjusting for

differencesdifferences Should clearly state that this was done and howShould clearly state that this was done and how

If adjusted and unadjusted analyses show If adjusted and unadjusted analyses show similar resultssimilar results You can be more confident of the findingsYou can be more confident of the findings

Not Randomized…So What?Not Randomized…So What?

Non-randomised studies overestimate Non-randomised studies overestimate treatment effect by 41% with inadequate treatment effect by 41% with inadequate method, 30% with unclear methodmethod, 30% with unclear method JAMA 1995 273: 408-12.JAMA 1995 273: 408-12.

Completely different result between Completely different result between randomized and non-randomized studiesrandomized and non-randomized studies BritJAnaesth1996; 77: 798-803.BritJAnaesth1996; 77: 798-803.

Red and Yellow Lollipops!!Red and Yellow Lollipops!!

RedRed YellowYellow

MaleMale

FemaleFemale

Born Jan-JuneBorn Jan-June

Watched the Watched the news last nightnews last night

Plays an Plays an instrumentinstrument

Has a petHas a pet

Allocation Allocation Concealment (AA)Concealment (AA)

Example:Example:

GP in his office with 2 elderly gentlemen in GP in his office with 2 elderly gentlemen in his waiting roomhis waiting room He knows one guy well and this man has had He knows one guy well and this man has had

terrible luck with his healthterrible luck with his health He is enrolling men into a clinical trialHe is enrolling men into a clinical trial He knows that the next person is going to get Box He knows that the next person is going to get Box

A and the one after that gets Box BA and the one after that gets Box B Last month he enrolled a man who got Box A and Last month he enrolled a man who got Box A and

has not improved rather he has gotten worsehas not improved rather he has gotten worse

What is AA?What is AA?

Is it blinding?Is it blinding? Can it always be done?Can it always be done? Can blinding always be done?Can blinding always be done? What is selection bias?What is selection bias? Example…Example…


Definition:Definition: “…“…shields those involved in a trial from knowing shields those involved in a trial from knowing

upcoming assignments. Without this protection, upcoming assignments. Without this protection, investigators and patients have been known to investigators and patients have been known to change who gets the next assignment, making the change who gets the next assignment, making the comparison groups less equivalent”comparison groups less equivalent”

Purpose?Purpose? To reduce “selection bias”To reduce “selection bias”

Evid.Based Med. 2000;5;36-38


If AA is not successful what design feature is If AA is not successful what design feature is compromised and why?compromised and why?

“…“…if the investigator or clinician (or the patient) is if the investigator or clinician (or the patient) is able to identify the impending treatment allocation able to identify the impending treatment allocation and is able to influence the enrolment (or selection) and is able to influence the enrolment (or selection) of participating patients, the value of randomisation is of participating patients, the value of randomisation is compromised.”compromised.”

May lead to imbalances in prognostic factors between May lead to imbalances in prognostic factors between groupsgroups

MJA 2005;182(2):87-89

What is good AA?What is good AA?

Opaque-sealed envelopesOpaque-sealed envelopes Pharmacy-controlled allocationsPharmacy-controlled allocations Coded identical medication containersCoded identical medication containers Telephone or web-based central randomizationTelephone or web-based central randomization

Who Cares if AA Wasn’t Done?Who Cares if AA Wasn’t Done?

Those trials that report inadequate methods of Those trials that report inadequate methods of AAAA Report ~30% larger effect sizes compared to trials Report ~30% larger effect sizes compared to trials

that use sound methods of AAthat use sound methods of AA Those trials that do not mention AAThose trials that do not mention AA

Report ~40% larger effect sizes compared to trials Report ~40% larger effect sizes compared to trials that reported AAthat reported AA

JAMA 1995;273:408–12.

What do you do if AA isn’t done What do you do if AA isn’t done well?well?

If you meta-If you meta-analyze trials…analyze trials… Do a sensitivity Do a sensitivity

analysis of trials analysis of trials with good AA with good AA versus no AAversus no AA

E.g.Atypical E.g.Atypical Coverage in Coverage in patients with CAPpatients with CAP

What do you do if AA isn’t done?What do you do if AA isn’t done?

If there is no meta-analysis…If there is no meta-analysis… Assume the effect size may be exaggerated and base Assume the effect size may be exaggerated and base

decisions on conservative estimate of effect sizedecisions on conservative estimate of effect size i.e look at the conservative end of the confidence intervali.e look at the conservative end of the confidence intervalMortality reduction 6% (95% CI 1% to 8%)Mortality reduction 6% (95% CI 1% to 8%) May conclude reduction is probably between 1% and %5May conclude reduction is probably between 1% and %5 CAUTION: This is not scientific! This is my attempt at CAUTION: This is not scientific! This is my attempt at

common sense…common sense…

ReportingReporting

Do not assume that AA was not done if not Do not assume that AA was not done if not reportedreported Can ask the authors if they did or notCan ask the authors if they did or not

Only 9-15% of trials adequately report these Only 9-15% of trials adequately report these methodsmethods

Bad reportingBad reporting

Good reportingGood reporting

MJA 2005;182(2):87-89

Blinding (Masking)Blinding (Masking)

DefinitionsDefinitions

Single blindSingle blind Either clinician OR patient is unaware of assigned Either clinician OR patient is unaware of assigned

treatmenttreatment Double blindDouble blind

Both clinician and patient are unaware of assigned Both clinician and patient are unaware of assigned treatmenttreatment

Triple blindTriple blind Clinician, patient, and people who adjudicate outcomes are Clinician, patient, and people who adjudicate outcomes are

unaware of treatment assignmentunaware of treatment assignment

A questionA question

What is “double-dummy”?What is “double-dummy”?

The purpose of blindingThe purpose of blinding

Attempts to minimizeAttempts to minimize Reporting biasReporting bias

E.g.E.g. Assessment biasAssessment bias

E.g.E.g. Concomitant treatment biasConcomitant treatment bias

E.g.E.g.

How can you tell if blinding is How can you tell if blinding is broken?broken?

If the authors test for success of blindingIf the authors test for success of blinding Blinding may be broken whenBlinding may be broken when

One treatment has a particular side effect that One treatment has a particular side effect that would give it awaywould give it away

E.g. infusion site reactionsE.g. infusion site reactions Look at ADR table and see if this may be occurringLook at ADR table and see if this may be occurring

Testing the success of blindingTesting the success of blinding

Some would argue that success of blinding testing is Some would argue that success of blinding testing is not reliablenot reliable (Sackett Int J Epi 2007;36:665-666)(Sackett Int J Epi 2007;36:665-666) These tests only tell us about the “hunches” people haveThese tests only tell us about the “hunches” people have It is better to measure the effects of lost blindingIt is better to measure the effects of lost blinding

Co-intervention (get the study drug by other means)Co-intervention (get the study drug by other means) Contamination (controls get open label treatment with another Contamination (controls get open label treatment with another

drug)drug) Reporting bias (e.g.study drug people down-play symptoms)Reporting bias (e.g.study drug people down-play symptoms)

What is the consequence of broken What is the consequence of broken blinding?blinding?

Studies with poor/absent blinding tend to over Studies with poor/absent blinding tend to over estimate treatment effects by ~17%estimate treatment effects by ~17% JAMA 1995;273:408-12.JAMA 1995;273:408-12.

What to do if blinding is broken?What to do if blinding is broken?

Assume the effect size is an over-estimateAssume the effect size is an over-estimate Look to the conservative end of the confidence Look to the conservative end of the confidence

intervalintervalMortality reduction 6% (95% CI 1% to 8%)Mortality reduction 6% (95% CI 1% to 8%) May conclude reduction is probably between 1% May conclude reduction is probably between 1%

and %5and %5 CAUTION: This is not scientific! This is my CAUTION: This is not scientific! This is my

attempt at common sense…attempt at common sense…

How do you interpret open-label How do you interpret open-label trials?trials?

If there is a meta-analysisIf there is a meta-analysis Do a sensitivity analysis on blinded versus un-Do a sensitivity analysis on blinded versus un-

blinded trials and see if the effect size changesblinded trials and see if the effect size changes If there is no meta-analysisIf there is no meta-analysis

Assess whether blinding was possibleAssess whether blinding was possible Interpret findings carefully knowing they could be Interpret findings carefully knowing they could be

an overestimation of an effectan overestimation of an effect Especially if blinding was possibleEspecially if blinding was possible

MJA 2005;182(2):87-89

Intention-to-treat AnalysisIntention-to-treat Analysis

Very Important!!Very Important!! Definition:Definition:

Analyze participants into the groups to which they Analyze participants into the groups to which they were randomizedwere randomized

Even if they did not take assigned treatment, dropped Even if they did not take assigned treatment, dropped out early, or did not follow protocol exactlyout early, or did not follow protocol exactly

Intention-to-treat AnalysisIntention-to-treat Analysis

Loss to Follow UpLoss to Follow Up

Each trials should report the number of people Each trials should report the number of people lost and how many per grouplost and how many per group

Usual assumptionUsual assumption Nothing bad happened to people that were lostNothing bad happened to people that were lost

General rule:General rule: >20% of randomized population lost…analysis >20% of randomized population lost…analysis

becomes unreliablebecomes unreliable Consider worse case scenario and see if this Consider worse case scenario and see if this

changes the resultchanges the result

A Primer on the A Primer on the Interpretation ofInterpretation of

Subgroup Analyses (SA) Subgroup Analyses (SA) in Clinical Trialsin Clinical Trials

George Bernard ShawGeorge Bernard Shaw

““Beware of false knowledge, it is more Beware of false knowledge, it is more dangerous than ignorance.”dangerous than ignorance.”

Different?Different?

What is a Subgroup Analysis?What is a Subgroup Analysis?

Drug A versus Drug B are equal with respect Drug A versus Drug B are equal with respect to mortality in patients with coronary artery to mortality in patients with coronary artery diseasedisease You want to see if this finding is the same in men You want to see if this finding is the same in men

versus womenversus women Or do Diabetics have a mortality benefit as Or do Diabetics have a mortality benefit as

compared to non-Diabetics?compared to non-Diabetics?

Why is SA So Common in Why is SA So Common in Trials?Trials?

Basically to see if different types of patients Basically to see if different types of patients respond differently to the same treatmentrespond differently to the same treatment If there is an overall benefit of treatmentIf there is an overall benefit of treatment

See if there is more or less benefit in a subgroupSee if there is more or less benefit in a subgroup If there is no overall effectIf there is no overall effect

See if there is an effect in at least a certain type of See if there is an effect in at least a certain type of patientpatient

What Questions Should SA Attempt What Questions Should SA Attempt to Answer?to Answer?

At what stage of disease is treatment most At what stage of disease is treatment most effective?effective?

How are the risks and benefits of treatment How are the risks and benefits of treatment related to co-morbidity?related to co-morbidity?

What time after an event is treatment most What time after an event is treatment most effective?effective?

Example: CAPRIE TrialExample: CAPRIE Trial Lancet 1996; 348: 1329–39Lancet 1996; 348: 1329–39

Clopidogrel versus ASA for reducing ischemic Clopidogrel versus ASA for reducing ischemic events in patients at riskevents in patients at risk Patients with MI, Stroke, or PAD historyPatients with MI, Stroke, or PAD history DBRCTDBRCT Clopidogrel 75mg po daily n>30,000Clopidogrel 75mg po daily n>30,000 ASA 325 po dailyASA 325 po daily Duration 1-3 yearsDuration 1-3 years Primary: ischemic stroke, MI, vascular deathPrimary: ischemic stroke, MI, vascular death

Example: CAPRIE TrialExample: CAPRIE Trial Lancet 1996; 348: 1329–39Lancet 1996; 348: 1329–39

Problems with SAProblems with SA

1. Multiple testing1. Multiple testing The more tests you do, The more tests you do,

the greater the the greater the probability of finding a probability of finding a difference (that is really difference (that is really due to chance) due to chance)

Need to correct P value Need to correct P value for multiple comparisonsfor multiple comparisons

Crude wayCrude way 0.05/ # tests0.05/ # tests

Cook DI et al. MJA 2004;180(3):289-Cook DI et al. MJA 2004;180(3):289-9191

Adjusting for MultiplicityAdjusting for Multiplicity

K=number of independent tests performedK=number of independent tests performed p* is the smallest p-value calculatedp* is the smallest p-value calculated Corrected p= 1-(1-p*)Corrected p= 1-(1-p*)kk

The adjusted p-value can then be compared to The adjusted p-value can then be compared to the traditional p=0.05 as being statistically the traditional p=0.05 as being statistically significant or notsignificant or not


Example: CHARISMAExample: CHARISMA Clopidogrel+ASA vs ASA alone in patients Clopidogrel+ASA vs ASA alone in patients

with CVDwith CVD NOTE: No benefit in overall populationNOTE: No benefit in overall population Symptomatic vs asymptomaticSymptomatic vs asymptomatic

N Engl J Med 2006;354:1706-17.


Example: CHARISMAExample: CHARISMA

N Engl J Med 2006;354:1706-17.


Example: CHARISMAExample: CHARISMA 12 subgroups12 subgroups Unadjusted p=0.046Unadjusted p=0.046 Adjusted p-value for symp vs asymp=Adjusted p-value for symp vs asymp=

1-(1-0.046)1-(1-0.046)1212=0.43=0.43

NEJM 2006;354:1706-17NEJM 2006;354:1667-1669

Probability of False PositivesProbability of False Positives

NEJM 2006;354:1667-1669

e.g. 4 subgroups anayzed

Approximately 15-20% chance of false-positive…instead of a 5% chance (i.e p=0.05)

Problems with SAProblems with SA 1. Multiple testing1. Multiple testing

Using the CAPRIE exampleUsing the CAPRIE example 3 subgroups ~ 3 tests for the primary outcome3 subgroups ~ 3 tests for the primary outcome 0.05 / 3 = 0.0167 is the adjusted p-value0.05 / 3 = 0.0167 is the adjusted p-value E.g. for any of the subgroups would have to have p<0.0167 E.g. for any of the subgroups would have to have p<0.0167

to be SS or adjusted p=0.008 =1-(1-p)to be SS or adjusted p=0.008 =1-(1-p)kk

Lancet 1996; 348: 1329–39Lancet 1996; 348: 1329–39


2. Lack of statistical power for SA2. Lack of statistical power for SA Most studies powered for the whole population Most studies powered for the whole population

onlyonly Studies usually do not have power to detect Studies usually do not have power to detect

differences in subgroupdifferences in subgroup If a difference is seen, there is enough power If a difference is seen, there is enough power

but it may be a false positivebut it may be a false positive If no difference is seen in a subgroup it may be If no difference is seen in a subgroup it may be

due to actual lack of powerdue to actual lack of power

CautionCaution

When there is no overall effectWhen there is no overall effect Subgroups will show an effect 7-21% of the time Subgroups will show an effect 7-21% of the time

that aren’t realthat aren’t real When there is an overall effectWhen there is an overall effect

Subgroups won’t show a difference 41-66% of the Subgroups won’t show a difference 41-66% of the timetime

Health Technol Assess 2001 5: 1–56.

Tests for InteractionTests for Interaction

A test for “heterogeneity of treatment effect”A test for “heterogeneity of treatment effect” The appropriate statistical testThe appropriate statistical test

Does notDoes not test the magnitude of difference between test the magnitude of difference between subgroups and the overall populationsubgroups and the overall population

Does Does test to see if the subgroup effect is different test to see if the subgroup effect is different form the overall effect but says nothing of by how form the overall effect but says nothing of by how muchmuch


E.g. CAPRIEE.g. CAPRIE

Lancet 1996; 348: 1329–39Lancet 1996; 348: 1329–39


E.g. CHARISMAE.g. CHARISMA

Lancet 1996; 348: 1329–39Lancet 1996; 348: 1329–39


3. Subgroups are not truly “randomized”3. Subgroups are not truly “randomized” Randomization works to balance known and Randomization works to balance known and

unknown factors in the overall populationunknown factors in the overall population Subgroups are NOT truly randomized unless Subgroups are NOT truly randomized unless

randomization was:randomization was: Stratified Stratified This allows SA to be based on pre-randomization This allows SA to be based on pre-randomization

characteristicscharacteristics


3. Subgroups are not truly “randomized”3. Subgroups are not truly “randomized” E.g. TNT trialE.g. TNT trial

NEJM 2005;352(March 08):



NEJM 2005;352(March 08):



NEJM 2005;352(March 08):



NEJM 2005;352(March 08):



NEJM 2005;352(March 08):


3. Subgroups are not truly “randomized”3. Subgroups are not truly “randomized” E.g. TNT trialE.g. TNT trial Achieving a certain LDL level is a post-randomization Achieving a certain LDL level is a post-randomization

phenomenonphenomenon No test for interaction were done/reported on the effect in No test for interaction were done/reported on the effect in

patients who achieved lower vs higher LDL levelspatients who achieved lower vs higher LDL levels Should have randomized to titrating to LDL<2.5 and Should have randomized to titrating to LDL<2.5 and

LDL<2.0LDL<2.0 This trial was a high vs low dose This trial was a high vs low dose

NEJM 2005;352(March 08):


4. The play of chance4. The play of chance Even properly done SA can yield significant tests Even properly done SA can yield significant tests

of interaction by chance aloneof interaction by chance alone Especially when the subgroup effect is not plausible or Especially when the subgroup effect is not plausible or

it is unanticipatedit is unanticipated

CHARISMACHARISMALancet 1996; 348: 1329–39Lancet 1996; 348: 1329–39

Replication is the SolutionReplication is the Solution

Due to lack of power and high probability of Due to lack of power and high probability of false-positivesfalse-positives Subgroup differences should be “hypothesis Subgroup differences should be “hypothesis

generating”generating” The findings should be tested in a trial designed for this The findings should be tested in a trial designed for this

purposepurpose

Replication is the SolutionReplication is the Solution

Checklist for “Good” SAsChecklist for “Good” SAs

1. Subgroups should be:1. Subgroups should be: Based on pre-randomization characteristics and Based on pre-randomization characteristics and

stratified accordinglystratified accordingly YESYES

Based on intent-to-treat populationBased on intent-to-treat population To maintain the benefits of randomizationTo maintain the benefits of randomization YESYES


2. Must be pre-defined2. Must be pre-defined And have a plausible biological reason for And have a plausible biological reason for

choosing the subgroupchoosing the subgroup YESYES

Should justify the direction of the expected Should justify the direction of the expected difference in the subgroupdifference in the subgroup

NONO


3. Reporting3. Reporting All numerators and denominators should be All numerators and denominators should be

reportedreported YESYES

# of planned subgroups# of planned subgroups YESYES

Address the issue of multiple testingAddress the issue of multiple testing NONO


4. Statistical Analysis4. Statistical Analysis Need to do tests for interaction (heterogeneity of Need to do tests for interaction (heterogeneity of

effect between overall population and subgroup)effect between overall population and subgroup) A significant interaction test tells youA significant interaction test tells you

The subgroup effect is different than overall effectThe subgroup effect is different than overall effect DOES NOT tell you anything about the magnitude of DOES NOT tell you anything about the magnitude of

differencedifference


5. Interpretation of findings5. Interpretation of findings The overall effect should be stressedThe overall effect should be stressed

Due to the high risk of false-positive findings in subgroupsDue to the high risk of false-positive findings in subgroups Not unusual to find a SS difference in a subgroup when NSS Not unusual to find a SS difference in a subgroup when NSS

overall findingoverall finding

YES, the overall effect was stressedYES, the overall effect was stressed

In General…In General…

Focus on overall resultsFocus on overall results Use results in subgroups only ifUse results in subgroups only if

Significant interaction test (heterogeneity)Significant interaction test (heterogeneity) You still want to explore possible reasons for You still want to explore possible reasons for

interactioninteraction i.e make sure there is a biological reason for the i.e make sure there is a biological reason for the

subgroup differencesubgroup difference Has this difference been found in other studies?Has this difference been found in other studies?

Practical ApplicationPractical Application

•So for CAPRIE…

•Could use the results to treat PAD patients with Clopidogrel but not stroke or MI patients

•Might want to study the effect of clopidogrel in MI patients

Subgroup Analyses ReferencesSubgroup Analyses References

Lagakos SW. NEJM 2006;354(16):1667-9Lagakos SW. NEJM 2006;354(16):1667-9 Cook DI et al. MJA 2004;180(3):289-91Cook DI et al. MJA 2004;180(3):289-91 Simes RJ et al. MJA 2004;180(5):467-9Simes RJ et al. MJA 2004;180(5):467-9 Rothwell P. Lancet 2005;365:176–86Rothwell P. Lancet 2005;365:176–86

DemographicsDemographics

Composite Outcome Composite Outcome (CO) Interpretation(CO) Interpretation

What is a Composite Outcome?What is a Composite Outcome?

Example #1Example #1 TIME Trial (TIME Trial (Lancet Lancet 2001; 358: 951–957)2001; 358: 951–957)

Invasive therapy versus medical management for Invasive therapy versus medical management for symptomatic coronary artery diseasesymptomatic coronary artery disease

Death, non-fatal MI, admission for ACSDeath, non-fatal MI, admission for ACS ““The primary endpoint was analysed by The primary endpoint was analysed by

intention to treat as a composite endpoint, and intention to treat as a composite endpoint, and all components separately as secondary all components separately as secondary endpoints.”endpoints.”


Example #2:Example #2: CAPRIE Trial (Lancet 1996; 348: 1329–39)CAPRIE Trial (Lancet 1996; 348: 1329–39)

Clopidogrel versus ASA and ischemic events in patients Clopidogrel versus ASA and ischemic events in patients at riskat risk

The first occurrence of ischaemic stroke, myocardial The first occurrence of ischaemic stroke, myocardial infarction, or vascular death.infarction, or vascular death.

No stated/planned assessment of individual components of the No stated/planned assessment of individual components of the compositecomposite


Example #3:Example #3: ValHeFT Trial ValHeFT Trial (N Engl J Med 2001;345:1667-75.)(N Engl J Med 2001;345:1667-75.)

Added Valsartan versus standard therapy for CHFAdded Valsartan versus standard therapy for CHF Mortality and the combined end point of mortality and Mortality and the combined end point of mortality and

morbidity:morbidity: defined as cardiac arrest with resuscitation, hospitalization for defined as cardiac arrest with resuscitation, hospitalization for

heart failure, or administration of intravenous inotropic or heart failure, or administration of intravenous inotropic or vasodilator drugs for four hours or more with hospitalization.vasodilator drugs for four hours or more with hospitalization.

Benefit of Composite OutcomesBenefit of Composite Outcomes

Statistical EfficiencyStatistical Efficiency Improved medical care has led to low event rates Improved medical care has led to low event rates

(e.g. fewer MIs, strokes, etc…)(e.g. fewer MIs, strokes, etc…) Need large trials with long follow up to Need large trials with long follow up to

demonstrate differences between treatmentsdemonstrate differences between treatments Not “feasible” for many researchersNot “feasible” for many researchers

COs allow researchers to show differences in smaller COs allow researchers to show differences in smaller trials with shorter durationstrials with shorter durations

What do you think?What do you think?

Death / MI/ Admission for ACS??Death / MI/ Admission for ACS??

Death / MI / Stroke??Death / MI / Stroke??

Cardiac arrest with resuscitation, Cardiac arrest with resuscitation, hospitalization for heart failure, or IV hospitalization for heart failure, or IV inotropic or vasodilator drugs for four hours or inotropic or vasodilator drugs for four hours or more without hospitalization??more without hospitalization??

The Primary Question About COThe Primary Question About CO

Can I use the analysis of the composite Can I use the analysis of the composite outcome comparison between treatment and outcome comparison between treatment and control as the basis for a decision?control as the basis for a decision? E.g. If the CO is lower for Intervention X E.g. If the CO is lower for Intervention X

compared to control would I prescribe X?compared to control would I prescribe X? OR OR

If the CO is lower for X do I need to look at the If the CO is lower for X do I need to look at the analysis of the components before I make a analysis of the components before I make a decision?decision?

Checklist for COsChecklist for COs1,21,2

The benefit of using a CO is realized if:The benefit of using a CO is realized if:1.1. □□ Individual components are of equal importanceIndividual components are of equal importance

2.2. □□ Effects of intervention on components will be similar Effects of intervention on components will be similar (i.e occur at similar frequency)(i.e occur at similar frequency)

3.3. □□ The more important components should not be The more important components should not be negatively affected by the interventionnegatively affected by the intervention

4.4. □□ Counting rules are appropriate and individual Counting rules are appropriate and individual component data is presentedcomponent data is presented

1. Individual Components are of Equal Importance1. Individual Components are of Equal Importance

a.a. Example#1TIME TrialExample#1TIME Trial– Death, non-fatal MI, admission for ACSDeath, non-fatal MI, admission for ACS

Could argue that death and MI are more important to patients Could argue that death and MI are more important to patients than admission for ACSthan admission for ACS

It then becomes important to know:It then becomes important to know:– Which part is driving the reduction in the CO?? Which part is driving the reduction in the CO?? If it is If it is

admission due to ACS then invasive treatment may not be admission due to ACS then invasive treatment may not be worth it for a patientworth it for a patient

Invasive Med

2. Components Occur at Similar Frequency2. Components Occur at Similar Frequency


•Easy to see that Death and MI occur much less than Admission for ACS

•In this case the using the CO result when making decisions could be problematic

•The total CO event rate is primarily driven by admissions and not the outcomes that “may” hold greater importance to a patient

3. Important Components are Not Negatively 3. Important Components are Not Negatively AffectedAffected


•The MOST important outcome… numerically occurs more frequently in the Invasive treatment group but this is NSS

4. Counting Rules are Appropriate and Component 4. Counting Rules are Appropriate and Component Data is presentedData is presented


•YES…Component data is presented…but was the counting proper?

•They don’t tell us if it is “first occurrrence of” or if individual component rates are for entire study

•What if you had an MI and then a month later you were admitted for ACS?

•Were both events accounted for?

TIME Trial ConclusionTIME Trial Conclusion

Do not base decisions on analysis of CO rates Do not base decisions on analysis of CO rates alonealone Need to look at the details of the components Need to look at the details of the components

Example #2 CAPRIE TrialExample #2 CAPRIE TrialIschemic stroke/MI/Vascular DeathIschemic stroke/MI/Vascular Death

1. Components are of equal importance1. Components are of equal importance Yes, most would agree that they are all clinically important Yes, most would agree that they are all clinically important

outcomesoutcomes

In that case looking at the overall result could be used to make a In that case looking at the overall result could be used to make a decisiondecision

Be carefulBe careful All cause death is not part of the compositeAll cause death is not part of the composite

Assumes clopidogrel won’t effect non-vascular death negativelyAssumes clopidogrel won’t effect non-vascular death negatively Need to consider the rest of the checklistNeed to consider the rest of the checklist


2. Occur at similar frequencies2. Occur at similar frequencies ?????? NFMI occur less frequently than vascular deaths and strokesNFMI occur less frequently than vascular deaths and strokes


3. Important outcomes are not negatively affected3. Important outcomes are not negatively affected None of the components were negatively affectedNone of the components were negatively affected


4. 4. Counting rules appropriate and component data presentedCounting rules appropriate and component data presented Component data presented BUT…Component data presented BUT… Not sure if counting rules were OKNot sure if counting rules were OK The main table reports “first occurrence of” ratesThe main table reports “first occurrence of” rates

So it is possible that a patient died at some point after a stroke and this So it is possible that a patient died at some point after a stroke and this death was not counteddeath was not counted

CAPRIE Trial ConclusionCAPRIE Trial Conclusion

Could use CO analysis as the basis for a Could use CO analysis as the basis for a decisiondecision The only issue is the lower MI event rates versus The only issue is the lower MI event rates versus

other components but this is debatableother components but this is debatable

Example#3 ValHeFT TrialExample#3 ValHeFT Trial

Focus on counting rules onlyFocus on counting rules only Appropriate for DEATHAppropriate for DEATH

E.g. the “total deaths” are greater than “death (as first event)”E.g. the “total deaths” are greater than “death (as first event)”

Use the CO Analysis if…Use the CO Analysis if…

1.1.□□ Individual components are of equal importanceIndividual components are of equal importance

2.2.□□ Effects of intervention on components will be Effects of intervention on components will be similar (i.e occur at similar frequency)similar (i.e occur at similar frequency)

3.3.□□ The more important components should not be The more important components should not be negatively affected by the interventionnegatively affected by the intervention

4.4.□□ Counting rules are appropriate and individual Counting rules are appropriate and individual component data is presentedcomponent data is presented

Composite Outcome ReferencesComposite Outcome References

Kleist P. Applied Clinical Trials 2006 (May Kleist P. Applied Clinical Trials 2006 (May Issue)Issue)

Montori VM et al. BMJMontori VM et al. BMJ 2005;330;594-5962005;330;594-596

Superiority, Non-Superiority, Non-inferiority, and inferiority, and

Equivalence Trials.Equivalence Trials.Aaron TejaniAaron Tejani

April 18. 2007April 18. 2007

The Main Purpose of EachThe Main Purpose of Each

Superiority Superiority Is one treatment better than another?Is one treatment better than another?

Non-inferiorityNon-inferiority Is one agent no worse than a standard therapy (based on a Is one agent no worse than a standard therapy (based on a

pre-defined “no worse” margin)?pre-defined “no worse” margin)? EquivalenceEquivalence

Is one agent no worse or no better than a standard therapy Is one agent no worse or no better than a standard therapy (based on pre-defined limits of no worse or better)?(based on pre-defined limits of no worse or better)?

Superiority TrialsSuperiority Trials

E.g. GUSTO IIIE.g. GUSTO III Designed to show that RPA would lower mortality Designed to show that RPA would lower mortality

more than TPA in MI patientsmore than TPA in MI patients RPA 7.47% vs TPA 7.24 RPA 7.47% vs TPA 7.24

Risk difference 0.23 % (2-sided 95%CI, -0.66 to 1.10 Risk difference 0.23 % (2-sided 95%CI, -0.66 to 1.10 percent). percent).

Incorrect conclusion: No difference in mortalityIncorrect conclusion: No difference in mortality Correct conclusion: not sure what the difference in Correct conclusion: not sure what the difference in

mortality ismortality is

Non-inferiority TrialsNon-inferiority Trials

E.g. St. Johns Wort vs ParoxetineE.g. St. Johns Wort vs Paroxetine Designed to show that SJW was no worse than Designed to show that SJW was no worse than

paroxetine at decreasing HamD scoresparoxetine at decreasing HamD scores


E.g. St. Johns Wort vs ParoxetineE.g. St. Johns Wort vs Paroxetine Key pointsKey points

Defined the “no worse” margin a prioriDefined the “no worse” margin a priori Basically they were saying thatBasically they were saying that

E.g. If paroxetine reduced HamD by 15 points thenE.g. If paroxetine reduced HamD by 15 points then Then the worse case end of the confidence interval for the Then the worse case end of the confidence interval for the

difference between SJW and paroxetine would have to be difference between SJW and paroxetine would have to be no more than 2.5 pointsno more than 2.5 points


E.g. St. Johns Wort vs ParoxetineE.g. St. Johns Wort vs Paroxetine Paroxetine HamD decrease 11.4Paroxetine HamD decrease 11.4 SJW HamD decrease 14.4SJW HamD decrease 14.4 The difference is 3 points more with SJWThe difference is 3 points more with SJW

The range of the difference is 1.5 to 4.0The range of the difference is 1.5 to 4.0 The worse case is only a 1.5 differenceThe worse case is only a 1.5 difference This worse case is better than a -2.5 difference (the This worse case is better than a -2.5 difference (the

defined margin)defined margin) Hence non-inferiority is provenHence non-inferiority is proven


E.g. St. Johns Wort vs ParoxetineE.g. St. Johns Wort vs Paroxetine Hypothetically non-inferiority would not have been proven Hypothetically non-inferiority would not have been proven

ifif Paroxetine decrease was 15Paroxetine decrease was 15 SJW decrease was 13SJW decrease was 13 The difference was -2 (range -4 to -1)The difference was -2 (range -4 to -1) The worse case for the difference is -4 pointsThe worse case for the difference is -4 points This is lower than the margin on -2.5 pointsThis is lower than the margin on -2.5 points

Hence SJW would be considered not non-inferior BUT Hence SJW would be considered not non-inferior BUT need to look at per protocol analysis need to look at per protocol analysis


Per protocol and Intention to treat should be done for non-Per protocol and Intention to treat should be done for non-inferiority trialsinferiority trials Per protocolPer protocol

Randomization benefit lost hence many differences between treatment Randomization benefit lost hence many differences between treatment groupsgroups

Analyzing only those that follow protocol so some randomized people are Analyzing only those that follow protocol so some randomized people are censoredcensored

As a result it becomes harder to prove one agent is no worse than another As a result it becomes harder to prove one agent is no worse than another because there are now more confounding variablesbecause there are now more confounding variables

Authors should always see similar findings in both analyses to Authors should always see similar findings in both analyses to support a non-inferiority claimsupport a non-inferiority claim

E.g. St. Johns Wort vs ParoxetineE.g. St. Johns Wort vs Paroxetine The worse case was a 0.7 point difference hence non-inferiority was The worse case was a 0.7 point difference hence non-inferiority was

provenproven

Equivalence TrialsEquivalence Trials

3 month vs 6 month follow up of BP patients 3 month vs 6 month follow up of BP patients by GPsby GPs Designed to show that the difference between 3 Designed to show that the difference between 3

and 6 month visit would be less than 10%, either and 6 month visit would be less than 10%, either better or worse (plus/minus 5mmHg)better or worse (plus/minus 5mmHg)

Equivalence TrialsEquivalence Trials

3 month vs 6 month follow up of BP patients 3 month vs 6 month follow up of BP patients by GPsby GPs Equivalence was proven for 6 month visits vs 3 Equivalence was proven for 6 month visits vs 3

month visitsmonth visits

Is Sample Size Different?Is Sample Size Different?

Sample size needs to be larger for a non-Sample size needs to be larger for a non-inferiority trialinferiority trial Harder to show differences within small rangeHarder to show differences within small range Need more people to be that preciseNeed more people to be that precise I.e. expecting small differencesI.e. expecting small differences

Small differences will only surface with large numbers Small differences will only surface with large numbers of patientsof patients

Can You Switch?Can You Switch?

If you prove non-inferiority can you then If you prove non-inferiority can you then conclude superiority?conclude superiority? Yes, as the sample size for needed for superiority Yes, as the sample size for needed for superiority

would be met by a non-inferiority trialwould be met by a non-inferiority trial As long as it is a pre-specified analysisAs long as it is a pre-specified analysis P-value needs to be re-calculatedP-value needs to be re-calculated Clinical relevance of superiority needs to be Clinical relevance of superiority needs to be

thought of thought of

Can You Switch?Can You Switch?

If you prove non-inferiority can you then If you prove non-inferiority can you then conclude superiority?conclude superiority? E.g. SJW vs paroxetineE.g. SJW vs paroxetine

SJW decreased HamD more than paroxetineSJW decreased HamD more than paroxetine They pre-specified that they would do thisThey pre-specified that they would do this BUT they di not calculate a new p-value BUT they di not calculate a new p-value

Non-inferiority MarginNon-inferiority Margin

This is the most important thing to look atThis is the most important thing to look at Needs to be chosen ahead of timeNeeds to be chosen ahead of time Needs to be based on statistical and clinical Needs to be based on statistical and clinical

reasoningreasoning Should be derived from the benefit seen with Should be derived from the benefit seen with

standard therapy over placebostandard therapy over placebo

Non-inferiority MarginNon-inferiority Margin

Should be derived from the benefit seen with standard Should be derived from the benefit seen with standard therapy over placebotherapy over placebo E.g Drug A is standardE.g Drug A is standard

Reduces MIs by 2% vs placebo (95%CI 1-4)Reduces MIs by 2% vs placebo (95%CI 1-4) Drug B is new and is being studied vs Drug A in a non-Drug B is new and is being studied vs Drug A in a non-

inferiority trialinferiority trial The margin should be set as no worse than 1% mortality The margin should be set as no worse than 1% mortality

difference with B vs Adifference with B vs A 1% comes from the worse case of the 95%CI versus placebo of 1% comes from the worse case of the 95%CI versus placebo of

standard therapystandard therapy Should not be a margin of 2% (this doesn’t take into account the Should not be a margin of 2% (this doesn’t take into account the

uncertainity in the benefit of Drug A vs placebo)uncertainity in the benefit of Drug A vs placebo)

ChecklistChecklist

Is this equivalence or non-inferiority?Is this equivalence or non-inferiority? Is there a margin pre-specified?Is there a margin pre-specified? Is the margin appropriately justified by authors Is the margin appropriately justified by authors

or is it arbitrary?or is it arbitrary? Did they do a per protocol and an ITT Did they do a per protocol and an ITT

analysis?analysis? Did they say they would look at superiority Did they say they would look at superiority

ahead of time and was a p-value re-calculated?ahead of time and was a p-value re-calculated?

Remember…Remember…

If no statistially significant difference seen in a If no statistially significant difference seen in a superiority trialsuperiority trial DO NOT conclude “absence of a difference”DO NOT conclude “absence of a difference” Conclude there is “absence of evidence of a Conclude there is “absence of evidence of a

difference”difference”

Checklist cont’dChecklist cont’d

If they claimed non-inferiority after they If they claimed non-inferiority after they couldn’t show superiority did they check to couldn’t show superiority did they check to see if they had enough power to do so?see if they had enough power to do so? They would need more people to conclude non-They would need more people to conclude non-

inferiority (hence probably under-powered) or inferiority (hence probably under-powered) or have enrolled more than needed for superiority so have enrolled more than needed for superiority so the seocnd analysis would be OK?the seocnd analysis would be OK?

Surrogate OutcomesSurrogate Outcomes

Outcomes that are substitutes for measures of Outcomes that are substitutes for measures of how a person functions, feels, or if they how a person functions, feels, or if they survivesurvive

Which are surrogates?Which are surrogates? BPBP LDL cholesterolLDL cholesterol HbA1CHbA1C StrokeStroke

What is a Serious Adverse Event?What is a Serious Adverse Event?

An adverse event is any undesirable experience associated with the use of a An adverse event is any undesirable experience associated with the use of a medical product in a patient. The event is SERIOUS and should be reported when medical product in a patient. The event is SERIOUS and should be reported when the patient outcome is: the patient outcome is:

Death Death Report if the patient's death is suspected as being a direct outcome of the adverse Report if the patient's death is suspected as being a direct outcome of the adverse

event.event. Life-Threatening Life-Threatening Report if the patient was at substantial risk of dying at the time of the adverse event Report if the patient was at substantial risk of dying at the time of the adverse event

or it is suspected that the use or continued use of the product would result in the or it is suspected that the use or continued use of the product would result in the patient's death.patient's death.

ExamplesExamples: Pacemaker failure; gastrointestinal hemorrhage; bone marrow : Pacemaker failure; gastrointestinal hemorrhage; bone marrow suppression; infusion pump failure which permits uncontrolled free flow resulting suppression; infusion pump failure which permits uncontrolled free flow resulting in excessive drug dosing. in excessive drug dosing.

Hospitalization (initial or prolonged) Hospitalization (initial or prolonged) Report if admission to the hospital or prolongation of a hospital stay results because Report if admission to the hospital or prolongation of a hospital stay results because

of the adverse event.of the adverse event. ExamplesExamples: Anaphylaxis; pseudomembranous colitis; or bleeding causing or : Anaphylaxis; pseudomembranous colitis; or bleeding causing or

prolonging hospitalization.prolonging hospitalization.

What is a Serious Adverse Event?What is a Serious Adverse Event?

Disability Disability Report if the adverse event resulted in a significant, persistent, or permanent change, Report if the adverse event resulted in a significant, persistent, or permanent change,

impairment, damage or disruption in the patient's body function/structure, physical activities impairment, damage or disruption in the patient's body function/structure, physical activities or quality of life.or quality of life.

ExamplesExamples: Cerebrovascular accident due to drug-induced hypercoagulability; toxicity; : Cerebrovascular accident due to drug-induced hypercoagulability; toxicity; peripheral neuropathy. peripheral neuropathy.

Congenital Anomaly Congenital Anomaly Report if there are suspicions that exposure to a medical product prior to conception or during Report if there are suspicions that exposure to a medical product prior to conception or during

pregnancy resulted in an adverse outcome in the child.pregnancy resulted in an adverse outcome in the child. ExamplesExamples: Vaginal cancer in female offspring from diethylstilbestrol during pregnancy; : Vaginal cancer in female offspring from diethylstilbestrol during pregnancy;

malformation in the offspring caused by thalidomide. malformation in the offspring caused by thalidomide. Requires Intervention to Prevent Permanent Impairment or Damage Requires Intervention to Prevent Permanent Impairment or Damage Report if you suspect that the use of a medical product may result in a condition which Report if you suspect that the use of a medical product may result in a condition which

required medical or surgical intervention to preclude permanent impairment or damage to a required medical or surgical intervention to preclude permanent impairment or damage to a patient.patient.

ExamplesExamples: Acetaminophen overdose-induced hepatotoxicity requiring treatment with : Acetaminophen overdose-induced hepatotoxicity requiring treatment with acetylcysteine to prevent permanent damage; burns from radiation equipment requiring drug acetylcysteine to prevent permanent damage; burns from radiation equipment requiring drug therapy; breakage of a screw requiring replacement of hardware to prevent malunion of a therapy; breakage of a screw requiring replacement of hardware to prevent malunion of a fractured long bone.fractured long bone.

SAE ReportingSAE Reporting

Required in all clinical trialsRequired in all clinical trials Problem…Problem…

Not sure which outcomes are included in SAE Not sure which outcomes are included in SAE totalstotals

Not always reported in publicationsNot always reported in publications

Example: Finasteride for BPHExample: Finasteride for BPH

TI Letter #58 Jan-Mar 2006

Example: Finasteride for BPHExample: Finasteride for BPH

TI Letter #58 Jan-Mar 2006

e.g. Lumiracoxib (TARGET Trial)e.g. Lumiracoxib (TARGET Trial)

Lancet 2004; 364: 665–74

e.g. Lumiracoxib (TARGET Trial)e.g. Lumiracoxib (TARGET Trial)GI Adverse EventsGI Adverse Events

Lancet 2004; 364: 665–74

e.g. Lumiracoxib (TARGET Trial)e.g. Lumiracoxib (TARGET Trial)CV Adverse EventsCV Adverse Events

Lancet 2004; 364: 665–74

e.g. Lumiracoxib (TARGET Trial)e.g. Lumiracoxib (TARGET Trial)SAE Data Not Reported in Published SAE Data Not Reported in Published

TrialTrial Total number of patients with SAEs both substudies Total number of patients with SAEs both substudies

pooled: pooled: Lumiracoxib 588 (6%), NSAIDs 566 (6%) Lumiracoxib 588 (6%), NSAIDs 566 (6%)

Total number of patients with SAEs ibuprofen sub-Total number of patients with SAEs ibuprofen sub-study: study: Lumiracoxib 297 (7%), Ibuprofen 272 (6%) Lumiracoxib 297 (7%), Ibuprofen 272 (6%)

Total number of patients with SAEs naproxen sub-Total number of patients with SAEs naproxen sub-study: study: Lumiracoxib 291 (6%), Naproxen 294 (6%)Lumiracoxib 291 (6%), Naproxen 294 (6%)

Regardless of drug attributionRegardless of drug attribution

Data received via personal communication with Dr. Hawkey, March 2007

TNT SAE Data RequestTNT SAE Data Request From:From: John C LaRosa [mailto:[email protected]] John C LaRosa [mailto:[email protected]]

Sent:Sent: Tuesday, August 14, 2007 1:53 PM Tuesday, August 14, 2007 1:53 PMTo:To: Tejani, Aaron Tejani, AaronSubject:Subject: Re: TNT Serious adverse event data request Re: TNT Serious adverse event data request

David David When you return, can you have someone provde me with answers? When you return, can you have someone provde me with answers? Thanks Thanks John John

"Tejani, Aaron" <[email protected]>"Tejani, Aaron" <[email protected]> 08/14/2007 04:27 PM 08/14/2007 04:27 PM To<[email protected]> ccSubjectTNT Serious adverse event data requestTo<[email protected]> ccSubjectTNT Serious adverse event data request

Dr. La Rosa We are reviewing the TNT trial with our pharmacy students and were wondering if you were Dr. La Rosa We are reviewing the TNT trial with our pharmacy students and were wondering if you were able to answer 2 questions: able to answer 2 questions: 1. Could you provide us with the serious adverse event (SAE) rates in both groups? 1. Could you provide us with the serious adverse event (SAE) rates in both groups? 2. Were the components of the composite outcome considered and counted as SAEs? E.g were Mis 2. Were the components of the composite outcome considered and counted as SAEs? E.g were Mis included in the SAE totals? included in the SAE totals? Many thanks, Many thanks, Aaron Aaron

Sample Size and Study PowerSample Size and Study Power

What it takes to calculate sample sizeWhat it takes to calculate sample size Relation of sample size to the primary Relation of sample size to the primary

outcomeoutcome Issues related to secondary outcomes, subgroup Issues related to secondary outcomes, subgroup

differences, etcdifferences, etc

Components of a Sample Size Components of a Sample Size CalculationCalculation

PowerPower The ability to detect a difference that truly existsThe ability to detect a difference that truly exists Type II error (beta): missing a difference that Type II error (beta): missing a difference that

exists i.e insufficient powerexists i.e insufficient power E.g 80% power means a 20% chance of missing E.g 80% power means a 20% chance of missing

a true differencea true difference Power= 1 - betaPower= 1 - beta


Level of significanceLevel of significance An alpha level must be chosenAn alpha level must be chosen Alpha relates to Type I errorAlpha relates to Type I error Type I error= detecting an effect when none existsType I error= detecting an effect when none exists Chosen alpha becomes your p-valueChosen alpha becomes your p-value E.g. alpha=0.05 then p of less than 0.05 is significantE.g. alpha=0.05 then p of less than 0.05 is significant What does p-value (= 0.0X) than what we choose tell us?What does p-value (= 0.0X) than what we choose tell us?

Treatment effect found is due to chance X % of the timeTreatment effect found is due to chance X % of the time


Underlying population event rateUnderlying population event rate Look at previous studies to get thisLook at previous studies to get this E.g TARGET TrialE.g TARGET Trial

Lumiracoxib versus naproxenLumiracoxib versus naproxen What is the expected rate of GI complications with What is the expected rate of GI complications with

NaproxenNaproxen


Size of treatment effectSize of treatment effect This should be the minimal clinically important This should be the minimal clinically important

differencedifference This should be justifiedThis should be justified


Adjusted sample size requirement for lack of Adjusted sample size requirement for lack of compliancecompliance

Achievable treatment effect (which is a Achievable treatment effect (which is a component of the sample size calculation) is component of the sample size calculation) is dependent on compliance to treatmentdependent on compliance to treatment

E.g trial with 100 people per arm if 100% E.g trial with 100 people per arm if 100% compliance…compliance…

If only 80% compliance, need 280 per armIf only 80% compliance, need 280 per arm

ChecklistChecklist

MJA 2002;157:256-7.

General CommentsGeneral Comments

In calculating a sample size a balance is In calculating a sample size a balance is required between risk of a Type I error versus required between risk of a Type I error versus a Type II errora Type II error

Sample size and all assumptions apply to only Sample size and all assumptions apply to only the primary endpointthe primary endpoint Likely that Type I and II errors will occur for Likely that Type I and II errors will occur for

anything other than the primary endpoint, even anything other than the primary endpoint, even with subgroupswith subgroups

i.e. be aware that false negatives and false positives i.e. be aware that false negatives and false positives likely to occurlikely to occur

Remember…Remember…

““Torture numbers and they will confess to Torture numbers and they will confess to anything.”anything.”

Gregg EasterbrookGregg Easterbrook

Definition of PICODefinition of PICO

Four components of Four components of PICOPICO Used to formulate a clinical question before you read the trialUsed to formulate a clinical question before you read the trial

PPatient or problematient or problem Description of patient or target disorderDescription of patient or target disorder

IInterventionntervention Could include exposure, diagnostic test, prognostic factor, therapy or Could include exposure, diagnostic test, prognostic factor, therapy or

patient’s perceptionpatient’s perception

CComparison interventionomparison intervention Relevant most often when looking at therapy questions Relevant most often when looking at therapy questions

OOutcomeutcome Clinical outcome of interest to you and your patientClinical outcome of interest to you and your patient DOES NOT HAVE TO BE WHAT THE AUTHORS MEASURED!DOES NOT HAVE TO BE WHAT THE AUTHORS MEASURED!

Arch Int Med 2001;134:657-62

CASP RCT ChecklistCASP RCT Checklist

User’s GuideUser’s Guide

Appraising Systematic Reviews and Appraising Systematic Reviews and Meta-analysesMeta-analyses

First 2 Screening QuestionsFirst 2 Screening Questions

1. Did the review ask a focused question?1. Did the review ask a focused question? PICO?PICO? Recommend thinking of your PCIo first before Recommend thinking of your PCIo first before

reading the reviewreading the review 2. Did the review include the right type of 2. Did the review include the right type of

study?study? If it was therapy did they look at RCTs only?If it was therapy did they look at RCTs only?

Finding Studies to IncludeFinding Studies to Include

Should search at least these databases:Should search at least these databases: Medline/PubmedMedline/Pubmed EMBASEEMBASE Cochrane’s CENTRAL database of RCTsCochrane’s CENTRAL database of RCTs

No language restrictionsNo language restrictions Should also searchShould also search

Reference lists, experts, conference proceedings, Reference lists, experts, conference proceedings, unpublished studiesunpublished studies

Assessing Quality of StudiesAssessing Quality of Studies

What are quality indicators of clinical trials?What are quality indicators of clinical trials? If you measure quality need to do something If you measure quality need to do something

with the informationwith the information Do sensitivity analysesDo sensitivity analyses

Extracting dataExtracting data

Need at least 2 people extracting dataNeed at least 2 people extracting data Need a standard data extraction formNeed a standard data extraction form Why?Why?

To avoid transcription errorsTo avoid transcription errors

Was it appropriate to Meta-analyze?Was it appropriate to Meta-analyze?

Does it make sense to combine data from Does it make sense to combine data from included trials?included trials?

Was heterogeneity assessed?Was heterogeneity assessed? If found were reasons for this explored?If found were reasons for this explored? If found was random effects model used to analyze If found was random effects model used to analyze

data?data?

Forest PlotsForest Plots

Final Notes on Systematic ReviewsFinal Notes on Systematic Reviews

Systematic, reproducible, defendableSystematic, reproducible, defendable Only as good as the trials includedOnly as good as the trials included Main questions to ask:Main questions to ask:

Did they attempt to get all the trials?Did they attempt to get all the trials? Did they compile the important information and Did they compile the important information and

critically appraise each trial that was included?critically appraise each trial that was included? Should be the first place you go to answer Should be the first place you go to answer

clinical questions…clinical questions…

CASP SR ChecklistCASP SR Checklist

User’s Guide for OverviewsUser’s Guide for Overviews

Questions or Comments

In dwelling, live close to the ground. In dwelling, live close to the ground. In thinking, keep to the simple. In thinking, keep to the simple. In conflict, be fair and generous. In conflict, be fair and generous. In governing, don't try to control. In governing, don't try to control. In work, do what you enjoy. In work, do what you enjoy. In family life, be completely present.In family life, be completely present.When you are content to be simply yourself When you are content to be simply yourself and don't compare or compete, everybody will respect you.and don't compare or compete, everybody will respect you.

Tao Te ChingTao Te ChingVerse 8Verse 8

AcknowledgementsAcknowledgements

Therapeutic Initiative GroupTherapeutic Initiative Group CASPCASP

Critical Appraisal Skills Program (CASP)Critical Appraisal Skills Program (CASP) JAMA Users’ Guides to Evidence-based JAMA Users’ Guides to Evidence-based

practicepractice BandolierBandolier Fraser Health ResearchFraser Health Research Susan and Rosa!Susan and Rosa!

Date post:	26-Dec-2015
Category:	Documents
Upload:	marian-matthews
View:	217 times
Download:	2 times

How to Interpret Research Evidence EBM Workshop September.2007 Aaron Tejani...

Documents