Download - Research 3.03 Critical Appraisal of Medical Journals 2 - Dr. Zulueta

!!

Trans!Group:!TERE,%BELLE,%CHEV,%ROG!Edited!By:!!

Subject:!Topic:!Lecturer:!Date:!

RESEARCH!CRITICAL%APPRAISAL%OF%MEDICAL%JOURNAL%2%Dr.!Jose!Luisito!A.!Zulueta,!M.D.!September!11,!2013!

Page!1!of!5!

!

OUTLINE%CRITERIA!

I. VALIDITY!CRITERIA!II. RESULTS!CRITERIA!III. APPLICABILITY!CRITERIA!

!OBJECTIVES%

At!the!end!of!the!lecture,!the!student!should!be!able!to:!1. Enumerate!the!criteria!used!to!appraise!an!article!on!

diagnosis.!2. Explain!the!important!points!in!the!criteria!used.!3. Appraise!published!research!and!judge!its!reliability.!

!References:%Dr.%Zulueta’s%PPT:%!

• Katz,!D.L.,!Elmore,!J.G.,!Wild,!D.M.G.,!Lucan,!S.C.,!Jekel’s!Epidemeology,!Biostatistics,!and!Preventive!Medicine,!Third!Edition,!W.B.!Saunders!Co.,!USA,!2014!

• !Longo,!D.,!Fauci!A.,!Kasper!D.,!Hauser!S.,!Jameson,!J.L.,!Loscalzo,!J.,!Harrison’s!Principles!of!Internal!Medicine,!18th!Edition,!McGraw[Hill!Companies,!USA,!2012.!

• !Zulueta!FM,!Perez!JR,!Methods!of!Research,!Thesis!Writing,!and!Applied!Statistics,!National!Bookstore,!Philippines,!2009.!

!Legend:!Italicized!–!quoted!from!the!lecturer!!

CRITERIA%VALIDITY%CRITERIA%

• Was!there!an!independent!blind!comparison!with!a!reference!standard?!!

! INFORMATION%BIAS%• Did!the!patient!sample!include!an!appropriate!spectrum!of!

patients!to!whom!the!diagnostic!test!will!be!applied!in!clinical!practice?!

! SELECTION%BIAS%• Did!the!results!of!the!test!being!evaluated!influence!the!

decision!to!perform!the!reference!standard?!! MISSING%DATA%BIAS%

• Were!the!methods!for!performing!the!test!described!in!sufficient!detail!to!permit!replication?!!

The,reference,standard,is,also,the,gold,standard.,If,that,particular,test,would,yield,a,positive,result,,then,most,likely,the,patient,has,the,disease.,,

!RESULTS%CRITERIA%

• Are! likelihood! ratios! for! the! test! results!presented!or!data!necessary!for!their!calculation!provided?!

!APPLICABILITY%CRITERIA%

• Will!the!reproducibility!of!the!test!result!and!its!interpretation!be!satisfactory!in!my!setting?!

• !Are!the!results!applicable!to!my!patient?!• !Will!the!results!change!my!management?!!• !Will!patients!be!better!off!as!a!result!of!the!test?!

!!!

I. VALIDITY%CRITERIA!1.%Was%there%an%independent%&%blind%comparison%with%a%reference%standard?%!To%Avoid%Information%Bias%(see%blinding)%

• !Comparator!test!should!be!the!gold!standard!• !Comparison!should!be!“blind”!• !Procedure!should!be!described!in!sufficient!detail!

%1) Gold!Standard!What!is!a!gold!standard?!

• The!accepted!definition!of!disease![!presence!or!absence!• !Test!that!is!100%!sensitive!and!100%!specific!!or!as!close!

as!one!can!get!to!100%!Examples:!

• Ovarian!CA!"!Histopath! !• PTB!!"!DSSM!(Direct!Sputum!Smear!Microscopy)!!

2) BLINDING!• Person/s!doing!the!diagnostic!test!&! person/s!doing!gold!

standard!should!be!unaware!of!each!other’s!results!o Test!results!are!objective!

ex.!Hormone%assays%" %“Machine”!interpretation%!

Person/s(doing(the(diagnostic(test,and,the,gold(standard,should,be,unaware,of,each,other’s,results.,That,is,why,some,clinicians,prefer,the,hormone,assay,because,this,is,a,test,which,has,machine,interpretation.,There,is,objectivity,in,them.,,

,Effect%of%nonYblinding:%Test!seems!better!than!it!really!Is!(Overestimate!of!accuracy)!!APPRAISAL:%

Was%there%an%independent%&%blind%comparison%with%a%reference%standard?%%

YES!!!2.%Did%the%patient%sample%include%an%appropriate%spectrum%of%patients%to%whom%the%diagnostic%test%will%be%applied%in%clinical%practice?%

!To%Avoid%Selection%Bias%

• !Subjects!should!be!representative!of!those!to!which!the!test!will!be!applied!in!clinical!practice!

%1)%Appropriate%spectrum%

• Represent!ALL!types!of!patients!to!which!the!test!will!be!applied!in!clinical!practice!# !Type!of!disease!$ !Benign!&!malignant!$ !Specific!disease!subtypes!# !Severity!of!disease!$ !Non[palpable!&!palpable!$ !Early!&!advanced!!$ Stage!I,!II,!III,!IV!

In,appropriate,spectrum,,we,want,a,diagnostic,test,to,be,able,to,catch,the,disease,in,its,earlier,stages.,!

!!!

Page!2!of!5!

!Figure%1.%NonYrepresentative%Sample%

• When,you,have,a,non@representative,sample,and,given,that,

severe/advanced,cases,are,easier,to,diagnose,and,mild,cases,are,

harder,to,diagnose,,you,might,be,including,only,those,who,are,

with,severe,stages,of,the,disease.,!• There,will,be,an,overestimation,of,accuracy,of,the,test,because,

the,test,will,actually,be,measuring,those,with,the,severe,disease,

who,are,easy,to,diagnose.,,

!%%%APPRAISAL:%%

Did%the%patient%sample%include%an%appropriate%spectrum%of%patients%to%whom%the%diagnostic%test%will%be%applied%in%clinical%

practice?%YES%

%3.%Did%the%results%of%the%test%being%evaluated%influence%the%decision%to%perform%the%reference%standard?%

!RULES%to%avoid%Missing%Data%Bias:!

$ All!subjects!should!undergo!the!gold!standard!test.!$ Result!of!the!test!should!NOT!influence!the!decision!to!

perform!the!gold!standard!test.!!

Which%is%more%correct?!“All!recruited!patients!underwent!biopsy“!

or!"All!patients!who!underwent!biopsy!were!recruited"!

**(The,first,one,is,correct),,

,Figure%2:%Subjects%and%Gold%standard%

• If,not,all,subjects,underwent,the,gold,standard,test,,you,are,only,

including, those, with, severe/advanced, cases, because, they, are,

easier, to, diagnose,, and, so, the, accuracy, of, the, test, is,

overestimated.,

• Diagnostic, test, should,be,able, to, catch, the,disease, in, its, further,

stages., It, will, prove, if, the, test, conducted, during, the, diagnostic,

test,would,get,the,same,results.,

,APPRAISAL:%%

Did%the%results%of%the%test%being%evaluated%influence%the%decision%to%perform%the%reference%standard?%

NO%%%

4. Were%the%methods%for%performing%the%test%described%in%sufficient%detail%to%permit%replication?%

!“Repeatability”%of%Test%Method!RULES:!

$ Procedure!for!doing!the!test!should!be!described!in!sufficient!detail.,(so,that,the,readers,would,be,able,to,follow,them)!

$ Readers!should!be!able!to!replicate!the!test!procedure.!!Vague!description!of!procedure!

!Readers!cannot!perform!test!as!well!as!study!authors!

!Readers!obtain!less!accurate!results!

!REPORTED!ACCURACY!OVERESTIMATES!

ACTUAL!ACCURACY!!

APPRAISAL:%Were%the%methods%for%performing%the%test%described%in%sufficient%

detail%to%permit%replication?%YES%%

APPRAISAL%SUMMARY%Y%VALIDITY%% To%avoid%information%bias:!

# Comparator!test!should!be!the!gold!standard!# Comparison!should!be!“blind”/objective!# Procedure!should!be!described!in!sufficient!detail!

% %To%avoid%selection%bias:%# Subjects!should!be!representative!of!those!to!

which!the!test!will!be!applied!in!clinical!practice!% %To%avoid%missing%data%bias:%

# All!subjects!should!undergo!the!gold!standard.!!1.!Was!there!an!independent!blind!comparison!with!a!reference!standard?!YES!2.!Did!the!patient!sample!include!an!appropriate!spectrum!of!patients!to!whom!the!diagnostic!test!will!be!applied!in!clinical!practice?!YES!3.!Did!the!results!of!the!test!being!evaluated!influence!the!decision!to!perform!the!reference!standard?%NO!4.!Were!the!methods!for!performing!the!test!described!in!sufficient!detail!to!permit!replication?!YES%

%%%

!!!

Page!3!of!5!

II.%RESULTS%CRITERIA%Are%likelihood%ratios%for%the%test%results%presented%or%data%necessary%for%their%calculation%provided?%%

• Measures!of!a!diagnostic!test:!# !Sensitivity!# !Specificity!# !Positive!predictive!value!# !Negative!predictive!value!# !Likelihood!ratio!

!

%Figure%3:%Table%used%to%calculate%the%Sensitivity,%Specificity,%Positive%

Predictive%Value,%and%Negative%Predictive%Value%%CLASSIC%2x2%Table%

!On,the,classic,2,x,2,table,,the,gold,standard,test,is,always,on,the,top,,diagnostic,test,on,the,left.,,False,positive,=,tested,positive,on,the,test,but,do,not,have,the,disease.,,False,negative,=,tested,negative,on,the,test,but,actually,has,the,disease.,,!

A. Sensitivity![!ability!of!the!test!to!label!as!positive!those!who!truly!have!the!disease;!used,when,you,want,to,rule(out,a,disease,(snOUT)!

!"#$%&%'%&( = ! !!!!!!! !

! ! Where:!A!=!Number!of!true!positives!C!=!Number!of!false!negatives!

Therefore:!

!"#$%&%'%&( = ! !". !"!!"#$!!"#$%$&'#!". !"!!"#$!!"#$%$&'# + !". !"!!"#$%!!"#$%&'"(!

*Sensitivity,test,example,is,a,screening,test,for,HIV,(ELISA).,,

When,negative,,you,can,say,that,the,person,do,not,have,HIV.,

Confirmatory,test,for,HIV,is,Western,Blot.,If,positive,,you,can,say,

that,the,person,has,HIV.,,

Sensitivity,determines,the,percentage,of,sick,people,who,are,

correctly,identified,as,having,the,condition,among,those,who,

really,have,the,condition.,

B. Specificity![!ability!of!the!test!to!label!as!negative!those!who!don’t!have!the!disease;!used,when,you,want,to,rule(in,a,disease,(spIN)!

!!!!!!!!! ! ! ! ! !"#$%&%$%'(! = !!!!!!!!

! Where:!! ! D!=!Number!of!true!negatives!! ! B!=!Number!of!false!positives!! Therefore:!! ! ! ! ! !

! !"#$%&%$%'( = ! !".!"!!"#$!!"#$%&'"(!".!"!!"#$!!"#$%&'"(!!".!"!!"#$%!!"#$%$&'#!

!Specificity,is,the,percentage,of,people,who,are,correctly,identified,as,not,having,the,condition,among,those,who,really,don’t,have,the,condition,%C. Positive%Predictive%Value%(PPV)![!probability!that!a!subject!has!

the!disease!if!the!test!is!positive!

!!!!!!!! ! ! ! ! !!!!!!"! = !!!!!!!

! Where:!! ! A!=!Number!of!true!positives!! ! B!=!Number!of!false!positives!! Therefore:!

!!" = ! !". !"!!"#$!!"#$%$&'#!". !"!!"#$!!"#$%$&'# + !". !"!!"#$!!"#$%&'"(!

! ! ! ! ! ! ! !D. Negative%Predictive%Value%(NPV)![!probability!that!a!subject!

does%not%have!the!disease!if!the!test!is!negative!

!"#!! = !! !! + !!

! Where:!! ! D!=!Number!of!true!negatives!! ! C!=!Number!of!false!negatives!! Therefore:!!

!"# = ! !". !"!!"#$!!"#$%&'"(!". !"!!"#$!!"#$%&'"( + !". !"!!"#$%!!"#$%&'"(!

%E. Likelihood%ratio%(LR)[!refers!to!how!much!the!test!will!raise!or!

lower!the!pretest!probability!%INTERPRETATION%of%LR:%

• !The!larger!the!value!from!1,!the!better!the!test!result!will!detect!those!who!have!the!disease!

Ideally:!(+)!LR!>!1! !• !The!smaller!the!value!from!1,!the!better!the!test!result!will!

detect!those!w/o!the!disease!!! ! ! ! Ideally:!([)!LR!<!1!

• !Value!of!1:!test!is!useless!!Other%Notes:%The!further!away!a!likelihood!ratio!(LR)!is!from!1,!the!stronger!the!evidence!for!the!presence!or!absence!of!disease.!

• LR%>1!indicates!that!the!test!result!is!associated!with!the!presence!of!the!disease.%

• LR%<0.1!indicates!that!the!test!result!is!associated!with!the!

!!!

Page!4!of!5!

absence!of!disease.%!

!!

!Figure%4:%LR%=%Diagnostic%Weights%

%LR%–%Statistical%Significance%

• !Ideally:!(+)!LR!>!1!! ! ! (+)!LR!=!16.6!(95%!CI:!2.5!–!24.5)! !! ! ! STAT!SIGNIFICANT!!

• !Ideally:!([)!LR!<!1!! ! ! ([)!LR!=!0.8!(95%!CI:!0.6!–!2.7)!! ! ! NOT!STAT!SIGNIFICANT!!LR!is!stat!significant!if!the!CI!does!NOT!cross!“1”.!!Interpret%the%following:!

CA!LR:!! ! ! 16.6!Susp!LR:! ! 3.31!Benign!LR:!! ! 0.21!

!$ Probability!of!CA!is!increased!17x!if!test!is!(+)!$ Probability!of!CA!is!increased!3x!if!test!result!shows!

“suspicious”.!$ Probability!of!CA!is!increased!0.21x!if!test!is!([)!$ Probability!of!CA!is!decreased!1!/!0.21x!or!4.76x!if!test!is!([).!

%APPRAISAL:%

%Are%likelihood%ratios%for%the%test%results%presented%or%data%necessary%for%their%calculation%provided?%%

YES%%%%

MANAGEMENT%%Decision%Making%Line%

* !PreYTest%Probability%Y!probability!of!disease!prior!to!doing!the!test!!

* !PostYTest%Probability[!probability!of!disease!after!doing!the!test!

* !Diagnostic%Threshold%* !Therapeutic%Threshold%* !Nomogram![!graphical!tool!for!estimating!how!much!the!

result!on!a!diagnostic!test!changes!the!probability!that!a!patient!has!a!disease.!

,PostYtest%Probability%(PTP)%

• Will!the!reproducibility!of!the!test!result!and!its!interpretation!be!satisfactory!in!my!setting?!!

!!PreYtest%Probability%

!%

,By, comparing, the, pre@, and, post@test, probabilities,, it, is, possible, to,determine,whether,probability,of,diagnosis,has,risen,(i.e.,the,post@test,probability, has, increased), or, fallen, (i.e., post@test, probability, has,decreased)., In, practice,, assessing, post@test, probability, is, commonly,done,by,using,a,Likelihood(Ratio(Nomogram.(

,Nomogram!Given!the!following!set!of!values:!!Pre[Test!Probability!=!60%!LR!(+)!=!1.47!Post[Test!Probability!~!70%!!!

!Figure%5:%Nomogram%

!To(better(understand,(here’s(the(explanation(we(got(from(other(resources:(

!!!

Page!5!of!5!

PreYtest%probability%(~%prevalence)!$ This!is!the!proportion!of!people!in!the!population!at!risk!

who!have!the!disease!at!a!specific!time!or!time!interval!(ex.!prevalence!of!the!disease).!In!other!words,!it%is%the%probability%−before%the%diagnostic%test%is%performed%−%that%a%patient%has%the%disease.!!

$ Pre[test!probabilities!may!be!estimated!from!routine!data,!practice!data!or!clinical!judgement!like!history!taking!and!physical!examination)!!

PostYtest%probability!$ This!is!the!proportion!of!patients!testing!positive!who!truly!

have! the! disease.! It! is! similar! to! the! positive! predictive!value!but!apart! from! the! test!performance!also! includes!a!patient!based!probability!of!having!disease.!

!EXAMPLE:!!

!

!!! We,begin,with,a,pretest,probability,estimate,of,10%,for,strep,as,the, cause, of, sore, throat, in, adults, presenting, to, a, primary, care,physician, with, a, sore, throat, as, their, "chief, complaint"., We, gather,information,by,asking, the,patient,a, series,of,questions,and,examine,them,(the,"history,and,physical").,Each,question,is,actually,a,type,of,"test",, and,we, use, this, information, to, revise, the, probability, from, a,pretest,probability,of,10%,to,a,post@test,probability,of,50%.,But,we're,not,done,(because,we,have,to,overcome,the,treatment,threshold),,we,may,order,additional,rapid,strep,test,using,a,throat,swab,,so,now,the,50%, is, the,pretest,probability, in, relation, to, the, throat, swab.,With,a,positive, result,,we, estimate, an, 80%, likelihood,of, strep,,which, is, the,final,post@test,probability.,!Pretest, probability/prevalence, at, 10, %,", , do, history, and, physical,

exam,", probability, raised, to, post, test, probability, 50%, , (needs,further,diagnostic,test,like,throat,swab,according,to,threshold,graph),

",50%,(now,becomes,the,pretest,probability),is,raised,to,80%,(post,

test,probability),",now,,treat,the,patient,since,it,already,surpassed,the,treatment,threshold,according,to,threshold,graph,,After,which,,you,can,graph,the,results,using,Nomogram,to,estimate,how,much,the,result,on,a,diagnostic,test,changes,the,probability,that,a,patient,has,a,disease.,!!%%%%%

III.%APPLICABILITY%CRITERIA%1) Will%the%reproducibility%of%the%test%result%and%its%interpretation%

be%satisfactory%in%my%setting?!Consider:!

$ objectivity!of!how!test!was!done!in!interpreting!test!results!$ comparison!of!test!results!with!other!studies!

,This,means,that,in,appraising,literature,,it,is,imperative,that,you,

compare,the,article,being,evaluated,with,other,published,studies.!!2) Are%the%results%applicable%to%my%patient?!Look!at:!!

$ demographic!characteristics!$ Inclusion!&!exclusion!criteria!

!!!!3) Will%the%results%change%my%management?!!

$ Look!at!diagnostic!&!therapeutic!thresholds.!!!4) Will%patients%be%better%off%as%a%result%of%the%test?!

Aside!from!determining!accuracy,!did!article!determine!if!the!test!will!improve!disease!prognosis?!

Example:!$ RCT!on!mammography!!$ longer!survival!for!mammo[detected!breast!CA!

%%%

GOODLUCK!%