Literature EvaluationLiterature Evaluation

Dr.Dr.Ramesh ParajuliRamesh Parajuli MS (Otorhinolaryngology, Head and Neck Surgery)MS (Otorhinolaryngology, Head and Neck Surgery)Chitwan Medical College Teaching Hospital, Bharatpur-10,Chitwan, NepalChitwan Medical College Teaching Hospital, Bharatpur-10,Chitwan, Nepal

Reasons for Surgeons to Read Medical Literature:

1.Improve Patient Care

2.Learn about Research

3.Educate Peers and Students about Clinical Care

To ensure Best Practice and Treatment for Patients

Doctor must :-• Access Medical Information• Acquire New Knowledge • Achieve Information Mastery in their field

• Without systematic approach to identify & critically appraise clinical research, doctor might become dependant on inappropriate or outdated information

Keeping Medical Informations up to date

Past: • Reading few journals every month• Occasional conferences

Present: • Internet • Information- available to everyone

• Over 10,000 new articles per week are archived by the National Library of Medicine, and at least twice that many are published worldwide. (Rudmik LR,2008)

• More than 2 millions/year articles published in the biomedical press

(scott Brown 7th edn)

• Most published articles are not translatable to clinical practice, or their methods are not clinically sound

Publication Bias:-Articles with Positive findings: Easier to get published

Articles with Negative findings: Author/Editors

• Publication Bias Biased sample of article

• As many as 30% of journal articles may contain errors. (Victor A F,2003)

• Among health professionals, medical literature is the preferred source of new knowledge

• Medical informations doubles every 5 years (Castillo D L)

To keep up with available clinical evidence, today’s dedicated physician would have to read 19 articles everyday, of which only15% would provide information of practical relevance. (Diana L C,2008)

Journal Article• Rigid structure• Extracting information easier

Format:1.Abstract2.Intorduction3.Methods4.Results5.Discussion6.Conclusion

1.Abstract–overview of • What the research is about• What it did• How it did• What is found • What those results mean

2.Introduction –• Review of previous research• Rationale for the research• What the research is aiming to do

3.Method: • Methods used in the Research

4.Result : • Describe the result found 5.Discussion: • What the results actually mean6.Conclusion: • Results of several experiments presented and discussed,

Implications of the research• Recommendations about further research or policy and

practice in the relevant area

Knowledge of the Structure and Function of the Sections of Journal Article:-

• Reduces the time to locate information• Improves efficiency of reading• Reduces the time to read the article

• Reading Articles UncriticallyAcceptance of Text and Arguments Flawed, Biased and Subjectively written

• Efficient Reading is not the only skill we need Critical Reading

Critical Reading:Suspending the Judgment on a text until

1.Have understood the message being put forward

2.Evaluated the evidence supporting that message

3.Evaluated the writer’s perspective

Critical Reading Checklist

•To avoid reading poor quality literature and To avoid reading poor quality literature and mastering irrelevant information mastering irrelevant information Systematic Systematic StrategyStrategy

•A skill that a busy otolaryngologist can learnA skill that a busy otolaryngologist can learn

•Critical reading check list Critical reading check list for critical readingfor critical reading

Critical Reading for Research Articles

• Are the limitations of the procedures clear?

• Is the methodology valid? (eg.sample size, sampling method)

• Are the results consistent with the objectives?

• Are the claims the author makes about his or her own research internally consistent?

• Are the diagrams clear to the reader?

• Critical Thinking/Critical Analysis:

””thethe intellectually disciplined process of actively and skillfully conceptualising, applying, analysing, synthesising and/or evaluating informations gathered from ,or generalised by, observation, experience, reflection, reasoning or communication ,as a guide to belief or action (or argument)” (Scriven & Paul 2001)

What critical analysis is NOT:

1.A summary of a text2.A descriptive comment on the text3.A rehash of the theory learnt in a class4.A statement of one’s own unsupported views

• Harm to pts can be aboutHarm to pts can be about:-:- Diagnosis:-Diagnosis:- Therapy:-Therapy:- Prognosis:-Prognosis:-

• To prevent the harm to the pts:-most To prevent the harm to the pts:-most up to dateup to date knowledgeknowledge available available

• Evidence Based Medicine-”evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients” (Sackett et al. 1996 )

• CONSCIENTIOUS - To do more good than harm - To do right things in right way - Carefulness and attention

• EXPLICIT - Openness and transparency -To evaluate and criticize the methods used - Doctor- Patient relationship

• JUDICIOUS - To evaluate and weigh the evidence of the work -To consider Benefit & balance it against harm• Current Best Evidence

Evidence Based Practice (EBP)• Begins and ends with patients

• Integrating individual clinical expertise with the best available external clinical evidence from systematic research

• Application of the knowledge to the prevention, diagnosis or management of the disease

Evidence Based Medicine –practice of medicine with an emphasis relying on the medical literature for clinical decision making

EBM approach-• Improves ability to evaluate clinical literature• Enhances life long learning skills in day to day medical practice• Keeping up to date • Continuing medical education (CME)

Purpose of EBM-• To propagate the evidence but not to replace individual’s

clinical experience

• One shouldn’t expect clinical practice to be 100%evidence based-better patient care shouldn’t be compromised by not counting on evidence when it is readily available by different sources.

• EBM is key component of modern medical practice

• EBM is necessary but not sufficient

• Otolaryngologists should acquire the fundamentals skills of searching, appraising, and synthesizing, required to practice EBM and should do so with the same diligence they apply to learning surgical skills

Critical Appraisal –one aspect of EBM• The ability to apply the principals of analysis • To identify those studies which are unbiased and valid.

Critical Appraisal of Research Studies helps to:- 1.Improve patient care 2.Improve teaching 3.Become a better researcher 4.Prepare for journal club 5.Gain respect among the peers

Practising Evidence Based Health Care:-EBM process involves 5 steps1.Ask clinical question:-formulation of “P I C O” question2.Search literature3.Appraisal (critically evaluate the literature)4.Share the knowledge5.Apply to the patient

• Forming Answerable Questions:

1.Background questions:-understand the problem in general2.Foreground questions:-decision making questions

• The PICO Format for Foreground Questions

1.P=Patient and Problem (population-kids, women, patients)

2.I=Intervention( test)

3.C=Comparison intervention (Control group)

4.O=Outcome

• To avoid reading poor quality literature and mastering irrelevant information Systematic Approach

1.Retrieve

2.Review

3.Reject

4.Read

• 5s model for Evidence-Based Health Care Decisions (Hynes R B; ACP J Club 2006;145:A8)

To answer a particular question: evidence Based Research Resources

1.The cochrane collaboration2.National library of medicine-gateway3.Open clinical4.bestBETs

Selection of medical database: • Type of question• Ease of use for a particular problem• Need to limit the search for highest quality studies

• Selecting an Article: Filtering Process1.Primary Survey (initial evaluation and brief overview)

(A) Analyse the Title :

(B) Review the list of authors:

(C) Read the summary or abstract beginning with conclusion: • If the conclusion if valid ,important to me?• Results if true, how useful they are?• Is the primary outcome measured important for me?• Do the interventions make sense?• Can the information be generalised to my patients?

Secondary Survey:1.Introduction : • Problem under study, context of the study, reasons for conducting study• Importance of the topic, what is known and what is unknown about the topic• Specific questions (objective, goal of the study, hypothesis) to be evaluated• Study sample, primary outcome & intervention being evaluated• Method design• Conclusion shouldn’t extend beyond the stated objective

2.Methods: • Research design-descriptive or comparative study• Single or multicenter

3. Study Sample: • How were the subjects and controls selected? • Are the inclusion & and exclusion criteria sufficiently clear to

describe the target population?

4.Treatment Allocation: • Randomization• Masking (blinding)

5.Outcomes: primary outcome-all studies

secondary outcome-some studies• How it was measured? • was the measurement free of bias ?• How reproducible were the results? • How to standardize measurements & to minimize inter-observer variability?

6.Statistical Analysis: • Statistical Tests

7.Results: • Tables and Figures, • How many pts eligible for the study? • How many enrolled?• How many completed?

8.Discussion : • Author’s interpretation of clinical relevance• Comparison with previous studies, similarities & differences• Limitation of the study• Suggest new directions for appropriate study

9.Conclusion:• Must consistent with the study objective• Justified by the study results• Shouldn’t over generalize the results of the study

• If there is not concealment of Randomizationmay exaggerate efficacy of the treatment by as much as 30% more

• If there is no Blinding may exaggerate the effectiveness of the treatment by 15%

• If <80% pts Followed Upresults become meaningless

Types of Study by Content:

1.Evaluation of a new therapy2.Evaluation of a new diagnostic test3.Determinnation of the etiology of a condition4.Prediction of the outcome5.Natural course of a condition

Levels of evidence• Level 1 : 1a-SR(homogeneity) of RCTs 1b-Individual RCT 1c-All or None• Level 2 :2a-SR(with homogeneity) of Cohort studies 2b-Individual Cohort Study (including low quality RCT; eg

<80% follow up) 2c-Outcomes research; ecological studies• Level 3 3a-SR(with homogeneity) of case control studies 3b-Individual case-control study• Level 4 - Case Series(& poor quality cohort & case control studies)• Level 5 - Expert opinion

Grades of Recommendation

• Grade A- consistent level 1 studies• Grade B- consistent level 2 or 3 studies or

extrapolations from level 1 studies• Grade C- level 4 studies or extrapolations

from level 2 or 3 studies• Grade D- level 5 evidence or troublingly

inconsistent or inconclusive studies of any level

once the trial or trials have been found that seems to answer the questions check three things:

1.Is the study valid?

2.What are the results?

3.Can these be applied to my patients?

(I) Appraising the Diagnostic Articles:1.Is the study valid? Did the authors answer the questions?1.What were the characteristics of the groups?2.Is it clear how the test was carried out?3.Is the test result reproducible?4.Was the reference standard (gold standard) appropriate?5.Were the reference standard & the diagnostic test interpreted blind

and independently of each other?6.Was the reference standard applied to all patients?7.Was the test evaluated on an appropriate spectrum of patients?

2. What were the results? • Are the sensitivity/specificity and/or likelihood ratios presented?• Could the results have occurred by chance?• Are there confidence limits?

Likelihood Ratio :-

• Likelihood Ratio (+ve results)= sensitivity (100% - specificity)• Likelihood Ratio (-ve results)= (100% - sensitivity) specificityIf LR > 10 + ve < 1 - ve

3. Will the Results help my Patients?

1.Is the diagnostic test available, affordable, accurate & precise in my setting?

2.Are the results applicable to my pts? Do my pts have a similar mix of disease severity & competing conditions?

3.Will the results change my case management ?will the information gain be sufficient to change a clinical decision (rule in or rule out)?

4.Will patients be better off as a result of performing the test?

(II) Appraising Therapy Articles1.IS THE STUDY VALID?

Did the authors answer the questions?1.What were the characteristics of the pts?2.Were the groups similar at the start of the trial?3.Aside from the experimental treatment, Were the groups treated equally?4.What was the treatment & what was it compared against placebo?5.Was randomization used?6.Were all pts followed up at its conclusion?7.Wre they analyzed in the groups to which they were randomized?8.Were blinding used?9.Was the length of study appropriate?

2.WHAT WERE THE RESULTS?Event rates-control event rate (CER), experimental event rate (EER)Relative risk CER-EERRelative risk reduction (RRR) = ------------------ CER Absolute risk reduction (ARR) = CER- EER

Number needed to treat (NNT) = 1 ------- ARR

3.WILL THE RESULTS HELP MY PATIENTS?

1.Can the results be applied to my patient care? Would my pts eligible for the study? Are there any reasons why the results shouldn’t be applied to my

pts?

2.Were all clinically important outcomes consider?

3.Are the likely treatment benefits worth the potential harm and costs?

(III) Appraising the Systematic Review1.IS THE REVIEW VALID?Did the authors answer the questions?

1.What databases & other sources did the authors of this review search?2.What were their eligibility criteria (inclusion & exclusion)? do these seems

appropriate?

3.Was there independent data extraction of the result by the reviewers (then compared later)?

4.Is there a description of the quality of each trial?5.Were the results consistent from study to study (homogeneous)?

(2)WHAT Were The Results? If the results of the studies have been combined, was

it reasonable to do so? consider whether:• the results of each study are clearly displayed?• the results were similar from study to study?

How are the results presented and what is the main result?

How the results are expressed (eg. odds ratio, relative risk etc)

How precise are these results?

3 .CAN I APPLY EVIDENCE FROM THIS SYSTAMATIC REVIEW IN CARING FOR MY PTS?

• Whether the population sample different from my population ?• Whether my local setting differs ?• Were all clinically important outcomes considered?• Should policy or practice change as a result of the evidence

contained in this review?• Whether any benefit reported outweighs any harm/or cost

(IV) Critical Appraisal Skills for the Cohort Study1.ARE THE RESULTS VALID?• Did the study address a clearly focused issue?

• Did the authors use an appropriate method to answer their question?

• Was the cohort recruited in an acceptable way? • Was the exposure accurately measured to minimize bias?• Was the outcome accurately measured to minimize bias?

• Have the authors identified all important confounding factors?

• Was the follow up of subjects complete enough?

• was the follow up of subjects long enough?

2.WHAT ARE THE RESULTS? • What are the results of this study?

• How precise are the results? • Do I believe the results?

3.WILL THE RESULTS HELP ME LOCALLY?

• Can the results be applied to the local population?

• Do the results of this study fit with other available evidence?

(V) Appraisal of a Case Control Study1.ARE THE RESULTS OF THE STUDY VALID?1.Did the study address a clearly focused issue?

2.Did the authors use an appropriate method to answer their question?

3.Were the cases recruited in an acceptable way?

4.Were the controls selected in an acceptable way?

5.Was the exposure accurately measured to minimize bias?

6.(A)what confounding factors have the authors accounted for? (B) Have the authors taken account of the potential confounding factors in

the design and/or in their analysis?

2.WHAT ARE THE RESULTS?

7.What are the results of this study?

8.How precise are the results? how precise is the estimate of risk?

9.Do I believe the results?

3.WILL THE RESULTS HELP ME LOCALLY?

10.Can the results be applied to the local population?

11.Do the results of this study fit with other available evidence?

REFERENCES:1.Scott-Brown’s Otorhinolaryngology, Head & Neck Surgery(7th edition)2.Critical appraisal skills programme (CASP)-making sense of evidence public health resource unit;england(2006)3.Evaluating the literature (emedicine)4.Unilearing-University of wollongong, australia5.Evidence based medicine (EBM)-what, why and how(KUMJ 2003)6CEBM-EBM tools7.Assesing the medical literature: let the buyer beware8.Knoweledge managemet:how to keep up with the literature9.Literature review(st.kate’s libraries guides)10.How to evaluate the literature(stanford university)11.Guidelines for evaluating writing about literature (john jay college of criminal justice)12.Critical analysis-so what does that mean(university of bradford)13.Evaluating the medical literature(clista clantin 2009)14.Literature evaluation (hussain al awami)15.Introduction to critical appraisal(jason curtis-NHS)

16.How to read a paper? (BMJ)17.Crafting the literature review (massey university;2006)18.Argument based medical ethics19.Assessment of the critical appraisal (american journal of surgery,2004)20.Teaching critical appraisal skills to medical students in obstetrics and

gynecololgy (university of california OBGYN department)21.A simple method for evaluating clinical literature (JF Robert)22.Evidence based medicine literature evaluation questions (washington state

university,2006)23.Evidence based medicine capitol conference 2007

Thank you