Critical appraisal of the literatureMichael Ferenczi

Head of Year 4Head of Molecular Medicine Section,

National Heart and Lung Institute

Trends in MEDLINE journal articles, 1978–2001 (n = 8,123,392)

Appraisal: increases the effectiveness of reading by enabling you to exclude research studies that are too poorly designed to inform practice.

This frees time to concentrate on a more systematic evaluation of those studies that cross the quality threshold and then to extract their salient points.

Why is this necessary?

So much information is now easily available, or forced upon us, that it is not possible to read or study it all.

Selection is required

Criteria for selection need to be established

Selection criteria:

* Systematically evaluating scientific literature.

* Sifting the "wheat" from the "chaff" when your literature search harvests conflicting studies

* Filtering out original research or meta-analyses which are methodologically sound

* Deciding which papers are going to be relevant

* Breaking down barriers between research (pure science) and practice (applied science) (e.g. in medical practice)

* Supporting the development of Evidence Based Practice (EBP).

Selection criteria:

1. Relevance - the topic of the study, how important it is to you at the moment, the similarity of the setting to the one you work in, the level at which it is written, the professional group or discipline for whom it is written, etc.

2. Intrinsic factors - (i.e. those factors that relate to the study itself) - the appropriateness of the study design to the question being asked, the suitability of the sample, the methods used to recruit the sample, methods used to obtain the results, etc.

3. Extrinsic factors - (i.e those external factors which are assumed (but are not necessarily) to be associated with the quality of the article) - who wrote it, where they work, what their job or qualifications are, whether you have heard of them, who paid for the study, which journal it is written in, whether they have written on the subject before, etc.

Are extrinsic factors acceptable, or are they too subjective?

Evidence Based Medicine Working Group

A. Are the results of the study valid? [Validity]Are the conclusions justified by the description of the methodology and the findings? Is the methodology sound, have the authors made reasonable assumptions, are there confounding factors they have failed to consider? If they are using a sample, have they selected this to avoid bias?

B. What are the results? [Reliability]What are the findings of this article? Is the effect demonstrated large enough to be of significance? How confident are we that the results fall within the bounds of reasonable expectation and are not fluke?

C. Will the results help locally? [Applicability]Are the problems I deal with sufficiently like those in the study to extrapolate the findings? Can I generalise from this study to my work place, my experimental conditions, my patients?

Appraisal of reviews1. Does the review set out to answer a precise question? (i.e. it is something more scientific than an attempt to gather everything written about malaria).

2. Has a thorough search been carried out? Anything missing?- Medline and other relevant bibliographic database- Cochrane controlled clinical trials register - Foreign language literature - "Grey literature" (unpublished or unindexed reports: theses, conference proceedings, internal reports, pharmaceutical industry)- Reference chaining from any articles found- Personal approaches to experts in the field - Hand searches of the relevant specialised journals.- (for meta-analysis, it may be important to track down the raw study data for re-analysis, rather than the final report)

3. Have the authors included explicit inclusion and exclusion criteria for studies, taking account of the patients in the studies, the interventions used, the outcomes recorded and the methodology? 4. Are results applicable locally

Evidence Based Medicine Working Group

A. Are the results of the study valid? [Validity]Are the conclusions justified by the description of the methodology and the findings? Is the methodology sound, have the authors made reasonable assumptions, are there confounding factors they have failed to consider? If they are using a sample, have they selected this to avoid bias?

B. What are the results? [Reliability]What are the findings of this article? Is the effect demonstrated large enough to be of significance? How confident are we that the results fall within the bounds of reasonable expectation and are not fluke?

C. Will the results help locally? [Applicability]Are the problems I deal with sufficiently like those in the study to extrapolate the findings? Can I generalise from this study to my work place, my experimental conditions, my patients?

Evaluating websitesDo not believe all what you read – be critical

The URL

.com, .biz and .co addresses suggest commercial bodies; .org suggests non-profit making sites; .edu and .ac suggest academic institutions; .gov, .doh etc. suggest government departments.

What is a paper?

• A critical report of a scientific study.

What factors will determine where you publish?

• Selecting a journalIs the work suitable for a general or specialist

audience?

– Which journal attract this readership

– Impact factor of the journal

• The peer review process

The Impact Factor

• More than 6000 journals in Medicine and Life Sciences. Journals are ranked according to the number of citations for each article in the most recent 3 years

• The Impact factor, often abbreviated IF, is a measure of the citations to science and social science journals.

• It is frequently used as a proxy for the importance of a journal to its field.

• It is the average number of citations in a year given to those papers in a journal that were published during the two preceding years

The Impact Factor• Abbreviated Journal Title ISSN Total Cites Impact Factor Immediacy Index Articles

Cite Half-lifeN NAT REV DRUG DISCOV 1474-1776 4445 18.78 3.36 66 2.6

N NAT REV GENET 1471-0056 7753 19.21 5.48 75 3.3

N NAT CELL BIOL 1465-7392 17741 19.72 3.95 135 3.9

A ANNU REV PHARMACOL 0362-1642 5994 19.83 5.79 29 6.3

C CHEM REV 0009-2665 51878 20.87 4.52 132 7.4

N NAT REV NEUROSCI 1471-0048 8447 20.95 3.29 75 3.4

E ENDOCR REV 0163-769X 10851 22.54 3.28 40 7.1

N NAT BIOTECHNOL 1087-0156 20914 22.74 5.21 124 4.3

J JAMA-J AM MED ASSOC 0098-7484 95715 23.33 5.08 380 6.5

L LANCET 0140-6736 131616 23.41 6.25 423 7.1

A ANNU REV CELL DEV BI 1081-0706 7097 23.69 0.86 28 6.3A ANNU REV NEUROSCI 0147-006X 8563 24.18 2.26 19 6.3

N NAT GENET 1061-4036 52387 25.8 5.92 190 5.6

N NAT IMMUNOL 1529-2908 16989 27.01 5.36 130 3.2

P PHYSIOL REV 0031-9333 14943 28.72 4.79 33 6.7

N NAT MED 1078-8956 40386 28.88 6.6 155 5

N NATURE 0028-0836 372784 29.27 5.83 1065 7.5

C CELL 0092-8674 132371 29.43 6.24 319 8.4

N NAT REV MOL CELL BIO 1471-0072 11438 29.85 6.23 80 3.2

R REV MOD PHYS 0034-6861 19446 30.25 5.63 30 >10.0

N NAT REV IMMUNOL 1474-1733 8686 30.46 3.79 72 2.8

S SCIENCE 0036-8075 345991 30.93 6.4 827 7.3

N NAT REV CANCER 1474-175X 9823 31.69 3.94 77 2.9

A ANNU REV BIOCHEM 0066-4154 16313 33.46 4.86 28 7.9

N NEW ENGL J MED 0028-4793 167894 44.02 13.42 308 6.9

A ANNU REV IMMUNOL 0732-0582 14745 47.4 10.83 29 6

C CA-CANCER J CLIN 0007-9235 4218 49.79 21.3 20 3

WHO CONTROLS WHAT IS PUBLISHED?The peer review system

• Editor: • selects 3 - 4 expert referees • Referees: • critically appraise the work and report back to

the editor• Editor:• evaluates the referees’ reports and makes a

decision based on• Quality of the work• Importance of the work and potential for

citation• Priority for publication.

Reading and analysing a scientific paper

What does a paper comprise?

• Abstract/Summary

• Introduction

• Materials and Methods

• Results

• Discussion

• References

• Acknowledgements

Approaching the paper

Title and short title: these will give and indication of the broad area

Abstract/summary: this is the overview.

Ask yourself - is this of interest to me?

What is it? Background information which leads up to the hypothesis to be tested.

Questions to askIs it a fair reflection of the literature?Is the hypothesis sound?

The introduction

Materials and MethodsWhat should it include?

A clear description of the design of the study and the methods used.

Questions to ask

Is the fundamental design appropriate?

Are the methods suitable?

Could you repeat the study with the information provided?

ResultsWhat should this section include?

A clear description of the data, drawing attention to points of significance

Presentation of the data as figures.

Questions to ask

Are the data expressed clearly?

Are the appropriate control data included?

Have the data been tested statistically?

Do the data show what the authors claim?

Discussion

This section should review the data critically in the light of

• the current literature• limitations posed by the design

of the study and the methods used.

Discussion

Questions to ask

Do the data justify the interpretations?

Are there other possible interpretations?

Are relevant published findings considered?

Are the limitations of the study considered?

Is the work a significant advance?

References

• Do these reflect fairly and appropriately the current state of knowledge?

• Is important work ignored (e.g. from competitors)?

• Is my relevant work cited?

Presentation

• Is the quality of English good enough• Is the text clear and succinct• Is the data presentation optimal: number and

clarity of figures, and tables