Is it True: Evaluating Medical Reviews
“The review article itself should be the
product of scientific investigation in which
the participants are original investigations
(research) rather than patients”
Usefulness
Work: Low, good source for POEMs Relevance: If title and abstract or article
conclusion hold promise of POEM, continue Validity: Uncertain
Reviews- Three Basic Types
Textbooks
Academic Reviews
“Translation” Journals
Textbooks
Collection of review articles Minimal, if any, supporting evidence Questionable validity, long lag time to publish
• Average 1-2 years
Most useful for retracing, less hunting and foraging
Academic Reviews
Summary: “Broadly paint landscape” Validity uncertain- begin with conclusions and
find supporting references References often inaccurate and out of date **Expertise of author varies inversely with
quality of review- Oxman/Guyatt**• More later
Must confirm POEMs with original research, increasing work
Academic Reviews
Synthesis: Systematic reviews• Meta-analysis or overviews
Answer one or two specific questions Review primary literature with strict criteria Conclusions supported by available evidence Meta-analysis: Achieve power not possible by
single study
Academic Reviews
Excellent source for hunting and foraging
The Cochrane Library - “Database of Systematic Reviews”
Clinical Evidence (BMJ-BMA)
Clinical Inquiries (FPIN)- SORT
AFP EB Reviews- SORT
Dynamed - SORT
Essential Evidence Plus- SORT
Translation Journals
Quick reads for retracing and sporting Low work, but with low validity, may be zero
usefulness Hunting/foraging: Entering jungle on
starless night
Translation Journals
Common POEMs need original data for
verification, greatly increasing work
Watch for “weasel words”, based on DOEs
and anecdotes
• “it seems”, “may be effective”, “so one may assume”, “it appears”, “in my experience”
Weasel Words
Translation Journals
“Buyer Beware: Unsystematic reviews lead to unsystematic conclusions. Readers looking for a shortcut to understanding evidence about health problems and patient care should at least look for reviews by those who have not taken shortcuts”
Determining the usefulness of reviews
Onto the worksheets!
Reviews: Determining Relevance
A. Addressing specific clinical question?
A. Patient-oriented evidence?
B. Common problem?
C. Change your practice?
Reviews: Determining Validity
Answer ALL worksheet questions Stop = “fatal flaw” Notice how hard this is! Average time for a
good systematic review- 2 years!• Much different from your “usual review/CME talk”
Worksheet Qs: Finding the studies
Clearly stated?• Terms appropriate? MESH-linked? None missing?
Comprehensive?• Medline + another
• MEDLINE misses >50% of articles• Cochrane registry is especially good source • Science Citation Index
• Bibliographic review • Unpublished literature
• conference abstracts, personal correspondence with important investigators or pharmaceutical companies
Done by more than one person and compared
Worksheet Qs: Selecting the studies
Inclusion Criteria• Established a priori
• Minimum factors: Population/problem; intervention/comparison; outcomes; study design
• Prefer no language restriction
• Sometimes validity criteria incorporated (random, blinded, appropriate follow-up, gold standard, etc.)
Best if done independently by 2 investigators• Possibly blinded to author/journal/study results
Worksheet Qs:Validity of included studies
Appropriate criteria?• Assurance that criteria specific to type of article employed
(therapy, diagnosis, prognosis, etc)• If therapy: randomization, blinding, concealed allocation,
follow-up Process independent by > 2 authors?
• Surprising differences!• Why blinding may be important:
• 2 sample articles, same study methods• One finds benefit, other does not• “serious flaws” in article without benefit
Worksheet Qs: Validity
Were the included studies valid?• Garbage in = garbage out
• If yes, no problem
• If no, how did authors handle this?• Exclusion/inclusion criteria for quality of study
• Subanalysis with comparison of results
• Need to consider how these flaws affect results/conclusions
Worksheet Qs: Analyzing the data
Homogeneity vs Heterogeneity: just finding the words and an explanation most important
If NOT homogeneous?• Need qualitative explanation. Is it due to chance vs study
design, population, exposure, or outcome?
Worksheet Qs: Analyzing the data
Appropriateness of combining data:• “Vote” count not usually appropriate
• Important to include ‘magnitude’ of the overall effect
• Cannot be done without some common ground- outcome
Publication bias• Small, negative trials less likely to be published
• Examined by funnel plot
• Number needed to change results
From: Cooper & Hedges: The handbook of research synthesis. 1994
Funnel plot examples
From: Cooper & Hedges: The handbook of research synthesis. 1994
Reviews: Major Points Validity traps to avoid
• Assertions based on DOEs -- avoid perpetuating medical gossip
• Unassessed validity -- Personal experience unreliable as a basis for therapeutic interventions
Missing pieces -- **Quality of the review varies inversely with the expertness of the writer**
Failure to identify level of evidence – Look for LOE’s/ SORT
Reviews- Three Basic Types
Highly Controlled ResearchRandomized Controlled TrialsSystematic Reviews
Physiologic ResearchPreliminary Clinical ResearchCase reportsObservational studies
Uncontrolled Observations&
Conjecture
Effect on Patient-Oriented OutcomesSymptomsFunctioningQuality of LifeLifespan
Effect on Disease MarkersDiabetes (microalbuminuria, GFR, photocoagulation rates)Arthritis (ESR, x-rays)Peptic Ulcer (endoscopic ulcers)
Effect on Risk Factors for DiseaseImprovement in markers (blood pressure, HbA1C, cholesterol)
SORTA
Validity of Evidence
Re
leva
nce
of O
utc
om
eR
ele
vanc
e o
f Ou
tco
me
SORTB
SORTC