EVALUATING MEDICAL L ITERATURE

Emily M. Ambizas, Pharm.D., MPH, CGPAssociate Clinical Professorarweilee@stjohns.edu

O B J E C T I V E S

Identify the components of an article to gauge its usefulness in clinical practice.

Discuss recommendations to screening articles to learn whether to use a new diagnostic test, gain information about the clinical course and prognosis of disease, the etiology or causation of disease, whether treatment’s effectiveness has been demonstrated in randomized trials and efficacious preventative measures for disease sequelae.

Critique adequacy of sample size and ability to generalize results, whether the presence and magnitude of bias was identified, whether confounding variables were limited in the study and whether limitations were identified and analyzed.

Distinguish if clinical and statistical significance were considered in an article.

Critically review the scientific literature, integrate data across studies and make appropriate community health recommendations based on medical knowledge.

Interpret results of an epidemiological study, including the relation to findings from other studies, the potential biases, limitations and community health implications

Extract the clinical message and apply it to the patient problem

Select original research journal articles from the medical literature and evaluate the study design, results, validity, peer review and relevance to clinical practice.

O B J E C T I V E S ( c o n t . )

M E D I C A L L I T E R AT U R EFewer than 15% of all articles published on a particular topic are useful

Not peer-reviewed

Sponsored by those with commercial interests

Throwaways

Average quality score of clinical trials is less

Widespread use of tests with uncertain efficacy

Treatments that are either ineffective or that may do more harm than good

So you need basic skills in judging validity and clinical importance of articles!

A R T I C L E T Y P E S

Report original research

OR summarize/draw conclusions from original research

Prim Care Clin Office Pract. 2006;33:839-62.

The anatomy of a scientific article

• Abstract

• Introduction

• Materials & Methods

• Results

• Discussion

• Conclusion

T H R E E S T E P P R O C E S S

Conduct initial validity and relevance screen

Determine the intent of the article

Evaluate the validity of the article based on its intent

12

3

Is the article from a peer-reviewed journal? Yes (go on) No (stop)

Is the location of the study similar to mine so the results, if valid, would apply to my practice?

Yes (go on) No (stop)

Is the study sponsored by an organization that may influence the study design or results?

Yes (go on) No (stop)

Will this information if true have a direct impact on the health of my patients, and is it something they will care about?

Yes (go on) No (stop)

Is the problem addressed one that is common to my practice, and is the intervention or test feasible and available to me?

Yes (go on) No (stop)

Will this information require me to change my current practice?

Yes (go on) No (stop)

STEP ONEConduct initial validity and relevance screen

• External validity• Inclusion criteria– Describe the population the patients represent– Determine whether the study sample resembles practice• Exclusion criteria– Help ensure study sample is homogenous– Identify patient subsets to which study should not be applied – Assure patient safety

Are the subjects in the study similar to mine ?

STEP ONEIs the article from a peer-reviewed journal? Yes (go on) No (stop)

“Internal validity” If in doubt look in the journal’s “Instructions for Authors” Articles have already undergone extensive process Has become accepted method for improving quality of the science

reported Does not guarantee article is without flaw or bias Positive results more likely to be published Fraudulent research is hard to identify

Conduct initial validity and relevance screen

Is the location of the study similar to mine so the results, if valid, would apply to my practice?

Yes (go on) No (stop)

o Where was the study conducted?o Usually can be determined by reviewing the author’s information on the first page

of an article

STEP ONEConduct initial validity and relevance screen

Is the study sponsored by an organization that may influence the study design or results?

Yes (go on) No (stop)

o Considers the potential bias that may occur from outside fundingo Investigators are required to identify sources of fundingo Be wary of published symposiums

o Promotional in natureo Misleading titleso Use brand nameso Less likely to be peer-reviewed in the same manner

Will this information if true have a direct impact on the health of my patients, and is it something they will care about?

Yes (go on) No (stop)

Is the problem addressed on that is common to my practice, and is the intervention or test feasible and available to me?

Yes (go on) No (stop)

Will this information require me to change my current practice?

Yes (go on) No (stop)

STEP ONEConduct initial validity and relevance screen

S T E P T W O :determine intentWhat is the clinical question?

Identification of the study objective

Who sponsored the study?Any override by sponsor based criteria? Impact Factor

Common flaw: Failure to present a clearly defined objective

"The following experiment was done to ascertain whether corticosteroids improve results of

spirometry or arterial blood gas levels, or both, in patients with chronic obstructive pulmonary

disease and acute respiratory failure.”

Vs. "We studied the use of corticosteroids in patients

with lung disease."

A S S E S S S T U D Y D E S I G N

• Read the abstract

• Survey boldface headings

• Review tables, graphs, illustrations

• Skim-read the first sentence of each paragraph

• Most important section

• Should be presented clearly and in detail so that the experiment could be reproduced

• Includes:

– Information concerning experimental design

– Study sample

– Treatment allocation

– Index of accomplishment

– Statistical test section

STEP THREE Evaluate the validity of the

article based on its intent

Major clinical categories of primary research and preferred study designs

The following questions will help you with step 3

RR

OR

Case control and cohort studies establish correlation NOT causation

Wisconsin study showing more heart attacks on during football season. However, this was due to more Sunday football parties.

Therefore correlation does not mean causation

Only way to establish causation is with an experimental study, not an observational study

Correlation does not equal causation.

• Randomized controlled trials are considered the gold standard

• Diminishes potential for investigators selecting individuals in a way that would unfairly bias one treatment over another – selection bias

• Randomization– Report group differences– Known risk factors– Demographics– Disease severity

• Do these differences influence outcomes?

Is the study a randomized controlled trial (RCT) ?

• Another strength of an RCT is that participants are followed prospectively

– Need to avoid a “loss-of-subject bias”

• Patient withdrawals– Poor patient compliance– Data that are uninterpretable– Eventual identification of factors that describe ineligible subjects

• Attrition rate greater than 10% for short-term trials and 15% for long-term trials may invalidate results

• Intention-to-treat analysis

?Are all participants who entered the trial properly accounted for at its conclusion?

• Controlled trials should be blinded to prevent potential biases of investigator and patients

• Single blind trial vs. double blind trials• Measurement of response to treatment is subjective• When knowledge of the assigned treatment risks the

introduction of bias• Should always verify that blinding maneuvers have been

accomplished successfully

Was everyone involved in the

study “blind” to treatment ?

• Groups should be similar at the beginning of a trial

• Comparisons are usually made for demographic factors, other known risk factors, and disease severity

Were the intervention & control groups similar at the start of the trial ?

• Compare interventions (i.e. follow ups)

• All groups should be treated the same except for the therapeutic intervention

• Everyone should be seen with the same frequency, and interventions should be similar

Were the groups treated equally ?

Outcomes • Superiority vs noninferiority

• Primary endpoints: sample size to meet power depends on rate of event occurring

• Some studies use composite end points i.e. reduces risk of stroke, HF, MI and death… make sure these are reported individually as weight is an issue.

• Surrogate endpoints: BP, WBC, etc. do not predict clinical outcomes. Cholesterol studies

• Secondary endpoints: not used to assess power, can inspire further studies. (i.e. survival at 60 days post intervention)

”Internal validity”

Statistical test depends on:

1. Type of data analyzed2. Whether groups are independent or paired

Are the results clinically as well

as statistically significant ?“ C l i n i c a l

S i g n i f i c a n c e ”The clinical importance of data in a study, irrespective of statistically generated results.

“ S t a t i s t i c a l S i g n i f i c a n c e ”

The importance of a study in terms of the outcome of statistical tests on the data. A study is said to be statistically significant when statistical tests demonstrate a difference between treatment groups.

NOTE: a study can have results that are not statistically significant but these results can still be clinically significant & vice versa

T Y P E S O F D A T AAre the results clinically as well as statistically significant?

Categorical/ Nominal

Ordinal Continuous/Interval

Categorical data with no order to categories

Simplest level of measurement (weakest/lowest)

Categorical data that has an implied order

difference between 2 adjacent categories are not equal or measurable

Strongest/ highest level of measurement

Differences between numbers are equal and measurable

I N D E P E N D E N T & P A I R E D S A M P L E S

Are the results clinically as well as statistically significant?

Use of paired or related samples is superior

Matched pairs

Cross-over study Patients are matched with respect to key variables

o Difference found when there isn’t one, usually due to chance

o To reject the null hypothesis, when in fact the null hypothesis is true; to falsely conclude that a significant difference exists between groups

o alpha () - the probability of making a type I error

o Usually set at 0.05 (<5% chance of making a type 1 error)

o P> 0.05 – not statistically significant

o P<0.05 – statistically significant

P values do not assist in determining the size of the difference or clinical significance!

T Y P E S O F E R R O R : T Y P E 1 ( )

Are the results clinically as well as statistically significant?

o Fails to find difference when there is

o To accept the null hypothesis, when in fact the null hypothesis is false; to falsely conclude that no significant difference exists between groups

o beta () - the probability of making a type II error

o beta usually set at 0.20 or 0.10 (20% or 10% chance of making a type II error)

If a negative trial, was a power analysis done?

T Y P E S O F E R R O R : T Y P E 2 ( )

o Power is calculated BEFORE the study starts and is dependent on sample size

o Sample size is the number of subjects in a study

o Sample size (N) must be adequate to determine whether a statistically significant difference exists between groups

An inadequate N (too small) leads to reduced power and therefore, less ability to detect a statistically significant difference if one truly exists (inconclusive results)

An excessively large N can make a small clinically insignificant difference statistically significant

If a negative trial, was a power analysis done?

S A M P L E S I Z E A N D P O W E R

R R R ,A R R ,N N T

&N N H

• Relative Risk Reduction (RRR): measures how much the risk is reduced in the experimental group compared to a control group. For example, if 60% of the control group died and 30% of the treated group died, the treatment would have a relative risk reduction of 0.5 or 50% (the rate of death in the treated group is half of that in the control group).

• Absolute risk reduction (ARR): Absolute risk reduction is just the absolute difference in outcome rates between the control and treatment groups: CER - EER. The absolute risk reduction does not involve an explicit comparison to the control group as in the relative risk reduction and thus, does not confound the effect size with the baseline risk.

• The number needed to treat (NNT) is basically another way to express the absolute risk reduction. It is just 1/ARR and can be thought of as the number of patients that would need to be treated to prevent one additional bad outcome. We want this number to be small so we know the intervention works

• The number needed to harm is the number of patients that would need to be treated to cause a bad outcome or side effect. We want this number to be large so we know the intervention does not cause bad effects often.

CONFIDENCE INTERVALS

(CI)

• Confidence intervals (CIs) aim to give you an idea of how confident you can be about a study’s estimate of a treatment’s effects.

• Even when a study is of impeccable quality, the results may have happened by chance.

• The narrower the range, the more precise the study’s estimates, and the more confident you can be that it is a ‘real’ finding and not due to chance.

• Represented as 95%

A practical guide to commonly used statistical tests

PARAMETRIC VS .

NON-PARAMETRIC TESTS