Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Chapter 5: Critically Appraising Quantitative Evidence for Clinical Decision Making
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Critically Appraising Quantitative Studies
The process focuses on three questions
1. Are the results of the study valid? (Validity)
2. What are the results? (Reliability)
3. Will the results help me in caring for my patients? (Applicability)
Interpretation of results requires consideration of the clinical significance of the study findings and the statistical significance of the results
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
The Question of Validity
Bias - anything that distorts study findings in a systematic way and arises from the study methodology
Selection bias
Knowledge of who is or is not receiving an intervention
Measurement bias
Recall bias
Contamination
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Question
Tell whether the following statement is true or false.
The best way to prevent selection bias is to randomly assign study participants to groups.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Answer
True
Rationale: Selection bias in quantitative studies is best controlled by assigning participants to groups on a random basis. Other systematic and deliberate methods of assignment normally increase the chance of selection bias.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
The Question of Validity (cont’d)
Confounded Study Results
A study’s results may be confounded when a relationship between two variables is actually due to a third, either known or unknown variable
Often encountered in studies about lifestyle and health
Frequently a result of participant history
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Reliability Evaluate whether the sum of all n values equals the
original N
Magnitude of the effect
How strong is the difference between groups?
Tables
Statistical tests
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Reliability (cont’d)
Strength of association
Absolute risk reduction (ARR)
Absolute risk increase (ARI)
Relative risk (RR)
Relative risk reduction (RRR)
Odds ratio (OR)
Number needed to treat/harm (NNT/NNH)
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Question
The findings of a quantitative study testing a high school-based sexual health program reveal that for every 140 female students who take the program, one pregnancy is prevented. This conclusion indicates the:
a. OR
b. NNT
c. NNH
d. ARR
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Answer
b. NNT
Rationale: The number needed to treat (NNT) represents the number of people who would need to receive the therapy or intervention (the educational program) to prevent one bad outcome (teenage pregnancy).
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Reliability (cont’d)
Random error
Variations that occur purely by chance
The extent to which random error may influence a measurement can be reported using statistical significance (or p values) or by confidence intervals.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Reliability (cont’d)
Statistical significance
The aim of statistical analysis is to determine whether an observed effect arises from the study intervention or has occurred by chance
Study hypothesis and null hypothesis
The smaller the p value, the less likely the null hypothesis is true
Confidence interval - describes the range in which the true effect lies with a given degree of certainty
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Questions to Ask in a Critical Appraisal
Why was the study done?
What is the sample size?
Are measurements reliable and valid?
How were the data analyzed?
Did any untoward event happen during the study?
How do the findings fit with previous research?
What does this mean for practice?
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising Case Studies
Historically ranked lower in the hierarchy of evidence
Must be used with caution to inform practice, and any application requires careful evaluation of the outcomes
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising Case Control Studies
These investigate why certain people develop a specific illness, have an adverse event with a particular treatment, or behave in a particular way
Questions to ask
How were the cases obtained?
Were appropriate controls selected?
Were data collection methods the same for the cases and controls?
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising Cohort Studies
These are used for investigating the course of a disease or the unintended consequences of a treatment
Questions to ask
Was there a representative and well-defined sample of individuals at a similar point in the course of the disease?
Was follow up sufficiently long and complete?
Were objective and unbiased outcome criteria used?
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising Cohort Studies (cont’d)
Did the analysis adjust for important prognostic risk factors and confounding variables?
What is the magnitude of the relationship between predictors (i.e., prognostic indicators) and targeted outcome?
How likely is the outcome event(s) in a specified period of time?
How precise are the study estimates?
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Question
A team of researchers have received a grant to investigate the potential links between diet and the development of stomach cancer. What methodology is most likely to answer the researchers’ clinical question?
a. Case control
b. Case study
c. Randomized controlled trial (RCT)
d. Qualitative study
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Answer
a. Case control
Rationale: A case control study often selects individuals with a particular disease (e.g., stomach cancer) and looks back to identify factors that may underlie that disease (e.g., diet). Neither a case study nor a qualitative study would inform this relationship and an RCT would be unethical and impractical.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising Randomized Controlled Trials (RCTs) These can accurately establish cause and effect and can
inform the efficacy and effectiveness of interventions
Questions related to validity of RCTs
Were the subjects randomly assigned to the experimental and control groups?
Was random assignment concealed from the individuals who were first enrolling subjects into the study?
Were the subjects and providers kept blind to study group?
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising the Validity of RCTs
Were reasons given to explain why subjects did not complete the study?
Were the follow-up assessments long enough to fully study the effects of the intervention?
Were the subjects analyzed in the group to which they were randomly assigned?
Was the control group appropriate?
Were instruments valid and reliable?
Were the subjects in each of the groups similar on demographic and baseline clinical variables?
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising Systematic Reviews
Systematic reviews are a compilation of similar studies that address a specific clinical question
Not the same as a literature review or narrative review
The process the was used to conduce the review should be explicit
Look for a detailed description of the databases accessed, the search strategies, and the search terms
Clinicians should be able to clearly see which studies were included and which were excluded and how selected studies were assessed
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Appraising Systematic Reviews (cont’d) The report should inform clinicians about how data were
extracted from the individual studies
It should provide an overview of the evaluation of the included studies
Meta-analysis
The statistical approach to synthesizing the results of two or more studies
A relatively new methodology that has become a hallmark of EBP
As with all methodologies, applicability must be considered
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Evaluation and Synthesis of Quantitative Studies Once studies have been selected, they should be melded
together into a synthesis upon which to base practice and standards
Individual studies should be evaluated using an evaluation table - see Table 5-8
Synthesis is not reporting the findings of study after study; it is combining, contrasting, and interpreting a body of evidence to reach a conclusion
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Question
Tell whether the following statement is true or false.
Meta-analysis results in evidence that is applicable to a larger patient population than individual studies.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Answer
False
Rationale: In meta-analyses, combining the results of several studies produces a larger sample size and thus greater power to accurately determine the magnitude of the effect. This does not, however, increase the size of the relevant patient population.