Date post: | 14-Feb-2016 |
Category: |
Documents |
Upload: | victor-morando-nainggolan |
View: | 224 times |
Download: | 6 times |
CRITICAL APPRAISAL OF THE MEDICAL LITERATURE
Partini Pudjiastuti TChild Health Department
Faculty of Medicine University of Indonesia
Critical appraisal (Making Reading More Worthwhile)
What is Critical Appraisal?1. Critical appraisal = quality assessment2. ….process of weighing up evidence to see
how useful it is in decision making3. .…a process of assessing the validity,
reliability and usefulness of evidence4. Critical appraisal is about considering,
evaluating and interpreting information in a systematic and objective way
Critically Appraise What You Read.
• Separating the wheat from the chaff.• Time is limited – you should aim to quickly
stop reading the dross.• Others contain useful information mixed
with rubbish.• Simple checklists enable the useful
information to be identified.
Critical Appraisal – Critical Thinking
• Appraising (evaluating/reviewing) the available evidence to construct clinical reasoning strategies and to make decisions
• Finding strengths and limitations of written ‘evidence’
• You need to decide what evidence to pay attention to (what is “worthy” of your attention) versus what to ignore
Why critically appraise?
• Supports sound decision making based on best available evidence
• Helps us determine (three R’s):• How rigorous a piece of research is• What the results are telling us• How relevant it is to our patient
What is “Evidence”?
• People disagree on what constitutes “evidence”
• Evidence - what is generally regarded as a scientific fact
• Evidence - a combination of information obtained from 3 sources: research, clinical experience, and client preferences (Kitson, Harvey, & McCormack, 1998)
Why do we need evidence?
– Resources should be allocated to things that are EFFECTIVE
– The only way of judging effectiveness is EVIDENCE
– “In God we trust – all others bring data”
Why do we need evidence?
• Move towards: EVIDENCE-BASED MEDICINE
• Move away from: EMINENCE- BASED MEDICINE
What we really, really want is
EVIDENCE-INFORMED MEDICINE
Sources of Evidence
• Primary sources– Based on experiments and published
research• Secondary sources
– Systematic reviews– Clinical guidelines– Journals of secondary publication e.g.
Evidence Based Medicine
“5S” Pyramid of Evidence Resources
Levels of evidence 1. Systematic reviews of RCTs and high quality
RCTs2. Systematic reviews of Cohort studies, lower
quality RCTs, Outcomes research3. Systematic reviews of case controls, case
control studies4. Case Series5. Expert opinion
See http://www.cebm.net/levels_of_evidence.asp for full descriptions
Types of EvidenceQuestion Types
Type of Question Best EvidenceHealth care interventions: treatment, prevention
Quantitative: Systematic Review of RCTs or RCT
Harm or Etiology Quantitative: Observational Study - Cohort or Case Control
Prognosis Quantitative: Observational Study - Cohort, Case Control
Diagnosis or Assessment Quantitative: Comparison to Gold Standard
Economics Quantitative: Cost-effectiveness Study
Meaning Qualitative: case study, ethnography, grounded theory, phenomenologic approach
KEY QUALITY PARAMETERS
• VALIDITY
• RELIABILITY
• IMPORTANCE
VALIDITY• INTERNALIs the study designed in such a way that I can trust the findings?
• EXTERNALIs the study designed in such a way that Ican generalize the findings?
RELIABILITY
If the study was conducted again,would the results be the same?
Usually interpreted as the accuracyof measurement.
IMPORTANCE
What was the effect sizeor magnitude of effect?
Clinical vs. statistical significance.
Tools for Critical Appraisal
• What are the results?
• Are the results valid?
• Will the results help me in patient care?
EBM “simplified” approach:
I
V
A
Evidence based medicine5 steps
Formulate question
Efficiently track down bestavailableevidence
Critically review thevalidity and usefulnessof the evidence
Implement changes in clinical practice
Evaluate performance
CHECK LIST FOR MEDICAL LITERATURE (COMPLETENESS)
1. Title2. Authors3. Abstract: structured? Informative? Abbreviation?4. Introduction: length? Relevant references? Target
population?5. Methods:
– Design– Inclusion criteria – Exclusion criteria– Sample size, sampling method– Randomization technique– Intervention: masking?– Outcome measurement: blinding?
• Primary outcome: type of variable • Secondary outcome: type of variable
– Analysis
6. Results– Baseline characteristics– Main outcome– Secondary outcome
7. Discussion– General– Strength and weakness– Conclusions
8. References– Vancouver style– Constant
9. Acknowledgment10.Ethics approval11.Conflict interest
CHECK LIST FOR MEDICAL LITERATURE (COMPLETENESS)
What to assess?(in study of cause-effect relationship)
A. General description• Type of design• Target population, source population,
sample• Sampling method• Dependent and independent variables• Main results?
B. Internal validity, non-causal relationship– Influence of bias– Influence of chance– Influence of confounders
What to assess?(in study of cause-effect relationship)
BiasWhat is a bias? A process that tends to produce
results that depart systematically from the true values existing in the study population
Types of bias1.Sample (subject selection) biases, which may result
in the subjects in the sample being unrepresentative of the population which you are interested in
2.Measurement (detection) biases, which include issues related to how the outcome of interest was measured
3. Intervention (performance) biases, which involve how the treatment itself was carried out.
C. Internal validity, causal relationship• Temporality (cause precedes effect)• Strength of association (large difference, RR, OR, etc) or
small p value or narrow confidence interval• Biological gradient (dose dependence)• Consistency among studies (diff. populations or designs)• Specificity (certain factor results in certain effect)• Coherence (does not conflict with current knowledge)• Biological plausibility: can be explained with current
knowledge (at least in part)
What to assess?(in study of cause-effect relationship)
D. External validity– Applicable to study subjects– Applicable to source population– Applicable to target population
What to assess?(in study of cause-effect relationship)
11 items, each with 3 sections
1. Can you find this information in the paper?
2. Is the way this was done a problem?3. Does this problem threaten the validity of
the study?
11 items1. What is the research question?2. What is the study type?3. What are the outcome factors and how are they measured?4. What are the study factors and how are they measured?5. What important confounders are considered?6. What are the sampling frame and sampling method?7. In an experimental study, how were the subjects assigned to
groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (Etc)
8. Are statistical tests considered?9. Are the results clinically/socially significant?10. Is the study ethical? 11. What conclusions did the authors reach about the study question?
1.What is the research question?
• (Is the way this was done a problem?)– Is it concerned with the impact of an
intervention, causality or determining the magnitude of a health problem?
• (Does this problem threaten the validity of the study?)– Is it a well stated research
question/hypothesis?
2. What is the study type?• (Is the way this was done a problem?)
– Is the study type appropriate to the research question?
• (Does this problem threaten the validity of the study?)– If not, how useful are the results produced by
this type of study?
3. What are the outcome factors and how are they measured?
• (Is the way this was done a problem?)– a) are all relevant outcomes assessed– b) is there measurement error?
• (Does this problem threaten the validity of the study?)– a) how important are omitted outcomes– b) is measurement error an important source
of bias?
4. What are the study factors and how are the measured?
• (Is the way this was done a problem?)– Is there measurement error?
• (Does this problem threaten the validity of the study?)– Is measurement error an important source of
bias?
5. What important potential confounders are considered?
• (Is the way this was done a problem?)– Are potential confounders examined and
controlled for?
• (Does this problem threaten the validity of the study?)– Is confounding an important source of bias?
6. What are the sampling frame and sampling method?
• (Is the way this was done a problem?)– Is there selection bias?
• (Does this problem threaten the validity of the study?)– Does this threaten the external validity of the
study?
7. Questions of internal validity• (Is the way this was done a problem?)
– In an experimental study, how were the subjects assigned to groups?
– In a longitudinal study, how many reached follow-up?– In a case control study, are the controls appropriate?
• Note: other issues of relevance to internal validity are considered under the other headings in this critical appraisal system. You can add your own questions, and also design your own questions for other study types such as cross sectional studies and systematic reviews
• (Does this problem threaten the validity of the study?)– Does this threaten the internal validity of the study?
8. Are statistical tests considered?
• (Is the way this was done a problem?)– Were the tests appropriate for the data?– Are confidence intervals given?– Is the power given if a null result?– In a trial, are results presented as absolute
risk reduction as well as relative risk reduction?
• (Does this problem threaten the validity of the study?)– If not, how useful are the results?
9. Are the results clinically/socially significant?
• (Is the way this was done a problem?)– Was the sample size adequate to detect a
clinically/socially significant result?– Are the results presented in a way to help in
health policy decisions?
• (Does this problem threaten the validity of the study?)– Is the study useful?
10. Are ethical issues considered?
• (Is the way this was done a problem?)– Does the paper indicate ethics approval?– Can you identify potential ethical issues?
• (Does this problem threaten the validity of the study?)– Are the results or their application
compromised?
11. What conclusions did the authors reach about the study
question?• (Is the way this was done a problem?)
– Do the results apply to the population in which you are interested?
• (Does this problem threaten the validity of the study?)– Will you use the results of the study?
Appraisal Tools• Tools from the Critical Appraisal Skills
Programme (CASP)– Systematic Reviews– Randomised Controlled Trials– Qualitative Research Studies– Cohort Studies– Case-Control Studies– Diagnostic Test Studies– Economic Evaluation Studies
Available at: http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm
Study Designs Recap
Effectiveness of Therapy
Risk Factors / Prognosis
Diagnosis
Attitudes & Beliefs
Randomised Controlled Trial
Cohort Study
Survey using Gold Standard
Qualitative (Interviews, Observations, etc)
CRITICAL APPRAISAL
- VALID- IMPORTANT- APPLICABLE
METHODS RESULTS DISCUSSION
THANKS
From Data to Wisdom
• Data are what researchers collect• Information results when data is analyzed
and interpreted (EVIDENCE)• Knowledge results when information is
shared, acquired, and used• Wisdom is the ability to make the right
use of knowledge
Types of evaluations
Efficacy– treatment does more good than harm when
offered to those who adhere to treatment recommendations
– Does it work under ideal conditions?
Types of evaluations
Effectiveness– treatment does more good than harm in those
to whom it is offered, under ordinary (clinical) circumstances
– Can it work in the real world?
Critical review protocol - quantitative research
• Study purpose• Application to occupational therapy• Study design• Bias• Sampling issues/ sample size/ drop outs• Outcome measurement (reliability, validity)• Intervention description & implementation• Results - statistical & clinical significance• Implications