Date post: | 17-Jan-2018 |
Category: |
Documents |
Upload: | eric-osborne-richards |
View: | 219 times |
Download: | 0 times |
RTI International is a trade name of Research Triangle Institute
www.rti.org
Nancy Berkman, PhD Meera Viswanathan, PhDEmail: [email protected] Email: [email protected]
Development of a Tool to Evaluate the Quality of Non-randomized Studies of
Interventions or ExposuresPresented by
Nancy D Berkman, PhD & Meera Viswanathan, PhDPresented at
AHRQ 2009 Annual ConferenceBethesda, MarylandSeptember 15, 2009
www.rti.org
Acknowledgements
• Project funding provided by
– Phase 1:• Grant from RTI Independent Research and Development
(IR&D) funds
– Phase 2:• Contract from Agency for Healthcare Research and
Quality (AHRQ), U.S. Department of Health and Human Services, through the Evidence-based Practice Centers program (EPC)
www.rti.org
Context for the Project
• Increasing demand to include non-randomized studies in systematic literature reviews and comparative effectiveness reviews to capture– The effects of interventions or exposures on a more broadly defined
population than can be observed through RCTs– Topics where RCTs would be logistically or ethically inappropriate– Longer term outcomes and harms (side effects)
• The trade-off for wider applicability of findings among observational studies, compared with RCTs, is a potentially wider range of sources of bias, including in selection, performance, detection of effects, and attrition.
www.rti.org
Background: Rating the Quality of Non-randomized Studies
The quality (internal validity) of each study included in a review needs to be evaluated:
• Well-established criteria and instruments exist for evaluating the quality of RCTs, but not non-randomized (observational) studies
• PIs conducting systematic reviews generally lack access to validated and adaptable instruments for evaluating the quality of observational studies.
• Each new review often develops its own quality rating tool, “reinvents the wheel”, leading to inconsistent standards within and across reviews.
www.rti.org
Project Goals
To create a practical and validated tool for evaluating the quality of non-randomized studies of interventions or exposures that is:
• Reflects a comprehensive theoretical framework: captures all relevant domains
• Broad applicability: can be used "off the shelf" by different PIs • Modifiable: can be adapted to different topic areas • Easy to use and understand: can be used by reviewers with
varying levels of expertise or experience • Validated: users can be confident of their evaluation of study
quality • Advances the methodology in the field• Disseminated widely
www.rti.org
Methods: Phase 1
Item development
• Reviewed the literature on the evaluation of the quality of observational studies
• Collected quality review items used in early tools to evaluate non-RCTs through – Published literature– 90 AHRQ-sponsored EPC reviews
• Categorized all potential items into the 12 quality domains identified in Evaluating non-randomized intervention studies (Deeks et al., 2003)
www.rti.org
Methods: Phase 1 (continued)
Item Bank development
• Selected the best items for measuring each of the included domains
• Modified selected items where necessary to ensure that critical domains were included and to improve readability
• Developed a pre-specified set of responses
• Developed explanatory text to be used by PIs and abstractors to individualize as well as standardize interpretation
www.rti.org
Methods: Phase 2
Technical Expert Panel input– Conceptual framework to ensure that we included all
relevant domains– Face validity
• Cognitive interviews with potential users– Readability– Conceptualization
• Validation– Content/face validity– Inter-rater reliability testing
www.rti.org
Conceptual Underpinnings of the Instrument
Evaluation of quality can rely on either a description of methods or an assessment of validity and precision
• Methods description approach– Follows the reporting structure of many manuscripts– Relies less on judgment than on reporting
• Validity and precision approach– What we really care about– More challenging to evaluate– Greater reliance on judgment
www.rti.org
Domains for quality evaluation approaches
Methods description approach• Background/context• Sample definition and selection• Intervention/exposure• Creation of treatment groups• Follow-up• Specification of outcomes• Analysis: comparability of
groups• Analysis: outcomes• Interpretation
Validity and precision approach • Selection bias• Performance bias• Information bias• Detection bias• Attrition bias• Reporting bias• Precision
www.rti.org
Tool Results
• Comprehensive: bank of 39 questions• Modifiable: includes relevant items appropriate for all
non-randomized study types• Easy to use: instructions for PIs and abstractors to assist
in appropriate interpretation of questions. Example:
What is the level of detail in describing the intervention or exposure? [PI: specify which details need to be stated, e.g., intensity, duration, frequency, route, setting, and timing of intervention/exposure. For case-control studies, consider if the condition, timing, frequency, and setting of symptoms is provided in the case definition]
www.rti.org
Next Steps
• Finalize inter-rater reliability results• Publish findings and disseminate the tool
• Proposed Phase III:– Design specific validation including inter-rater reliability
testing by study type– Reduce the number of questions needed to address
specific domains– Develop a web-based platform for generating design and
topic-specific instruments from the item bank.