Tools for Evaluating Strength and Quality of...

Tools for Evaluating Strength and Quality of Evidence

The National Academies of Sciences Engineering Medicine

Global Harmonization of Methodological Approaches to Nutrient Intake Recommendations

Rome, Italy

September 21-22, 2017

George Wells

EVALUATING STRENGTH AND QUALITY AT THE INDIVIDUAL STUDY LEVEL

Levels of Evidence

RCT

Cross-sectional Study

Case-Control Study

Cohort Study

Quasi-Experimental

Conveys concept of moving from high to low quality generating studies Within each level the quality of the studies can vary depending on its specific design features and conduct

Experimental Studies Randomized Controlled Trial (RCT) Quasi-Randomized Controlled Trial (Q-RCT) Non-Randomized Controlled Trial (NRCT)

Quasi-Experimental Studies Controlled Before-After (CBA) Controlled Interrupted Time Series (CITS)

Observational Studies Cohort study (PCS, RCS, HCS) Case-Control study (CC, NCC) Cross-sectional study (XS)

Reeves BC, Wells GA, Waddington H, Quasi-experimental study designs series - paper 5: a checklist for classifying studies evaluating the effects on health interventions - a taxonomy without labels. Journal of Clinical Epidemiology, 2017 (in press).

Some bias domains through which bias may be introduced …

Target Population

Study Population

Non-participantsParticipants

A

Intervention A

Outcome

Intervention B

Outcome

Bias Domains

Bias in selection of participants into the study

Bias due to confounding

Bias in measurement of interventions

Bias due to departures from intended interventions

Bias due to missing data

Bias in measurement of outcomes

Bias in selection of the reported result

A review was conducted from January 2005 to December 2007 that extended the AHRQ review conducted to December 2004; QAIs identified and the generic checklists passing selection criteria and evaluated using the AHRQ evaluation grids:

Quality Assessment Instruments (QAI)

Bai A, Shukla VK, Bak G, Wells G. Quality Assessment Tools Project Report. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2012.

QAI/EGS Identified 2nd Level evaluation

Selection

RCT 94 32 SIGN 50

OBS 99 23 SIGN 50

SR 57 20 AMSTAR

EGS 60 23 GRADE

Evaluation of QAIs for Randomized Controlled Trials

Evaluation of QAIs for Observational Studies

SIGN 50 – Methodological Checklist - Controlled Trials INTERNAL VALIDITY

1.1 The study addresses an appropriate and clearly focused question.

1.2 The assignment of participants to intervention groups is randomized.

1.3 An adequate concealment method is used.

1.4 Participants and investigators are kept ‘blind’ about intervention allocation.

1.5 The intervention and control groups are similar at the start of the trial.

1.6 The only difference between groups is the intervention under investigation.

1.7 All relevant outcomes are measured in a standard, valid and reliable way.

1.8 What percentage of the individuals or clusters recruited into each arm of the study dropped out before the study was completed?

1.9 All the participants are analysed in the groups to which they were randomly allocated (often referred to as intention to treat analysis).

1.10 Where the study is carried out at more than one site, results are comparable for all sites.

OVERALL ASSESSMENT OF THE STUDY

2.1 How well was the study done to minimise the risk of bias or confounding? ++ / + / 0 Considering the quality criteria assessed, rate the overall methodological quality of the study, using the following as a guide: High quality (++): Majority of criteria met. Little or no risk of bias. Results unlikely to be changed by further research. Acceptable (+): Most criteria met. Some flaws in the study with an associated risk of bias, Conclusions may change in the light of further studies. Low quality (0): Either most criteria not met, or significant flaws relating to key aspects of study design. Conclusions likely to change in the light of further studies.

SIGN 50- Controlled Trials

S I G N Methodology Checklist 2: Controlled Trials

Study identification (Include author, title, year of publication, journal title, pages)

Guideline topic: Key Question No: Reviewer:

Before completing this checklist, consider:

1. Is the paper a randomised controlled trial or a controlled clinical trial? If in doubt, check the study design algorithm available from SIGN and make sure you have the correct checklist. If it is a controlled clinical trial questions 1.2, 1.3, and 1.4 are not relevant, and the study cannot be rated higher than 1+

2. Is the paper relevant to key question? Analyse using PICO (Patient or Population Intervention Comparison Outcome). IF NO REJECT (give reason below). IF YES complete the checklist.

Reason for rejection: 1. Paper not relevant to key question 2. Other reason (please specify):

SECTION 1: INTERNAL VALIDITY

In a well conducted RCT study… Does this study do it?


Yes Can’t say

No

1.2 The assignment of subjects to treatment groups is randomised. Yes Can’t say

No


Yes Can’t say

No

1.4 The design keeps subjects and investigators ‘blind’ about treatment allocation.

Yes Can’t say

No

1.5 The treatment and control groups are similar at the start of the trial. Yes Can’t say □

No

1.6 The only difference between groups is the treatment under investigation.

Yes Can’t say

No


Yes Can’t say

No

1.8 What percentage of the individuals or clusters recruited into each treatment arm of the study dropped out before the study was completed?

1.9 All the subjects are analysed in the groups to which they were randomly allocated (often referred to as intention to treat analysis).

Yes Can’t say

No Does not apply


Yes Can’t say

No Does not apply

SECTION 2: OVERALL ASSESSMENT OF THE STUDY

2.1 How well was the study done to minimise bias? Code as follows:

High quality (++)

Acceptable (+)

Low quality (-)

Unacceptable – reject 0

2.2 Taking into account clinical considerations, your evaluation of the methodology used, and the statistical power of the study, are you certain that the overall effect is due to the study intervention?

2.3 Are the results of this study directly applicable to the patient group targeted by this guideline?

2.4 Notes. Summarise the authors’ conclusions. Add any comments on your own assessment of the study, and the extent to which it answers your question and mention any areas of uncertainty raised above.

SIGN 50 Controlled Trials - Guidance

SIGN 50 – Methodological Checklist - Cohort Studies 1.1 The study addresses an appropriate and clearly focused question.

SELECTION OF SUBJECTS

1.2 The two groups being studied are selected from source populations that are comparable in all respects other than the factor under investigation.

1.3 The study indicates how many of the people asked to take part did so, in each of the groups being studied.

1.4 The likelihood that some eligible subjects might have the outcome at the time of enrolment is assessed and taken into account in the analysis.

1.5 What percentage of individuals or clusters recruited into each arm of the study dropped out before the study was completed.

1.6 Comparison is made between full participants and those lost to follow up, by exposure status.

ASSESSMENT

1.8 The outcomes are clearly defined.

1.9 The assessment of outcome is made blind to exposure status. If the study is retrospective this may not be applicable.

1.10 Where blinding was not possible, there is some recognition that knowledge of exposure status could have influenced the assessment of outcome.

1.10 The method of assessment of exposure is reliable.

1.11 Evidence from other sources is used to demonstrate that the method of outcome assessment is valid and reliable.

1.12 Exposure level or prognostic factor is assessed more than once.

CONFOUNDING

1.13 The main potential confounders are identified and taken into account in the design and analysis.

STATISTICAL ANALYSIS

1.14 Have confidence intervals been provided?



SIGN 50 – Cohort Studies

SIGN 50 – Methodological Checklist - Case-Control Studies 1.1 The study addresses an appropriate and clearly focused question.

SELECTION OF SUBJECTS

1.2 The cases and controls are taken from comparable populations.

1.3 The same exclusion criteria are used for both cases and controls.

1.4 What percentage of each group (cases and controls) participated in the study?

1.5 Comparison is made between participants and non-participants to establish their similarities or differences.

1.6 Cases are clearly defined and differentiated from controls.

1.7 It is clearly established that controls are non-cases.

ASSESSMENT

1.8 Measures will have been taken to prevent knowledge of primary exposure influencing case ascertainment.

1.9 Exposure status is measured in a standard, valid and reliable way.

CONFOUNDING

1.10 The main potential confounders are identified and taken into account in the design and analysis.

STATISTICAL ANALYSIS

1.11 Have confidence intervals been provided?



SIGN 50 – Case-Control Studies

Health Canada Quality Assessment Nutrition Project Currently, US and Canadian health agencies are evaluating the key scientific issues involved in using chronic disease endpoints for setting nutrient Dietary Reference Intakes (DRIs)

One problem identified is that existing QAI are not always sensitive enough to distinguish between nutrition studies that reduce potential biases and those that are not • Are there any nutrition-specific bias domains that should be considered? • Does such a tool for nutrition studies exist; if not, is such a tool worthwhile to develop?

Nutrition-specific QAIs could take into account specific types of covariates, confounders, and sources of error unique to nutrition

Such a tool could be used in systematic reviews undertaken to evaluate health claim submissions or to update dietary guidelines

Yetley, E., MacFarlane, A., Greene-Finestone, L., Garza, C., Ard, J., Atkinson, S., Bier, D., Carriguiry, A., Harian, W., Hattis, D., King, J., Krewski, D., O’Connor, D., Prentice, R., Rodricks, J. and Wells, G.A., “Options for Basing Dietary Reference Intakes (DRIs) on Chronic Disease Endpoints: Report from a Joint US-/Canadian-Sponsored Working Group”. The American Journal of Clinical Nutrition, (E-pub ahead of print – pii: alcn139097 – December 2016), Vol. 105, No. 1, 249S-285S, January 2017.

StepStepStep

General development principle

Identifying QAI for adaptation

Identifying nutrition-specific quality appraisal Items

Nutrition-specific items identified

Formatting the questions

Developing guidance

Reliability and validity testing

Developing a Nutrition-Specific QAI

StepStepStep


Literature was scanned for examples of topic-specific QAIs or ‘modules’ and the determination was made that ‘bolting’ on nutrition-specific quality appraisal items to an existing QAI was the most efficient approach

• Nutrition-specific quality appraisal items would be added after the general quality appraisal items of the QAI

• Psychometric properties (feasibility, reliability and validity) of the QAI could be taken and added specificity of nutrition-specific items assessed, as well as its psychometric properties





Goals of the ‘bolt-on’ • Maintain the original statement of the QAI questions • Can modify guidance for these questions if it clarifies the response

when considering a nutrition intervention

• After the QAI questions on internal validity, add the questions related to the nutrition-specific aspects

• Add the guidance for these nutrition-specific questions

StepStepStep



Literature was reviewed to identify general QAIs that could be adapted to include nutrition-specific quality appraisal items

• A formal review of QAIs was published in 2012, including instruments that were published to December 2007: RCTs - 94 identified and 32 evaluated; observational studies (OBS) - 99 identified and 23 evaluated

• Quality was judged based on the adequacy of addressing five domains identified by AHRQ; based on this evaluation and with input from experts, the SIGN 50 checklists were selected

• More recent publication of QAIs were reviewed and the decision remained to use the SIGN 50 checklists




Developing guidance



StepStepStep




A ‘scoping’ literature review was conducted to create a laundry list of existing nutrition-specific QAIs and/or quality appraisal items

• Preliminary nutrition-specific concepts were built on the Chung et al (2009) proposed 7 nutrition-specific items identified based on the concern that failure to consider these items could lead to biased syntheses or interpretation of results in nutrient-related SRs

At the 2014 DRI chronic disease workshop in Washington DC, 3 nutrition-specific items were proposed in the Workshop presentation: assessment of dietary intakes; compliance with dietary interventions; mapping dietary intakes into nutrient intakes

A panel was convened to refine the list of nutrition-specific items


• For intervention studies • sources of the nutrient interventions • doses of the nutrient interventions • baseline nutrient exposures in the study population

• For observational studies • methods/instruments for assessing intakes of nutrient exposures • ranges or distributions of the nutrient exposures • errors in assessing nutrient exposures • potential impacts of the errors from assessing the nutrient exposures on the nutrient-

outcome association

StepStepStep






Study design Nutrition-specific items

Controlled trial

1. The relevant intake/nutritional status at baseline is documented and taken into account.

2. The background intakes of all groups have been maintained over the course of the study.

3. The amount of intervention intake under study has been quantitatively measured and the form, where relevant, has been described.

4. Adherence to the intervention and control intakes is being monitored in a reliable and accurate manner.

5. The interval between the intervention and outcome is of sufficient duration to observe an effect, if there is one.

Cohort 1. The relevant intake/nutritional status at baseline is sufficiently documented.2. The amount of intake under study has been quantitatively measured and

the form, where relevant, has been described.3. The intake is being assessed/monitored in a reliable and accurate manner.4. The original exposure differences between the groups have been

maintained over the course of the study.5. The interval between the exposure and outcome is of sufficient duration to

observe an effect, if there is one.

Case-control 1. The amount of intake under study has been quantitatively measured and the form, where relevant, has been described.

2. The interval between the exposure and outcome is of sufficient duration to observe an effect, if there is one.

StepStepStep





Formatting the questions For the quality assessment of an RCT of nutrient-health outcome relationships, a QAI was developed keeping the general quality appraisal items exactly the same as the original SIGN 50 QAI and adding on the nutrition-specific quality appraisal items

• Relevance, uniqueness and feasibility of this nutrition-specific QAI for RCTs was considered in its development

• QAI was pilot tested for feasibility and revisions were made

Similar development of the QAI for cohort studies and case-control studies of nutrient-health outcomes conducted

Developing guidance



StepStepStep






Developing guidance Guidance for completing the questionnaire for each item was developed

• For the general quality appraisal items, some contextual aspects related to nutrition studies was added

• For the nutrition-specific quality appraisal items guidance was developed de novo

• Revisions were made after the pilot test

Similar development of the QAI for cohort studies and case-control studies of nutrient-health outcomes conducted



StepStepStep






Developing guidance


10 published studies of RCTs of nutrient-health outcomes were selected from the literature based on an AHRQ report

The QAI developed for RCTs of nutrient-health outcomes was applied to each study independently by 2 experienced reviewers:

• reliability of their assessments were evaluated • construct validity were determined • impact of additional information provided by the nutrition-specific items on the

quality assessment was identified

Similarly 10 published cohort studies and 10 published case-control studies were quality assessed using the QAI developed for cohort studies and case-controlled studies of nutrient-health outcomes respectively


StepStepStep






Developing guidance



Adapted for Nutrition Studies: Controlled Trials

S I G N

Methodology Checklist 2: Controlled Trials

Study identification (Include author, title, year of publication, journal title, pages)

Guideline topic: Key Question No: Reviewer:

Before completing this checklist, consider:

1. Is the paper a randomised controlled trial or a controlled clinical trial? If in doubt, check the study design algorithm available from SIGN and make sure you have the correct checklist. If it is a controlled clinical trial questions 1.2, 1.3, and 1.4 are not relevant, and the study cannot be rated higher than 1+

2. Is the paper relevant to key question? Analyse using PICO (Patient or Population Intervention Comparison Outcome). IF NO REJECT (give reason below). IF YES complete the checklist.

Reason for rejection: 1. Paper not relevant to key question 2. Other reason (please specify):

SECTION 1: INTERNAL VALIDITY

In a well conducted RCT study… Does this study do it?


Yes Can’t say

No

1.2 The assignment of subjects to treatment groups is randomised. Yes Can’t say

No


Yes Can’t say

No

1.4 The design keeps subjects and investigators ‘blind’ about treatment allocation.

Yes Can’t say

No

1.5 The treatment and control groups are similar at the start of the trial. Yes Can’t say □

No

1.6 The only difference between groups is the treatment under investigation.

Yes Can’t say

No


Yes Can’t say

No

1.8 What percentage of the individuals or clusters recruited into each treatment arm of the study dropped out before the study was completed?

1.9 All the subjects are analysed in the groups to which they were randomly allocated (often referred to as intention to treat analysis).

Yes Can’t say

No Does not apply


Yes Can’t say

No Does not apply

SECTION 2: INTERNAL VALIDITY - NUTRITION SPECIFIC ASPECTS

In a well-conducted nutrition RCT study… Does this study do it? 2.1 The relevant intake/nutritional status at baseline is sufficiently

documented. Yes No Can’t say

2.2 The baseline intakes of all groups have been maintained over the course of the study.

Yes No Can’t say

2.3 The amount of intervention intake under study has been quantitatively measured and the form, where relevant, has been described.

Yes No Can’t say

2.4 Adherence to the intervention and control intakes is being monitored in a reliable and accurate manner.

Yes No Can’t say

2.5 The interval between the intervention and outcome is of sufficient duration to observe an effect, if there is one.

Yes No Can’t say

SECTION 3: OVERALL ASSESSMENT OF THE STUDY

3.1 How well was the study done to minimize bias? When making your overall assessment of the design and analysis of the study, take into consideration the importance and relevance of each of the preceding 15 items as it relates to your research question and weight it in your overall assessment accordingly. The rationale for the assessment strategy should be documented.

High quality (++)

Acceptable (+)

Unacceptable – reject 0

3.2 Taking into account clinical/population health considerations, your evaluation of the methodology used, and the statistical power of the study, are you certain that the overall effect is due to the study intervention?

3.3 Are the results of this study directly applicable to the patient/population group targeted by this guideline?

3.4 Notes. Summarize the authors’ conclusions. Add any comments on your own assessment of the study, and the extent to which it answers your question and mention any areas of uncertainty raised above.

i For this checklist, the term ‘nutrition studies’ is used broadly to apply to studies that focus on intake of a food substance, food or dietary pattern. ‘Food substances’ are defined as essential or conditionally essential nutrients, energy nutrients, or other naturally-occurring bioactive food components. Food substance intakes can be derived from the intake of food sources (naturally-occurring foods, fortified/supplemented food) and dietary supplements. For some food substances, exposure may come from other sources (e.g., from industrial minerals; vitamin D from UV-B rays), which should be taken into account. Studies that focus on foods will generally require documentation of food composition and factors that alter composition, such as processing and fortification/supplementation. Studies of dietary patterns often also require documentation of food composition and/or prevalence of the use of specific types of foods. ii For this checklist, in the context of a nutrition study, ‘intervention’ refers to intake. The intervention group consumes the intervention that is being tested and the control group consumes a placebo, a different intake level, or a ‘usual’ intake. For example, an intervention group could consume a vitamin D supplement of 600 IU/day and the control group a placebo or a lower dose vitamin D supplement. If the trial is a dietary pattern study, the intervention group could consume a Mediterranean diet and the control group could consume a ‘usual’ dietary pattern or a ‘Western-style’ diet.

2.0 Assessing intake/nutritional status. For all study groups, baseline assessments are needed prior to the initiation of the study and background intakes/status assessments should be monitored throughout the duration of the study. Also, assessment of the intervention intake (e.g., from supplement, food or dietary pattern) and subject adherence to the intervention protocol are required. It is important to assess the potential impact of intervention measurement characteristics and errors on the diet-health outcomes relationship1. This is applicable to sections 2.2 and 2.4. Food substance intakes from self-reported diets: The study should report the method used to assess intakes (e.g. type of dietary assessment such as food frequency questionnaire, food record or 24-hour recall) and the approach used to validate the method.. Methods have different strengths and weaknesses (known errors). The best method depends on the research question1 and generally needs to be determined as part of the a priori PICO documentation process. The frequency of evaluation of the intake/status/intervention adherence will often depend on the food substance and the outcome. Food substance intakes from supplements or specific food products (e.g. bars, liquid drinks, fortified foods). The study should report product composition details and the methods used to confirm the composition of the intervention product (e.g., assay procedure) and to monitor adherence to use of these products (e.g. supplement count, assessment of changes in nutritional status). For dietary pattern or food intakes, the study should include ranges or distributions of any pre-specified food substance intakes of interest. The study should also include intervention measurement characteristics and errors, of which some can be controlled for statistically (e.g., ‘usual’ intakes) while others cannot (e.g., they may introduce bias)1. Biomarkers of intake and status are usually metabolites recovered from the blood or urine which objectively and accurately assess intake over a period of time. However, biomarker assays can vary in accuracy and reliability. If a biomarker has been accurately measured and appropriately qualified2 for its intended purpose, it is more reliable than intake assessed by dietary intake estimation methods. Rating: Documentation of intake/nutritional status should be sufficient to determine its reliability/accuracy rating. For example, intake based on a comparison to an accurate external method (e.g. a qualified biomarker) would have the highest rating. Self-reported intake or a biomarker assay procedure with accuracy or bias concerns that have been described/evaluated would have higher uncertainty. Self-reported intake with no description of accuracy or bias or not meeting criteria for a moderate rating would have the lowest rating. If intervention adherence is assessed by a supplement count then there is greater confidence in the measurement than if the assessment is achieved by recall. The tolerance for uncertainty will vary depending on the purpose of your review and will be reflected in the study rating. The rationale for the rating system used, including a description of tolerance for uncertainty, should be defined a priori.

Added Guidance – Controlled Trials

2.1 The relevant intake/nutritional status at baseline is sufficiently documented. It is important to note whether baseline intake/status has been accounted for in the analysis. Baseline intake level, or level of biomarker of intake (or status) allows for ascertainment of the nutritional similarity of intervention and control comparator groups at baseline, estimation of total intake (e.g., baseline intake plus supplement intake) and comparison of results across studies in different populations. Baseline intake or status can affect responsiveness to dietary intervention. For example, if a wide range of baseline statuses are present in a study group, the overall effect of the intervention may be null. This could happen if nutrient-depleted participants experience benefit but replete participants experience no effect or even harm.

2.2 The baseline intakes of all groups have been maintained over the course of the study.

All participants have background intakes of food substances, foods or dietary patterns. If the background intake for any or all of the groups changes to a substantial degree over the duration of the study, it could be more difficult to detect differences in outcome due to the intervention. Has the potential for this been monitored? And if this has occurred, does the study have appropriate methods to deal with it? If it is possible/likely that this has occurred, and has not been addressed, then the study should be rated lower. Higher rating would be given to studies where there are multiple measures of intake over time and especially where this is confirmed/assessed with biomarker data. Note: Section 2.0 offers guidance on how to assess the quality of the method to monitor baseline intake.

2.3 The amount of intervention intake under study has been quantitatively measured and the form, where relevant, has been described.

If the intervention composition is quantitatively defined (e.g. consumed as a supplement or controlled dietary intake), there is more confidence in the accuracy of the amount of the intervention than if it is calculated from a self-reported dietary intake method (self-reported intakes are subject to systematic biases). Knowledge about the form of the food substance intervention (e.g., naturally occurring vs. synthetic) and intake conditions (e.g., taken with or without meals; bolus vs multiple daily exposures, preparation practices for meal components) can be important especially when bioavailability and/or bioactivity are issues1. Other important considerations could be supplement brand name and components, or food types or recipes. Ideally, product composition would be confirmed by analysis because food/supplement label values and/or composition databases do not always accurately reflect actual composition. If the intervention is not measured in a quantitative and accurate manner or adequately characterized, confidence in the degree of intervention is reduced.

2.4 Adherence to the intervention and control intakes is being monitored in a reliable and accurate manner.

The study should report the method used to assess/monitor adherence to the intervention and control intake. The method and validation characteristics should be pre-specified. Methods may include dietary assessment such as food frequency questionnaire, food record, 24-hour recall, or supplement pill counts, or change in intake or nutritional status biomarkers. Failure to document the method used to monitor adherence to the intervention and control intake could result in the study being downgraded or rejected. Note: Section 2.0 offers guidance on how to assess the quality the method to monitor intake/status or supplement consumption.

2.5 The interval between the intervention and outcome is of sufficient duration to observe an effect, if there is one. In order to determine whether a ‘true’ relationship exists, the duration of the interval between the intervention and the outcome must be sufficient to allow for the outcome to occur. False positives can occur in studies of inadequate duration e.g., transient decreases that do not persist in long-term studies (e.g., LDL-cholesterol on short-term lipid studies that returns to normal over time). Confounding factors such as seasonal changes can transiently affect status or intake. Even study enrolment can transiently affect intake or status in the short-term. Identify a minimum follow-up time for each outcome during the PICO documentation step. If this minimum cannot be met, the study could be downgraded or rejected. If studies of insufficient duration are included, their results should be stratified and compared to the results of the longer term studies.

Health Canada Quality Assessment Nutrition Project - Status

• Nutrition-specific QAI and guidance for RCTs, cohort studies and case-control studies have been developed and being tested

• For RCT QAI, the testing has been conducted and modifications made • Currently testing cohort and case-control QAIs

• Manuscripts for submission being prepared

• Current trends consider risk of bias (ROB) at the level of individual studies and systematic reviews (Cochrane ROB Tool, ROBINS-I, ROBIS) – they focus on internal validity and specified outcome

• Critical Appraisal…” the use of explicit, transparent methods to assess the data in published research, applying the rules of evidence to factors such as internal validity, adherence to reporting standards, conclusions and generalizability”

• There is a need/demand for a pragmatic usable critical appraisal instrument to assist in the assessment of a body of evidence, for teaching and training, and as a checklist for the conduct of reviews

Moving from QAI to ROB - Value of Critical Appraisal

Domains assessed:

• Sequence generation • Allocation concealment • Blinding of participants, personnel and outcome assessors • Incomplete outcome data • Selective outcome reporting • Other sources of bias

Each domain is judged to be at Low, Unclear or High risk of bias • Guidance on making these judgments is available

Cochrane Tool for Assessing Risk of Bias for Randomized Controlled Trials

• ROBINS-I is concerned with evaluating the ROB in the results of non-randomized studies that compare the health effects of two or more interventions (NRSI)

• NRSI - quantitative studies estimating the effectiveness (harm and/or benefit) of an intervention, which did not use randomization to allocate units (individuals or clusters of individuals) to comparison groups

• Provides a systematic way to organize and present available evidence relating to ROB in NRSI with a focus on the internal validity of the study

ROBINS-I: “Risk Of Bias In Non-Randomized Studies of Interventions”

31

Published BMJ 2016;355:i4919

1. Bias domains

2. Signalling questions

A question that is broadly factual in nature providing insight into an aspect of bias under consideration and aims to facilitate judgements about the risk of bias

Responses to the signalling question are: NA not applicable; Y yes; PY for probably yes; PN for probably no; N for no ; NI for no information

An elaboration (free text description ) provided for each signalling question

3. Free text descriptions

4. Risk of bias judgements

Taking the responses to all the signalling questions for bias under consideration, a judgment is made on the risk of this bias

Guidance available for each domain of bias

Categories for risk of bias judgements are “Low risk”, “Moderate risk”, “Serious risk” and “Critical risk” of bias

5. (Predict direction of bias)

Optional - desirable to know magnitude and direction of any potential biases identified (i.e. towards or away from the null; or as increase or decrease in the effect estimate)

6. Overall risk of bias judgement

Judgements within each domain carry forward to an overall risk of bias judgement for the outcome being assessed

Overall judgement straightforward if domain-level judgements have same meaning with respect to concern about ROB of result

Guidance available

Categories for risk of bias judgements are “Low risk”, “Moderate risk”, “Serious risk” and “Critical risk” of bias

EVALUATING STRENGTH AND QUALITY AT THE SYSTEMATIC REVIEW LEVEL

• Assess QAI or ROB for each study in review • Assemble all assessments into summary table/figure • Assess overall quality/ROB

At Review Level - Collate QAI/ROB for Individual Studies

Risk of Bias for RCTs – Illustration: Summary of ROB for each study in review

Risk of Bias for RCTS – Illustration: Collate colors to provide ROB for review

• AMSTAR: A measurement tool to assess methodological quality of systematic reviews

• Widely used as a critical appraisal instrument: • to rate reviews • for teaching • a guide to the conduct of systematic reviews

• Was designed for RCTs, not NRSI

AMSTAR

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007 Feb 15; 7:10. PMID: 17302989.

AMSTAR – Methodological Checklist - Systematic Review and Meta-Analysis of RCTs 1.1 The study addresses a clearly defined research question.

1.2 At least two people should select studies and extract data.

1.3 A comprehensive literature search is carried out.

1.4 The authors clearly state if or how they limited their review by publication type.

1.5 The included and excluded studies are listed.

1.6 The characteristics of the included studies are provided.

1.7 The scientific quality of the included studies is assessed and documented.

1.8 The scientific quality of the included studies was assessed appropriately.

1.9 Appropriate methods are used to combine the individual study findings.

1.10 The likelihood of publication bias is assessed.

1.11 Conflicts of interest are declared.


2.1 What is your overall assessment of the methodological quality of this review? ++ / + / 0 Considering the quality criteria assessed, rate the overall methodological quality of the study, using the following as a guide: High quality (++): Majority of criteria met. Little or no risk of bias. Results unlikely to be changed by further research. Acceptable (+): Most criteria met. Some flaws in the study with an associated risk of bias, Conclusions may change in the light of further studies. Low quality (0): Either most criteria not met, or significant flaws relating to key aspects of study design. Conclusions likely to change in the light of further studies.

At the Review Level – Quality of SR of RCTs

Evaluation of QAIs for Systematic Reviews

Bai A, Shukla VK, Bak G, Wells G. Quality Assessment Tools Project Report. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2012.

• Maintain important characteristics of the original instrument: – Critical appraisal – Usable – Self-contained (self-explanatory as far as possible)

• Important updates:

– More comprehensive – Inclusion of non-randomized studies – Alignment with ROBINS-I – several investigators on both groups – Altered scaling of ratings

Aims of the Redevelopment of AMSTAR

Beverley Shea (Ottawa, Canada) George Wells (Ottawa, Canada) Micere Thuku (Ottawa, Canada) Barney Reeves (Bristol, UK) David Moher (Ottawa, Canada) Peter Tugwell (Ottawa, Canada) Candyce Hamel (Ottawa, Canada) Vivian Welch ((Ottawa, Canada) Jeremy Grimshaw (Ottawa, Canada) Lex Bouter (Amsterdam, NL) Maarten Boers (Amsterdam, NL) Betsy Kristjansson (Ottawa, Canada) Zulma Ortiz (Buenos Aries, AR) Gordan Guyatt (Hamilton, Canada) Alonso Carrasco (Hamilton, Canada) David Henry (Toronto, Canada)

AMSTAR2 Development Team

Shea BJ, Reeves B, Wells GA et al, AMSTAR2: a measurement tool to assess the methodological quality of systematic reviews. BMJ 2017.

AMSTAR 2 Study selection, and data extraction

Exclusions and included study characteristics

Authors’ assessment of ROB with individual studies

Funding

Inclusion of NRSI in meta-analysis

Impact of ROB on results and interpretation of the review

Recently developed Currently in development In planning

Study design taxonomy

(Reeves BC, Wells GA, Waddington H, Journal of Clinical Epidemiology, 2017 (in press))

Nutrition-specific QAI (SIGN 50 bolt-on) for RCT, cohort studies and case-control studies

Nutrition-specific QAI (de novo) for RCTs, cohort studies, case-control studies and cross-sectional studies

AMSTAR2 for evaluating RCT and NRS systematic reviews

(Shea B, Reeves BC, Wells GA et al, BMJ, 2017 (in press))

Nutrition-specific QAI (analogous to SIGN 50 QAI) for cross-sectional studies

Nutrition-specific ROB (bolt-on)

ROBINS-I for evaluating risk of bias for NRSI

(Sterne J, Heman M, Reeves BC et al, BMJ 2016)

Revised risk of bias tool for RCT (development led by Bristol/Cochrane)

Nutrition-specific tool (AMSTAR2 bolt-on) for evaluating RCT and NRS systematic reviews

Next steps …

Questions or Thoughts

Date post:	23-Aug-2019
Category:	Documents
Upload:	ngotuyen
View:	213 times
Download:	0 times

Tools for Evaluating Strength and Quality of...

Documents