Key words: STARD checklist, Cardiology, Diagnostic accuracy

Completeness of reporting of studies of diagnostic accuracy in Cardiology

journals: a STARD assessment

Key words: STARD checklist, Cardiology, Diagnostic accuracy

Class 6: Ana Borges, Ana Torres, André Tojal, Célia Soares, Francisca Trigo, Francisco Coutinho, Joana

Ribeiro, José Fernandes, Pedro Vasconcelos, Raquel Ferreira, Sílvia Madureira

Supervisors: Altamiro da Costa Pereira, MD, PhD; Filipa Almeida, MD

Introduction to Medicine; Porto Faculty of Medicine 2008

Objectives Methods AcknowledgmentsResultsIntroduction Conclusion

IntroductionIntroduction Objectives Methods AcknowledgmentsResults

• Accurate diagnosis in Cardiology is crucial for clinical intervention and is increasingly important as the number of validated treatment for specific conditions increases.

• The use of systematic literature review to inform evidence based practice in diagnostics is rapidly expanding.

• Studies of the new diagnostic tests, and their development, are reported in scientific articles. (1)

(1) - Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW. Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative. Ann Intern Med. 2003; 138:40-44

Conclusion


•Current cardiology practice relies on diagnostic tests using sophisticated technologies that are constantly evolving. Diagnostic accuracy studies are required to validate new diagnostic tests before they are introduced into clinical practice.

• These studies typically report sensitivity and specificity, likelihood ratios, diagnostic odds ratio, or area under a receiver operating characteristics (ROC) curve as measures of diagnostic performance. This information enables a clinician to make judgments regarding the potential utility of new tests. (2)

(2) Deeks J. Systematic reviews in health care: systematic reviews of diagnostic tests, BMJ 2001; 44:639-650

Conclusion


•A rigorous evaluation process of diagnostic tests before introduction into clinical practice could not only reduce the number of unwanted clinical consequences related to misleading estimates of test accuracy, but also limit health care costs by preventing unnecessary testing.

• Several systematic reviews (1995, Reid et al) (3) have emphasized the poor quality of reporting in diagnostic accuracy studies. This poor reporting hampers an adequate judgment of both the internal and the external validity of a study.

(3) Scholten RJ, Deville WL, Opstelten W, Bijl D, van der Plas CG, Bouter LM. The accuracy of physical diagnostic tests for assessing meniscal lesions of the knee: a meta-analysis. J Fam Pract. 2001; 50:938–944.

Conclusion


•In 19991999, in Rome, the Cochrane Diagnostic and Screening Test Methods Working group discussed the low methodological quality and sub-standard reporting of diagnostic test evaluations, that resulted in the intent of developing a checklist of items that should be included in those reports.

•The STARD steering committee started with an extensive search to identify publications on the conduct and reporting of diagnostic studies up to July 20002000 .(4)

•In January 20032003, after the discussion based on the search done, we have the first official version of the STARD, which was published simultaneously in eight medical journals.

History of the STARDHistory of the STARD

(4) Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003

Conclusion


STARD CHECKLISTSTARD CHECKLIST

Objective:Objective:•The aim of the Standards for Reporting of Diagnostic Accuracy (STARD) initiative is to improve the quality of reporting of studies on diagnostic accuracy.

How:How:• Using a checklist with 25-item and a flow diagram, which provides information about the method of patient recruitment, the order of test execution and the numbers of patients undergoing the test under evaluation, the reference standard or both. (5)

Advantage:Advantage:•Complete and accurate reporting allows the reader to detect the potential for bias in the study and to evaluate the general ability of the results.

(5) Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. JAMA. 1995; 274:645–651.

Conclusion

Objectives Methods AcknowledgmentsResults

• Cardiology is the area with the most published articles;

•This is the first study that analyses how reports on diagnostic accuracy in cardiology

comply with the STARD checklist;

IntroductionIntroduction

Importance of our analysisImportance of our analysis

Conclusion

Introduction ObjectivesObjectives Methods AcknowledgmentsResults

To assess the quality of reporting of studies on diagnostic tests in cardiology journals, based on the STARD checklist.

Main objectiveMain objective

Secondary objectivesSecondary objectives

•To analyse the quality of diagnostic tests studies in the course of the years;

•To compare the differences between the articles published before and after the creation of the STARD checklist;

•To evaluate the influence of the journals’ impact factor on the quality of the articles which report diagnostic tests in cardiology journals.

Conclusion

Articles’ searchArticles’ search

Articles’ selection

Articles’ evaluation

Statistical analyze

Introduction Objectives MethodsMethods AcknowledgmentsResults

Standard query to find articles that report studies of diagnostic accuracy (in PubMed)

(((((((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields)) OR (((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All fields]) OR "roc and"[All fields]) OR "roc area"[All fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields])) OR "likelihood ratio"[All Fields])

Devillé W. Conducting systematic reviews of diagnostic studies; didactic guidelines. Medical research methodology. 2002

Conclusion


Search the journals




Statistical analyze

Included all journals classified as “cardiac and cardiovascular systems” in the ISI web of knowledge except those who are not found on pubmed.

Included all 71 journals found on Pubmed

Conclusion


Final query (after addition of terms to specify the search to the journals)




Statistical analyze

(((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR ("likelihood ratio"[All Fields])) AND (Circulation [Journal] OR Circulation Research [Journal] OR Journal Of The American College Of Cardiology [Journal] OR European Heart Journal [Journal] OR Cardiovascular Research [Journal] OR Journal Of Molecular And Cellular Cardiology [Journal] OR American Journal Of Physiology-Heart And Circulatory Physiology [Journal] OR Journal Of Thoracic And Cardiovascular Surgery [Journal] OR Trends In Cardiovascular Medicine [Journal] OR American Journal Of Cardiology [Journal] OR Journal Of Heart And Lung Transplantation [Journal] OR Chest [Journal] OR American Heart Journal [Journal] OR Journal Of Cardiovascular Pharmacology [Journal] OR Progress In Cardiovascular Diseases [Journal] OR Journal Of Cardiovascular Magnetic Resonance [Journal] OR Heart [Journal] OR Journal Of Nuclear Cardiology [Journal] OR Annals Of Thoracic Surgery [Journal] OR Journal Of The American Society Of Echocardiography [Journal] OR Current Opinion In Cardiology [Journal] OR Current Problems In Cardiology [Journal] OR Basic Research In Cardiology [Journal] OR Catheterization And Cardiovascular Interventions [Journal] OR Respiratory Medicine [Journal] OR Canadian Journal Of Cardiology [Journal] OR European Journal Of Cardio-Thoracic Surgery [Journal] OR European Journal Of Heart Failure [Journal] OR Journal Of Heart Valve Disease [Journal] OR Clinical Cardiology [Journal] OR Cardiovascular And Interventional Radiology [Journal] OR International Journal Of Cardiology [Journal] OR Herz [Journal] OR Cardiovascular Drugs And Therapy [Journal] OR Journal Of Cardiothoracic And Vascular Anesthesia [Journal] OR Cardiovascular Drug Reviews [Journal] OR Zeitschrift Fur Kardiologie [Journal] OR Pediatric Cardiology [Journal] OR Cardiovascular Surgery [Journal] OR Thoracic And Cardiovascular Surgeon [Journal] OR Journal Of Interventional Cardiac Electrophysiology [Journal] OR Revista Espanola De Cardiologia [Journal] OR Cardiology [Journal] OR Heart & Lung [Journal] OR Cardiology In The Young [Journal] OR Heart And Vessels [Journal] OR Journal Of Electrocardiology [Journal] OR Journal Of Cardiovascular Surgery [Journal] OR International Journal Of Cardiac Imaging [Journal] OR Texas Heart Institute Journal [Journal] OR Scandinavian Cardiovascular Journal [Journal] OR Acta Cardiologica [Journal] OR Europace [Journal] OR Archives Des Maladies Du Coeur Et Des Vaisseaux [Journal] OR Cardiovascular Pathology [Journal] OR Japanese Heart Journal [Journal] OR Journal Of Cardiac Surgery [Journal] OR Kardiologiya [Journal])

Conclusion





Statistical analyze

• Random sample of 100 articles stratified by years

• Sample of 10 articles per year

• Years included:- Until 1998, 1999, 2000, 2001, 2002, 2003, 2004,

2005, 2006, 2007.

From all articles retrieved by the query

Conclusion

Introduction Objectives AcknowledgmentsResultsMethodsMethods

Reading and analyzing the abstractArticles’ search

Articles’ selectionArticles’ selection


Statistical analyze

InclusionInclusion

- to report a study of diagnostic accuracy - the diagnostic test(s) is (are) used in cardiology- the full text is available

The abstract is read and analyzed with the inclusion/exclusion criteria

Conclusion


Reading and analyzing the abstractArticles’ search



Statistical analyze

ExclusionExclusion

- Is a clinical trial, a letter, an editorial, a practice guideline, a meta-analysis or a review

-The article could not be found in the Internet, the facilities of the School of Medicine or the local libraries

-The article is written in languages other than English, Portuguese, French or Spanish

- The article found is not complete

Conclusion


RevisionArticles’ search



Statistical analyze

If excluded:- the reason of exclusion is registered

If doubted:- the abstract is reanalyzed by other selector

When disagreement or doubt occur, the article will be discussed by both selectors. When no decision could be achieved, a third selector will be consulted

When this situation occurs, PMID of each opened article will be registered

Conclusion


Final samples

Articles’ search



Statistical analyze

2007 → 139 articles









Until 1998 → 1585 articles

Conclusion


Final samples

Articles’ search



Statistical analyze

TOTAL → 2785 articles

Conclusion


Articles’ search



Statistical analyze

Flowchart


Articles’ search



Statistical analyze

Gantt Chart

Conclusion

Articles’ search


Articles’ evaluationArticles’ evaluation

Statistical analyze

Objectives ResultsMethodsMethods

Selection and Topic Item Describe

TITLE/ABSTRACT/KEYWORDS

1The article as a study on diagnostic accuracy (recommend MeSH heading 'sensitivity and specificity') ImpliedImplied

INTRODUCTION2

The research question(s), such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups The main objectives are presentThe main objectives are present

METHODS

Participants3

The study population: the inclusion and exclusion criteria, setting(s) and location(s) where the data were collected It has inclusion and exclusion criteriaIt has inclusion and exclusion criteria

4

Participant recruitment: was this based on presenting symptoms, results from previous tests, or

the fact that the participants had received the index test(s) or the reference standard? If it has two of the threeIf it has two of the three

5

Participant sampling: was this a consecutive series of patients defined by selection criteria in (3)

and (4)? If not specify how patients were further selected. If 3 and 4 have been markedIf 3 and 4 have been marked

6

Data collection: were the participants identified and data collected before the index test(s) and reference standards were performed (prospective study) or after (retrospective study)? If it says before ou after / impliedIf it says before ou after / implied

Reference standard 7 The reference standard and its rationale Nor that it doesn’t have rational analysis Nor that it doesn’t have rational analysis

Test methods8

Technical specification of material and methods involved including how and when measurements were taken, and/or cite references for index test(s) and reference standardNor that it doesn’t specify how and whereNor that it doesn’t specify how and where

9

Definition and rationale for the units, cutoffs and/or categories of the results of the index test(s) and the reference standard If it has the two topicsIf it has the two topics

10

The number, training and expertise of the persons (a) executing and (b) reading the index test(s) and the reference standard If it has at least two of the threeIf it has at least two of the three

11Whether or not the reader(s) of the index test(s) and reference standard were blind (masked) to the results of the other test(s) and describe any information available to them If it has the first topic or bothIf it has the first topic or both

Statistical methods12

Methods for calculating measures of diagnostic accuracy or making comparisons, and the statistical methods used to quantify uncertainty (e.g. 95% confidence intervals) Only if it has bothOnly if it has both

13 Methods for calculating test reproducibility, if done If yes / if it has been doneIf yes / if it has been done

STARD CHECKLIST

Each item values 1 point Several criteria was defined for the STAR application

Articles’ search



Statistical analyze

Objectives ResultsMethodsMethodsSection and topic Item Describe

RESULTS

Participants14

When study was done, including beginning and ending dates of recruitment If it has everything or the duration timeIf it has everything or the duration time

15

Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s),

comorbidity, current treatment(s), recruitment center) If it presents at least one of eachf it presents at least one of each

16

How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the reference standard; describe why participants failed to receive either test (a flow diagram is strongly recommended) If it has everything, even it doesn’t have the If it has everything, even it doesn’t have the diagramdiagram

Reference standard17

Time interval and any treatment administered between index and reference standard If it has everythingIf it has everything

18

Distribution of severity of disease (define criteria) in those with the target condition; describe other diagnoses in participants without the target condition If it has the first topic or If it has the first topic or bothboth

Test results

19

A cross tabulation of the results of the index test(s) by the results of the reference standard; for

continuous results, the distribution of the test results by the results of the reference standard If it has the first topic or both, in case the second is verifiedit has the first topic or both, in case the second is verified

20

Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result and how they were handled If it has the itemIf it has the item

21 Adverse events of index test(s) and reference standard If it has bothIf it has both

Estimation22

Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals) If it has two of the threeIf it has two of the three

23

Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done If it has everythingIf it has everything

24 Measures of test reproducibility, if done If it has the item and it’s doneIf it has the item and it’s done

DISCUSSION 25 The clinical applicability of the study findings If it has the itemIf it has the item

STARD CHECKLIST

Each item values 1 point Several criteria was defined for the STAR application


Evaluators – Distribution for year

Article’s components

- Title,

- Abstract,

- Keywords

- Introduction

- Methods

- Results

- Discussion

One evaluator for each year

Every article´s components are seen for the

same evaluator

10 articles for each evaluator

Articles’ search



Statistical analyze

Conclusion


Articles’ search



Statistical analyze

• Final score of each article

• Average score of the articles for each group (year of publication, impact factor)

• Interobserver variability

• The status of each STARD item in each year

Points to analyse:

Conclusion


Articles’ search



Statistical analyzeStatistical analyze

Statistical Analyse

Conclusion

Introduction Objectives AcknowledgmentsResultsResultsMethods

Final Selection States

From Articles’ SelectionFrom Articles’ Selection

From Articles’ Evaluation

Conclusion


Reasons for the Exclusion

From Articles’ From Articles’

SelectionSelection

From Articles’

Evaluation

Conclusion


Final Classification of the Items

From Articles’

Selection

From Articles’ From Articles’

EvaluationEvaluation

Conclusion


ICC for each Item of the List

From Articles’ Selection

From Articles’ EvaluationFrom Articles’ Evaluation

Conclusion


Evolution of the Articles’ Quality



Conclusion


The Impact Factor of the Journal



Conclusion

Introduction Objectives AcknowledgmentsResultsResultsMethods Conclusion

. . SSome items of evaluation are rigorously asked for in a section, but in the articles,

several times, they appear in another one;

. . Because some STARD items are potentially affected by the subjective judgment of

the reviewers due to differences in interpretation of the STARD, disagreement arises;

About the About the STARD CHECKLIST – STARD CHECKLIST – Evaluators’ point of viewEvaluators’ point of view


Relation Between the Articles’ Score and the Impact Factor of the Journal



Conclusion

Introduction Objectives AcknowledgmentsResultsResultsMethods Conclusion

. . The data of the participants (age, sex, symptoms,…) and their dates of

recruitment should be included in the methods;

. . The list has 25 items, but it’s possible each one can have more then one aspect

to evaluate, which creates confusion in the classification to attribute.

About the About the STARD CHECKLIST – STARD CHECKLIST – Evaluators’ point of view Evaluators’ point of view


Expected Results vs Obtained Results

Expected Expected •As we can see in other medical areas where this project had been done, such as

obstetrics and gynecology and others, we expect that this checklist had had impact

on the articles written after it as well as that they had improve its qualities due to

STARD Checklist.

Conclusion


Expected Results vs Obtained Results

ObtainedObtained. . The differences found in the articles were nothing to do with variances observed between groups, we mean the year of publish is not related with the improvement of the articles (p<0,005). . . Therefore we assume that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy and so we suppose that the improve in this kind of articles is due to the progress of knowledge along the years and not related with the creation of this checklist, as shown in the graphic.

. . We come to conclusion that there is no correlation between them, since we reach a p value equal to 0,210, that is not significant and by the way that we can declare that the year of publish or the improve verify in the articles does not interfere with the impact factor of the journals.

Conclusion

Introduction Objectives AcknowledgmentsResultsMethods

Conclusion

Our study shows that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy .

Conclusion

Introduction Objectives AcknowledgmentsAcknowledgmentsResultsMethods

• Altamiro da Costa Pereira, MD,PhD

• Filipa Almeida, MD, PhD

• Cristina Santos, MD, PhD

• Luís Azevedo, MD, PhD

AcknowledgmentsAcknowledgments

Date post:	22-Jan-2016
Category:	Documents
Upload:	awena
View:	35 times
Download:	0 times

Key words: STARD checklist, Cardiology, Diagnostic accuracy

Documents