J

B

Ad

DSV

a

ARRAA

KBAV

1

hAcits

rodtLtd

d

e(e

h0

ARTICLE IN PRESSG ModelVAC-15351; No. of Pages 4

Vaccine xxx (2014) xxx–xxx

Contents lists available at ScienceDirect

Vaccine

j our na l ho me page: www.elsev ier .com/ locate /vacc ine

rief report

n algorithm developed using the Brighton Collaboration caseefinitions is more efficient for determining diagnostic certainty

eepa Joshi ∗, Emily Alsentzer, Kathryn Edwards, Allison Norton,arah Elizabeth Williamsanderbilt University Medical Center, Light Hall, 2215 Garland Avenue, Mailbox #43, Nashville, TN 37232, USA

r t i c l e i n f o

rticle history:eceived 22 January 2014eceived in revised form 21 March 2014ccepted 22 April 2014vailable online xxx

a b s t r a c t

The Brighton Collaboration is a global research network focused on vaccine safety. The Collaboration hascreated case definitions to determine diagnostic certainty for several adverse events. Currently nestedwithin multi-page publications, these definitions can be cumbersome for use. We report the results of arandomized trial in which the case definition for anaphylaxis was converted into a user-friendly algorithmand compared the algorithm with the standard case definition. The primary outcomes were efficiency

eywords:righton Collaborationdverse events following immunizationaccine safety monitoring

and accuracy. Forty medical students determined the Brighton Level of diagnostic certainty of a samplecase of anaphylaxis using either the algorithm or the original case definition. Most participants in bothgroups selected the correct Brighton Level. Participants using the algorithm required significantly lesstime to review the case and determine the level of diagnostic certainty [mean difference = 107 s (95% CI:13–200; p = 0.026)], supporting that the algorithm was more efficient without impacting accuracy.

© 2014 Elsevier Ltd. All rights reserved.

. Introduction

Immunizations are powerful public health interventions thatave been very effective in reducing global disease burden [1].lthough vaccines are generally safe, there are certain risks asso-iated with their administration. While adverse events followingmmunization (AEFI), such as anaphylaxis, are rare, efficient sys-ems to monitor AEFIs are essential to systematically assess vaccineafety.

The Brighton Collaboration is a non-profit, internationalesearch network that provides standardized, validated, andbjective case definitions for monitoring vaccine safety [2]. Theefinitions provide clinical and diagnostic criteria to allow AEFIso be assigned to one of three levels of “diagnostic certainty”; aevel 1 indicates the highest level of confidence that an AEFI meets

Please cite this article in press as: Joshi D, et al. An algorithm develefficient for determining diagnostic certainty. Vaccine (2014), http://d

he corresponding diagnosis. The use of these standardized caseefinitions in research and clinical settings will more precisely

Abbreviations: AEFI, adverse events following immunization; LMICs, low to mid-le income countries; GVSI, Global Vaccine Safety Initiative.∗ Corresponding author. Tel.: +1 402 305 8135.

E-mail addresses: deepa.s.joshi@vanderbilt.edu (D. Joshi),ma2016@stanford.edu (E. Alsentzer), Kathryn.edwards@vanderbilt.eduK. Edwards), allison.norton@vanderbilt.edu (A. Norton),lizabeth.williams@vanderbilt.edu (S.E. Williams).

ttp://dx.doi.org/10.1016/j.vaccine.2014.04.070264-410X/© 2014 Elsevier Ltd. All rights reserved.

characterize events, leading to a better understanding of the truerisk of AEFIs [3].

Because the case definition format is generally a footnoted tablenested within a 10–20 page journal article, the goal of this studywas to convert one Brighton case definition into an algorithm, andevaluate the efficiency and accuracy of the algorithm compared tothe original case definition.

2. Materials and methods

2.1. Algorithm development

In July 2012 the Brighton Collaboration case definition of ana-phylaxis [4] was reviewed. Key clinical criteria that distinguishedthe levels of diagnostic certainty were abstracted and using Smart-Draw software [5] were transposed into a step-wise algorithm thatguided users to the appropriate level (Fig. 1). The algorithm wastested by applying it to cases of anaphylaxis identified through aPubmed search. The algorithm was reviewed by one pediatricianand one allergist to verify that the criteria matched the originalcase definition.

oped using the Brighton Collaboration case definitions is morex.doi.org/10.1016/j.vaccine.2014.04.070

2.2. Sample case

The sample case was selected from clinical cases presentedto the Clinical Immunization Safety Assessment network (CISA)

Please cite this article in press as: Joshi D, et al. An algorithm developed using the Brighton Collaboration case definitions is moreefficient for determining diagnostic certainty. Vaccine (2014), http://dx.doi.org/10.1016/j.vaccine.2014.04.070

ARTICLE IN PRESSG ModelJVAC-15351; No. of Pages 4

2 D. Joshi et al. / Vaccine xxx (2014) xxx–xxx

Fig. 1. Algorithm for anaphylaxis developed from the Brighton Collaboration case definition.

ARTICLE IN PRESSG ModelJVAC-15351; No. of Pages 4

D. Joshi et al. / Vaccine xxx (2014) xxx–xxx 3

Baby B is a 12-month-ol d m ale with a history of atopic d erm a��s , egg all ergy (pri ck +) , and report of hives a�er ea�ng carrots. The pa�ent visited the allergist’s office to receive his 1st influenza vaccine. Due to the pa�ent’s history of egg allergy, the Influenza, Inac�vated Vaccine (TIV) was administered per 2-part protocol to right quadrice p:

1. TIV 0 .05cc Intr amuscular ly: The pa�ent was note d to have full -stre ngth, whi ch was monitored for 30 minutes. No other adverse reac�on was noted. 2. TIV 0.20cc Intramuscularly: Within 30 minutes of the 2nd dose, the pa�ent experienced ur�caria of neck, anterior torso, and both arms. He also became fussy and started crying. Baby B had increased work of breathing, coughing, stridor, retrac�ons, mild erythema, and angioedema of uvula. The following lab values were noted:

Respiratory Rate

O2 Sat ura�o n

Pulse Systolic Blood Pressure

Befo re First Vaccine

24 bpm 97% 124bpm 110

A�er Second Vaccine

42 bpm 93% 137bpm 176

The pa�ent was treated with epinephrine intramuscularly along with ce�rizine, prednisolone, and albuterol. The child went home, and the night a�er the clinic vis it, the child experienced diarrhea, mild wheezing, and a blotchy rash. He was treated in the cli nic w ith albu terol and Benadryl. The family re ports th at the child h ad not eat en any e

(tlB

2

tcrwtPV

r1o(lvwartnnesb

tfTiease

Table 1Assignment of Brighton Level of diagnostic certainty for sample case by resourceused.

Brighton Level Case definition Algorithm Total

1 16 (80%) 15 (75%) 31 (77.5%)2 2 (10%) 5 (25%) 7 (17.5%)3 2 (10%) 0 2 (5%)

ggs, nuts, or carrots, and ther e were no changes in the home.

Fig. 2. Sample case.

Fig. 2) [6,7] and reviewed by a board-certified allergist using bothhe standard case definition and the algorithm to determine theevel of diagnostic certainty; the case was determined to meetrighton Level 1 criteria.

.3. Study design

Second year Vanderbilt medical students were recruited for par-icipation. The students had no prior experience reading a Brightonase definition. Students signed up electronically for time slots aftereceiving an electronic mailing introducing the study. Participantsho signed up received a reminder electronic mailing 1 h prior

o their scheduled time. Written, informed consent was obtained.articipants were provided a small financial compensation. Theanderbilt Institutional Review Board approved the study.

Using block randomization, participants were randomized toeceive 1 of 2 resources: the original case definition (nested in the0 page journal article [8]) or the algorithm (Fig. 1). Prior to studynset, students received identical introduction sheets explaining1) the purpose of the study and (2) the goal to determine theevel of diagnostic certainty for the sample case using the pro-ided resource. For the original case definition [8], participantsere directed on the first page of the printed article to turn to

table on a later page that summarized the criteria. The secondesource, the algorithm, was a two-page document that led the userhrough a series of questions until the appropriate level of diag-ostic certainty was reached (Fig. 1). Additional instructions wereot provided. Participant identification numbers were assigned toach study participant sequentially. Both participants and studytaff were blinded to the assigned resource until the participantsegan the study.

Participants were asked to use the provided resource to selecthe level of diagnostic certainty (1, 2, 3 or insufficient information)or the sample case. Each participant was given an electronic timer.he study staff read a standard instruction script when all partic-pants arrived and advised participants to start their timers at the

Please cite this article in press as: Joshi D, et al. An algorithm develefficient for determining diagnostic certainty. Vaccine (2014), http://d

nd of the verbal instructions and to stop their timers immediatelyfter marking their choice. Each participant’s time was recorded bytudy staff, and all study documents were collected at the end ofach session.

Total 20 20 40

Fisher’s exact; p = 0.246.

2.4. Statistical analysis

A sample size of 17 participants in each study arm was esti-mated to provide 80% power to detect a significant difference inthe time to complete the study between the two resources, assum-ing a standard deviation of 10 min and an alpha of 0.05. To evaluatethe efficiency of the algorithm compared to the standard case defi-nition, a comparison of the average time required to determine thelevel of diagnostic certainty for the sample case was achieved usinga two group, two sided t-test. The algorithm’s ability to achieve thesame level of diagnostic certainty for the sample case as the casedefinition was analyzed using Fisher’s exact test.

3. Results

Forty participants enrolled in the study; 20 in each study arm.The average age of participants was 23.7 years overall [24.1 yearsfor the case definition group, 23.7 years for the algorithm (p = 0.48)].All participants completed the study and selected a level of diag-nostic certainty (Table 1).

The majority of participants in both study arms selected the cor-rect Brighton Level (80% using the case definition and 75% using thealgorithm; p = 0.246). Among all participants who incorrectly iden-tified the case as not meeting Level 1, those using the algorithm allchose Level 2 (n = 5) whereas those using the case definition choseLevel 3 (n = 2) as well as Level 2 (n = 2). No participants selected“Insufficient Information” as a level.

The mean time for participants to review the sample case andselect a Brighton Level was 415 s (95% CI: 341–490) using the casedefinition, and 309 s (95% CI: 253–364) using the algorithm, repre-senting a significant difference in the mean time to review the caseand determine the level by resource used [mean difference = 107 s(95% CI: 13–200; p = 0.026)].

4. Discussion

This study supports the hypothesis that an algorithm devel-oped using the Brighton Collaboration case definition for an AEFIcan result in the assignment of an appropriate level of diagnos-tic certainty in a significantly shorter period of time. Because ourstudy design directed participants to the criteria table within thecase definition journal article, the time required to locate the cri-teria for a researcher unfamiliar with the current journal formatfor Brighton definitions was eliminated. If we had not includedthis instruction, the difference in time to assign a level betweenresources may have been much greater. Given that most peoplewho refer to the case definitions are familiar with the structure,this design best replicates the use of the Brighton case definitionsand most accurately demonstrates the time difference between thetwo resources.

The development of more efficient, practical tools for assess-

oped using the Brighton Collaboration case definitions is morex.doi.org/10.1016/j.vaccine.2014.04.070

ing AEFI is needed. New vaccine products used globally lead to agreater demand for vaccine safety surveillance, particularly in lowto middle income countries (LMICs) [9]. This is supported by theWorld Health Organization’s Global Vaccine Safety Initiative (GVSI)

ING ModelJ

4 cine xx

it[atAacit

tiwtpWotstti((a

rlttLLdc

5

dli

ARTICLEVAC-15351; No. of Pages 4

D. Joshi et al. / Vac

n 2012, whose objectives include the development of interna-ionally harmonized tools to support vaccine surveillance in LMICs10]. Aggregating data across multiple surveillance systems wouldllow for more statistical power and a greater probability of iden-ifying rare AEFI; however, assembling of data is only possible ifEFI are based on similar criteria [10]. Brighton case definitionsllow for standardization of AEFI reporting criteria, and more effi-ient methods for AEFI definition would enhance this effort, bothn LMICs and in countries with strong vaccine surveillance sys-ems.

Our study has several limitations. The sample size is small, andhe study may not be generalizable as all participants were med-cal students. However, many users of Brighton case definitions

ill be healthcare providers who have completed medical schoolraining. Another potential limitation is that the participants com-leted the study in the same room as up to four other participants.hen a participant stopped their timer, it beeped, signifying to all

ther participants in the room that the individual had completedhe study. Thus, some participants may have rushed to finish thetudy after hearing another participant’s timer beep. Additionally,he inaccurate assignment of the level by a few participants usinghe algorithm may be a result of its format. In the algorithm’s list-ng of minor respiratory criteria, “Stridor” appears on its own lineFig. 1). Participants may not have read the complete statement“difficulty breathing without wheeze or stridor”) and incorrectlyssigned the case a Brighton Level 2.

Future research should include testing the efficiency and accu-acy of the algorithm by application to multiple cases with differentevels of diagnostic certainty and with participants with variousraining. Furthermore, since all participants who incorrectly iden-ified the case as not meeting Level 1 using the algorithm all choseevel 2 (n = 5) whereas those using the case definition table choseevel 3 (n = 2) and Level 2 (n = 2), it would be interesting to eluci-ate the rationale for this inaccurate determination and thereafterorrect the algorithm to avoid this error.

. Summary

Please cite this article in press as: Joshi D, et al. An algorithm develefficient for determining diagnostic certainty. Vaccine (2014), http://d

An algorithm developed from the Brighton Collaboration caseefinition for anaphylaxis was more efficient in determining the

evel of diagnostic certainty for a case of anaphylaxis, withoutmpacting accuracy. Easily disseminated methods for identifying

[

PRESSx (2014) xxx–xxx

AEFIs and systematic methods for analysis would allow forenhanced comparability of vaccine safety data.

Conflicts of interest

The authors have no conflicts of interest relevant to this articleto disclose. The authors have no financial relationships relevant tothis article to disclose.

Acknowledgements

The Vanderbilt Institute for Clinical and Translational Researchgrant supported this project (UL1 TR000445 from NCATS/NIH).Study data were collected and managed using REDCap electronicdata capture tools hosted at Vanderbilt University.

References

[1] CDC. Impact of vaccines universally recommended for children – UnitedStates, 1990–1998. MMWR Morb Mortal Wkly Rep 1999;48(April (12)):243–8.PubMed PMID: 10220251.

[2] The Brighton Collaboration. Available from: https://brightoncollaboration.org/internet/en/index.html [23.10.10].

[3] Bonhoeffer J, Kohl K, Chen R, Duclos P, Heijbel H, Heininger U, et al. TheBrighton Collaboration: addressing the need for standardized case definitionsof adverse events following immunization (AEFI). Vaccine 2002;21(December(3–4)):298–302. PubMed PMID: 12450705.

[4] Ruggeberg JU, Gold MS, Bayas JM, Blum MD, Bonhoeffer J, Friedlander S,et al. Anaphylaxis: case definition and guidelines for data collection, analy-sis, and presentation of immunization safety data. Vaccine 2007;25(August(31)):5675–84. PubMed PMID: 17448577.

[5] SmartDraw; 2013. Available from: http://www.smartdraw.com/ [10.10.13].[6] LaRussa P, Edwards K, Dekker C, Klein N, Halsey N, Marchant C, et al.

Understanding the role of human variation in vaccine adverse events:the Clinical Immunization Safety Assessment (CISA) network. Pediatrics2011;127n3(March (Vaccine Safety Supplement)).

[7] CDC. Vaccine safety – Clinical Immunization Safety Assessment (CISA) net-work. Available from: http://www.cdc.gov/vaccinesafety/Activities/cisa.html[updated 02/25/201008/04/2010].

[8] Gold MS, Gidudu J, Erlewyn-Lajeunesse M, Law B, Brighton CollaborationWorking Group on A. Can the Brighton Collaboration case definitions beused to improve the quality of Adverse Event Following Immunization (AEFI)reporting? Anaphylaxis as a case study. Vaccine 2010;28(June (28)):4487–98.PubMed PMID: 20434547.

[9] Amarasinghe A, Black S, Bonhoeffer J, Carvalho SM, Dodoo A, Eskola J, et al.

oped using the Brighton Collaboration case definitions is morex.doi.org/10.1016/j.vaccine.2014.04.070

Effective vaccine safety systems in all countries: a challenge for more equitableaccess to immunization. Vaccine 2013;31(April (Suppl. 2)):B108–14. PubMedPMID: 23598471.

10] WHO Global Vaccine Safety Blueprint; 2012. Available from: http://www.who.int/vaccine safety/en/ [10.10.2013].