Inter-Rater Reliability amongst Clinical Team-Members for Scoring

the Acute Asthma Intensity Research Score (AAIRS)

Donald H Arnold, MD, MPH

Pediatric Emergency Medicine

Center for Asthma Research

Vanderbilt University School of Medicine

Nashville, Tennessee, USA

Conflict of Interest Statement Speaker: Donald H Arnold

Supported by UL1 RR024975 (Vanderbilt CTSA), T32 GM07569 (O’Connor).

The speaker has no other competing interests or disclosures.

Background: Acute Asthma Exacerbations

• Most common reason for childhood hospitalization in North America

• Frequent reason for Emergency Department (ED) visits

• Variability of assessment that drives management and hospitalization decision

0

1

2

3

4

5

0 1 2 3 4 5

Treatment Tier Received

by Patient

Acute Asthma Exacerbation Severity

Treatment tier administered to 933 patients according to exacerbation severity. Bubbles correspond to number of patients. Blue line bisects bubbles that represent appropriate treatment tier for exacerbation severity.

O’Connor MG, Saville BR, Hartert TV, Arnold DH. Clin Pediatr. 53:1288. 2014.

Need for Valid and Reliable Exacerbation Severity Assessment

• Emergency Department exacerbation care

– Pulmonary function testing not available

– Multiple clinical team members and hand-offs

– Need for bedside assessment tools to guide treatment and hospitalization decision-making

Objectives

• To examine the inter-rater reliability between nurses, respiratory therapists, and attending clinicians for

– The 7 individual components of the Acute Asthma Intensity Research Score (AAIRS)

– The total AAIR Score

• Drives treatment and hospitalization decision-making

The Acute Asthma Intensity Research Score (AAIRS)

• Patient characteristics chosen a priori that

– Are easily measured and available at the bedside

– Encompass multiple domains of exacerbation severity

• Validated against FEV1 and airway resistance

1. Criterion validity

2. Responsiveness

Arnold DH, Saville BR, Wang W, Hartert TV. Ann Allergy Asthma Immunol. 2012;109(1):78. Arnold DH, O’Connor MG, Hartert TV. Ann Allergy Asthma Immunol. 2015;115(1):69.

The Acute Asthma Intensity Research Score (AAIRS)

Component Exacerbation severity Domain

Component point range Accessory Muscle Use

Sternocleidomastoid

Work of breathing

0 or 2

Intercostal 0 or 2

Subcostal 0 or 2

Air entry Airway obstruction, atelectasis 0 – 3

Wheezing Airway obstruction 0 – 3

SpO2 on room air V/Q mismatch, mucosal edema 0 – 2

Expiratory phase Airway obstruction 0 – 2

Possible total score range 0 - 16

Severity levels: mild 1-6; moderate 7-11; severe 12-16.

Methods • Team member comparisons

1. Attending clinicians vs. Respiratory therapists

2. Attending clinicians vs. Nurses

3. Nurses vs. Respiratory therapists

• Verbal instruction in AAIRS use

• Score patients 3-17 years with acute asthma exacerbations before treatment and 1-hr after albuterol + systemic corticosteroid

Methods: Analytic approach

• Individual AAIRS components: weighted kappa statistic

• Totals AAIR Score: Intra-class correlation coefficient

O’Connor MG, Berg K, Stack LB, Arnold DH. Ann Allergy Asthma Immunol. In press.

Characteristics of clinical team members scoring AAIRS components and of children with acute asthma exacerbations Number of participating clinical team members

Respiratory therapist 22

Nurse 50

Physician / Nurse practitioner 48

Patients with acute asthma exacerbations

Unique patients, number 144

Median age, years [IQR] 5 [3.3, 8]

Male gender 67%

Measures of inter-rater reliability for individual AAIRS components before treatment

Component Resp

Therapist vs. RN

Resp Therapist vs.

Attending

RN vs. Attending

Accessory muscle use

Sternocleidomastoid-supraclavicular 0.51 0.49 0.35

Intercostal 0.76 0.53 0.27

Subcostal 0.71 0.67 0.64

Air entry 0.22 0.30 0.24

Wheezing 0.35 0.46 0.16

SpO2 on room air 0.62 0.60 0.40

Expiratory phase prolongation 0.21 0.26 0.06

Values are weighted Kappa statistic. Shaded values are > 0.6 with 0.6 – 0.8 considered good and > 0.8 very good.

Measures of inter-rater reliability for individual AAIRS components after 1-hour of treatment

Component Resp

Therapist vs. RN

Resp Therapist vs.

Attending

RN vs. Attending

Accessory muscle use

Sternocleidomastoid-supraclavicular 0.63 0.70 0.81

Intercostal 0.63 0.53 0.42

Subcostal 0.64 0.48 0.42

Air entry 0.14 0.29 0.25

Wheezing 0.34 0.25 0.32

SpO2 on room air 0.69 0.36 0.49

Expiratory phase prolongation 0.35 0.34 0.35

Values are weighted Kappa statistic. Shaded values are > 0.6 with 0.6 – 0.8 considered good and > 0.8 very good.

Measures of inter-rater reliability for total AAIR score amongst clinical team members

Component

Resp Therapist vs. RN

Resp Therapist vs. Attending

RN vs. Attending

Pre-tx Post-tx Pre-tx Post-tx Pre-tx Post-tx

Intra-class correlation coefficient for total score

0.74 0.64 0.77 0.66 0.61 0.53

Values are intra-class correlation coefficient . Shaded values are > 0.6 with 0.6 – 0.8 considered good and > 0.8 very good.

Conclusions

• AAIRS inter-rater reliability amongst clinical team members caring for children with asthma exacerbations

– Poor for

• Some individual components

• Attending clinicians vs. other clinical team members

– Good for

• Accessory muscle use

• Overall AAIRS

Limitations

• Inter-rater reliability does not indicate validity of assessment

• External validity not assessed

Implications

1. AAIRS total score has good inter-rater reliability.

2. There is a need for formal education program to further standardize AAIRS component assessment amongst clinical team members.

Future Plans

• Intervention: AAIRS teaching video, available at https://vimeo.com/122443037

– Post-intervention assessment of inter-rater variability

Berg KT, O’Connor, Stack LB, Arnold DH. Acad Emerg Med. Epub Sept, 2015

Acknowledgements Vanderbilt Children’s Hospital Emergency Department

Respiratory Therapists, Nurses, Physicians / Nurse Practitioners

Co-investigators Michael G. O’Connor, MD; Pediatric Pulmonary Medicine Kathleen Berg, MD; Emergency Medicine; Lawrence B. Stack, MD; Emergency Medicine Biostatistician Li Wang, MS