Robert Dunfield1
Jaime Riley2
Chris Vaillancourt3
James French4
Jacqueline Fraser5
Paul Atkinson6
Jennifer Woodland7
1. Dalhousie Medicine New Brunswick, Saint John, New Brunswick
2. Department of Emergency Medicine, Saint John Regional Hospital, Saint John, New Brunswick
3. Department of Emergency Medicine, Dalhousie University, Saint John Regional Hospital, Saint John, New Brunswick
4. Department of Emergency Medicine, Dalhousie University, Saint John Regional Hospital, Saint John, New Brunswick
5. Department of Emergency Medicine, Saint John Regional Hospital, Saint John, New Brunswick
6. Department of Emergency Medicine, Dalhousie University, Saint John Regional Hospital, Saint John, New Brunswick
7. Department of Research Services, Horizon Health, Saint John Regional Hospital, Saint John, New Brunswick
References1. Greenberger PA, Rotskoff BD, Lifschltz B. Fatal anaphylaxis: postmortem findings and
associated comorbid diseases. AAAI 2007; 98:252-7.
2. Song TT, Worm M, Lieberman P. Anaphylaxis treatment: Current barriers to adrenaline auto-injector use. Allergy 2014; 69:983-991.
3. Simons FER, World Allergy Organization. Epinephrine auto-injectors: first-aid treatment still out of reach for many at risk of anaphylaxis in the community. AAAI 2009; 102:403-409.
4. Ahmed A, Khan FA, Ismail S. Reliability and validity of a tool to assess airway management skills in anesthesia trainees. J Anaesthesiol Clin Pharmacol. 2016 Jul-Sep;32(3):333-8.
ContactEmail: [email protected]
Cell: 506-434-3289
Conflicts of interestNo conflicts of interest.
Results:
• Overall, there was poor agreement between the two raters.
• Checklist item #1 (contacted EMS) at 0 months had the highest level of
agreement, but did not achieve statistical significance (κ=0.59, moderate
agreement; p<0.05) (Table 3).
• Checklist item #1 at 3 months had the second highest level of agreement (κ=0.482, moderate agreement; p<0.05) (Table 3).
• The remaining checklist items had levels of agreement ranging from poor to slight agreement. None achieved the standard level of agreement of κ ≥ 0.70 (Table 3).
Conclusion: Although microskills checklists have been shown to identify areas where learners and interprofessional teams require deliberate practice, our results support previously published evidence that the use of microskills checklists in the assessment of skills has
poor reproducibility4. Performance will be further assessed in this study using global rating scales, which have shown higher levels of agreement in other studies.
Acknowledgements Sweet Caroline Foundation; Town of Hampton, NB; Dalhousie Emergency Medicine Research Committee.
The devil may not be in the detail: assessment of bystander training to administer publicly available epinephrine using microskills checklists shows low inter-rater reliability.
sjrhem.ca
Methods:
• Prospective, stratified, block randomized study
• 154 participants at 15 sites were block randomized to one of three education interventions:
1. didactic poster teaching; 2. poster with video teaching; and 3. poster, video, and simulation training
• Participants tested at 0-months and at 3-months (post-intervention).
– Assessment involved two blinded raters using a microskills checklist.
– Checklists used fourteen 3-point and 5-point Likert scale questions around anaphylaxis response (Table 1).
– Inter-observer reliability was assessed for each item and expressed as a kappa (κ) value that represented a level of agreement (Table 2).
Background:
• Improving public access and training for epinephrine auto-injectors (EAIs) can reduce time to initial treatment in anaphylaxis and therefore save lives, especially in rural settings.1,2,3
• In southern New Brunswick, this is being addressed with unlocked, alarmed EAI cabinets.
• We investigated the best teaching modality for effective EAI training using a microskillschecklist.
Table 1: Anaphylaxis response microskills checklist
Table 2: Reference κ values for inter-rater agreement.
1. 911/EMS contacted2. Patient positioned properly3. Cabinet properly accessed
4. Both EpiPen’s removed5. Properly handled EpiPen and removed safety guard
6. Correct dosage injected7. EpiPen injected into upper lateral thigh
8. EpiPen held in place for 10 seconds9. Patient encouraged to remain resting post-injection
10. Second dose considered11. Patient monitored until “EMS” arrived
12. Calm demeanour throughout their response13. Clear and effective communication14. Responded in an organized fashion
kappa value (κ) Level of agreement
<0 Poor agreement
0.0-0.20 Slight agreement
0.21-0.40 Fair agreement
0.41-0.60 Moderate agreement
0.61-0.80 Substantial agreement
0.81-1.00 Almost perfect agreement
Assessment Parameter & Measurement Time κ p value Agreement
1. Did the participant contact or instruct someone to contact
emergency medical services (EMS)?
0 months 0.590 <0.05 Moderate
3 months 0.482 <0.05 Moderate
7. Did the participant deliver the EpiPen in proper location (upper
lateral thigh)?
0 months 0.096 0.070 Slight
3 months 0.247 <0.05 Fair
12. Did the participant maintain a calm demeanour throughout the
anaphylactic event?
0 months 0.072 0.085 Slight
3 months -0.103 0.095 Poor
14. Did the participant show the ability to deliver the epinephrine in
an organized fashion?
0 months 0.065 0.112 Slight
3 months 0.067 0.311 Slight
Table 3: Levels of agreement for select checklist items
Generally, a κ ≥ 0.70 is preferred and set as the standard level of agreement.
Use of micro-skills checklists
in the assessment of skills has poor reproducibility
Link to all SJRHEM@CAEP 2019Research