Program Evaluation of

In Situ Simulation-based Team Training

Melissa Powell BSN, MSHCA, RN

I’m a novice to conducting research I’ve made a few mistakes along the way I’m eventually going to get there

Just getting started:This is an early pilot and investigation

Late 2009 and early 2010, multiple anecdotal stories of confusion and communication problems

Post Code Quality Survey Out of ICU/RRT debrief Committee meets

weekly. Remediation and debrief process by

educator in place for failure to initiate early rescue and RRS activation. Qualitative data collected after each meeting.

Needs assessment

1.       “He or she {physician} is right here at the bedside with me. And that feels very uncomfortable as though it says to the physician that I do not think they are capable.”

2.       “He or she went to medical school.  I didn’t.  I’m just a nervous new nurse.  They know what they are doing.”

3.        “I feel like I’m tattling on them.” 4.       “The doctor is responding to my concerns with orders.” 5.       “There is nothing anyone can do for this patient anyway.” 6.       “There is something not right about this patient but my vital signs are normal, I

just don’t want to call them over here and waste their time.” 7.       “There aren’t any critical care beds right now.  The ED is on diversion.  I know

they are busy.” 8.       “The {responder} nurses just roll their eyes at me and don’t do anything.” 9.       “The charge nurse thinks everything is okay so I don’t want to call and override

her.” 10.   “Multiple nurses at bedside:  “I didn’t call because my charge nurse was there and

there were others with me and felt I had enough different people involved”. 11.   “What else would RRT do, besides what we are already doing?” 12.   If the patient wasn’t transferred they think that it wasn’t the right thing to do to

call. 13.   If the patients code status is DNR/DNI they do not think they should call,

regardless of goals of care or team knowledge.

Qualitative data from nurses related to delay and failure to rescue

Authority gradient phenomenon Inability to express concern clearly Other staff discounting judgment Lack of organization support Issues of diminished culture of safety Responder communication poor Lack of knowledge System and process failure

Themes extracted from qualitative information

Ineffective communication is a root cause for nearly 66 percent of all sentinel events reported*• (The Joint Commission Root Causes and Percentages for

Sentinel Events (All Categories) January 1995−December 2005)

• Vanderbilt Innovation Pillar goal 2011: Innovate with new approaches to Interprofessional learning.

What is the problem? The driver?

Experiential learning theory -- (Kolb, 2000) Social Cognitive learning theory -- (Bandura,

1960) Interprofessional team training evidence --

Multiple studies (Salas, 2008) (Gaba, 2004) (Miller, 2008) (Small, 2008)

AHRQ TeamSTEPPS evidence (Salas, 2008) Diffusion through Innovations -- (Rogers,

1962)

Theoretical framework for developing the intervention

Will in situ team training improve team member perceptions of “chaos” during acute events?

Will in situ team training with TeamSTEPPS improve team member perceptions of “good” communication during acute events?

Will in situ team communication training with TeamSTEPPS curriculum decrease delay and failure to rescue?

Hypothesis Questions:

Program Evaluation using Moore’s Framework for determining Learning and Assessment

Level 1 • Participation

Level 2• Satisfaction - Learners

reactions to the training

Level 3

• 3a Declarative Knowledge - Modifications to knowledge, attitudes, perceptions

• 3b Procedural Knowledge - Acquisition of procedural knowledge and skills

Level 4• Competence - Demonstration

of skill in training

Level 5

• Performance - Changes in workplace behavior, competencies and attitudes

Level 6

• Patient Health – Changes in patient health or workplace systems as a result

Level 7 •Community Health – Changes in the health of a community

Unit determined by the following VUH standard:

Required Bi-annual AHA Training for all staff delivered by VRP. (Policy 30-1.24)

Education regarding adherence to Rapid Response System process (Policy 32.89.0) received in orientation.

Required Bi-annual “Mock Code” simulation in “Hands on Clinical Safety”.

All nurses debriefed about failure to rescue.

Description of control group

Convenience sample Required Bi-annual AHA Training for all staff.

(Policy 30-1.24) Education regarding adherence to Rapid

Response System process (Policy 32.89.0) Blended Learning Intervention:

◦ Online education regarding teamwork and communication

◦ On unit setting with interprofessional team ◦ Briefing before simulation◦ High fidelity Interprofessional simulation of patient

decline and cardiopulmonary arrest requiring early activation of rapid response system and BLS

◦ Debrief using Rudolph’s “non-judgmental” process

Description of Intervention group

Post code data & # failure to rescue events

Intervention group (100% unit) Level 1

Online education (Level 3a)

Simulation (Level 3b, Level 4)

Debrief (Level 4)

Survey (Level 2)

Real event (Level 5)

Post

code

quality

survey

(Level 5) &#

Failure to rescue events

(Level 6)

Program evaluation using Moore’s Method

Participation was mandatory for intervention unit. 100% completion.

Most staff was excited to have an opportunity to be a part of simulation training and signed up early for sim times.

Few staff had anxiety. Reassurance and encouragement was necessary for a few staff members.

Satisfaction: Initial paper survey response rate very highly rated simulation experience: 63 surveys received: 92% rated unit based training with simulation “very satisfactory” as a learning method.

Participation and Satisfaction:Level 1 and Level 2

Post training simulation survey

Strongly agree strongly disagree

Post training survey

Excellent poor

All complete AED Heartsaver or BLS course Performance of CPR Pass a test on AHA guidelines for rescue

All staff are communicated to about RRS All new hires are verified that they were informed

and read policy about RRS. Signs in all patient rooms and staff break room Cultural awareness No formal test of knowledge

Declarative and Procedural Knowledge Evaluation (Level 3a &b)

BLS skills (80%) Procedural knowledge (50%) Equipment knowledge (unable to assess during due

to SimMan3G barriers) Assessment and Intervention (99%) Recognizing and Initiating RRT immediately (55%) Communication skills

◦ SBAR (20%)◦ CUS (.1%)◦ Check Back (34%)◦ Two-Challenge Rule (0%)◦ Situation Monitoring including Shared Mental Model

knowledge (34%)

Simulation Checklist (Level 4)

***This data is based on video observation and very rough and unable to validate fully.

Post training survey for commitment to change practice

Strongly agree strongly disagree

Post RRT and Code event data obtained from team members obtained through convenience and ability to discover who would voluntarily complete online survey.

No systematic process in place currently to identify and contact all or some members of an acute event.

The sample frame or the population size is unknown and varies event to event.

Level 5 Performance

Pre July 2010

Post InterventionDec 2010

difference Post interventionConfidence Intervals

Intervention Unit

8% 8.3% 0.3% +/- 8%

Control Unit 9% 9.5% 0.5% +/- 8%

Post Acute Event Data Analysis % Team member perceptions of “good communication” “strongly agreed”

With 90% confidence 8% of the participants in the survey sample answered “strongly agreed," we could be sure that between 1% and 18% of the members of the entire target population would also say “strongly agreed" when asked the same question. The confidence interval, in this case,is +/- 8%.

My sample is not random so my ability to report is not accurate.

Pre July 2010

Post Dec 2010

difference Confidence Intervals

Intervention Unit

20% 0% 20% +/- 6%

Control Unit 19% 4.8% 14.2% +/- 6%

Post Acute Event Data Analysis % Team member perceptions of “chaotic” that “strongly agreed”

Interviewed 6 team leaders of STATS obtained through ability to identify and meet.

Questions asked: ◦ Tell me how the code went that occurred on (unit)?◦ How was the communication?◦ What could the team have done better?◦ Did you all debrief afterward?

Unable to record conversations completely. Some situations were random. Information regarding team work was consistently very satisfactory in regards to teamwork on intervention unit. Additionally, this approach yielded the highest quality feedback.

Attempted to triangulate through interview

Data obtained from Out of ICU meetings and review of events

There were fewer delay and failure to rescue.

The behavior is still occurring.

Level 6 Patient Health

Jan 2010 to July 2010

Jan 2011 to July 2011

Intervention Unit

23 6

Control Unit 18 12

Out of ICU Committee Data Analysis Number of delay and failure to rescue

Delay to rescue is determined by patient meeting a rapid response trigger criteria for more than 60 minutes.Failure to rescue is determined by patient declining over more than 6

hours before RRS activated.

Debriefing nurses post intervention yield stronger self reflective ability and that learning occurred

Using concepts such as authority gradient to communicate what they experienced

Staff now often identifies their own barriers and role in early activation and failure to rescue

Level 5 Performance

Did not continue qualitative data collection of debriefs

In Situ Simulation Team Training is effective. I recommend Vanderbilt leaders take a look at equipping unit educators with the skills.

Communication skills are a hard skill. Best methodology is not a power point but simulation.

Culture of safety is important and must take multiple angles to address.

To get to “zero” delay and failure to rescue, we must improve process, system, education and communication skills of providers.

Diffusion of Innovation is occurring

CONCLUSIONS

Researcher bias, confounded by known identity of intervention group

Training works versus no training. Team Training works with multiple levels of evidence for the last 10 years.

Teams that are trained in team communication rate themselves lower after training.

No triangulation of observed effects. No independent raters of performance.

Inability to train all educators to level of competence for intervention

Doesn’t inform beyond existing research literature. Must find measurable outcomes.

Limitations to generalization and publication of findings

Pilot idea, discussed in multiple meetings with leaders and experts in simulation team training.

Held initial simulations with team training experts and received training in event based scenario development, team observation, checklist development and rating, and debriefing skills.

Ran 68 simulations in a 10 month period. Videoed simulations and debriefs to improve and practice observation skills and debrief skills.

Evolution of investigation

July 2012, acquired TeamSTEPPS Master Trainer training and status through AHRQ.

Continue to refine process through improved coordination of departments and leadership buy in.

Created a temporary unit simulation space in a large equipment room.

Refined data collection methods and survey instrument. NO MORE SLIDER SCALES. LIKERT ONLY.

Begin collaboration with other units.

Evolution continues

Bandura. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, 52(1), 1. Clancy, C. M. (2007). TeamSTEPPS: assuring optimal teamwork in clinical settings. American journal of

medical quality, 22(3), 214. Gaba, D. M. (2004). The future vision of simulation in health care. Quality & Safety in Health Care, 13(suppl

1), i2. Holzman, R. S. (1995). Anesthesia crisis resource management: real-life simulation training in operating

room crises. Journal of clinical anesthesia, 7(8), 675. Kolb, D. A. (1983). Experiential Learning: Experience as the Source of Learning and Development (1st ed.).

Prentice Hall. Miller, K. K. (2008). In situ simulation: a method of experiential learning to promote safety and team

behavior. The Journal of perinatal & neonatal nursing, 22(2), 105. Moore, D. E., Green, J. S., & Gallis, H. A. (2009). Achieving desired results and improved outcomes:

Integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions, 29(1), 1–15. doi:10.1002/chp.20001

Nunnink, L. (2009). In situ simulation-based team training for post-cardiac surgical emergency chest reopen in the intensive care unit. Anaesthesia and intensive care, 37(1), 74–8.

Peberdy, M. A., Callaway, C. W., Neumar, R. W., Geocadin, R. G., Zimmerman, J. L., Donnino, M., Gabrielli, A., et al. (2010). Part 9: Post–Cardiac Arrest Care 2010 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation, 122(18 suppl 3), S768–S786. doi:10.1161/CIRCULATIONAHA.110.971002

Rogers, E. M. (1995). Diffusion of innovations. Free Press. Rudolph, Jenny, W. (2006). There’s No Such Thing as “Non-judgmental” Debriefing: A Theory and Method

for Debriefing with Good Judgement. Simulation in Healthcare, 49–55. Salas, E., DiazGranados, D., Klein, C., Burke, C. S., Stagl, K. C., Goodwin, G. F., & Halpin, S. M. (2008). Does

Team Training Improve Team Performance? A Meta-Analysis. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(6), 903–933. doi:10.1518/001872008X375009

Small, S. D. (2008). Demonstration of high‐fidelity simulation team training for emergency medicine. Academic emergency medicine, 6(4), 312.

Vanderbilt Policy Rapid Response Team Activation Policy number CL 30-08.16 Vanderbilt Policy Cardiopulmonary Resuscitation Policy number CL 30-08.21.

References