National Simulation-based High-Stakes Assessment: Challenges and Lessons Learned
Amitai Ziv MD, MHADeputy Director, Chaim Sheba Medical CenterDirector, Israel Center for Medical SimulationProfessor and Chairman, Medical Education, Tel-Aviv University School of Medicine
Hospital Authority Convention
Hong-Kong, June 8, 2011
id17164265 pdfMachine by Broadgun Software - a great PDF writer! - a great PDF creator! - http://www.pdfmachine.com http://www.broadgun.com
AgendaBackground� Assessment in Medicine - Paradigm Change?
MSR - The Israel Center for Medical Simulation� Simulation-based (Training) & High-Stakes*
Assessment as a cultural change vehicle on a National level
Lessons learned� Challenges / Barriers / Insights
*High Stakes: with regulatory implications
Aviation and MedicineSimilarities� High-risk and high-tech industries� Low tolerance to errors� Intolerance to not learning from errors� Diverse skills / personality characteristics required
Differences� Admission and screening culture� Training and certification culture� Reporting and debriefing culture
Assessment & Safety Culture
Personal Simulation StoryIsraeli Air Force > 20 years� Simulation-based training / Instructor (of instructors�)
Hadassah / Jerusalem � Med. School & Pediatric Residency
� CHOP / Philadelphia � Adolescent Medicine / Med Education
Simulated Patients (SPs) � Consultant, ECFMG�s CSA Programs (USMLE CS)- SP-based Test Center, Philadelphia
Medical Director - MedSim� MedSim � TJU Medical Simulation Center
Sheba Medical Center, Deputy Director� Director, Risk Management, QA, Medical Education � Patient Safety� Founder & Director, Israel Center for Medical Simulation (MSR)
Chairman Med. Education � TAU Medical School
The PremiseCurrent Patient Safety & Quality Care Reality is Suboptimal
The Underlying Assumption/Hypothesis is:� Medical Education and Medical Assessment has a share in
this reality� A change in Medical Education and in its Assessment
Paradigm could improve this Reality
Simulation-based Assessment has the power to serve as a cultural change vehicle leading towards:� More effective / safe / ethical medical education � Improved Readiness / Preparedness of Health Professionals� Improved Accountability of the Health Profession at large
The (assessment) Swiss Cheese Model
Sub-optimal ��Training & Formative Assessment
Competency �Assessment & Licensing Paradigm
Inadequate Screening & Selection Process
Recertification / Maintenance of Competency
The Flexner Report (1910)
�Birth of Modern Medical Education�Set Standards � admission, curriculum, duration�
Unchanged Educational Paradigm - 100 Years
Pre-Clinical Phase
Basic Scientific Knowledge
Traditional Educational Tools (lectures)�
Clinical Phase
Basic Clinical Skills
Apprentice-based Learning (bed-side)
Missing Components:Objective Standards for Proficiency Assessment
Why Simulation?Safe environment - mistake forgiving� Error driven education
Proactive and controlled training� Nightmare driven education / Just in Time
Trainee/ Team / System centered education� Experiential / Emotional learning - IPE
Feedback and debriefing-based education� Reflective learners / Process-based education
Reproducible, standardized, objective (CSA)
�Assessment driven education
McNamara�s FallacyUS Secretary of Defense (1961-1968)
The first step is to measure whatever can be easily measured.
This is OK as far as it goes.The second step is to disregard that which can't easily be measured or to give it an arbitrary quantitative value.
This is artificial and misleading.
McNamara�s Fallacy (cont.)
The third step is to presume that what can't be measured easily really isn't important.
This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist.
This is suicide.
McNamara�s Fallacy
The Consequence�We start out with the aim of making the important measurable, and end up making only the measurable important.
The Empty Raincoat (Charles Handy, 1994 p219)
Competent knowledge base
Technical skills
Non-technical skills Safety Skills
Professional Behaviour
Professionalism - Hierarchy��
Higher Order Competencies / Skills
Safety Skills (Error = Unasked Question)� Handover, Adherence to Guidelines, Error Recovery,
Calling for Help, Documentation, EMR: Doc/Pat/Comp Skills
Team Work Skills / �Microsystems Skills�� Leadership, Followership, Communication, Counseling,
IPE: Inter-professional Skills
Multi-Cultural Skills� Cultural, Use of Interpreter, Language Skills / Geriatric Care
Reflective Skills� Debriefing, Self assessment, Feedback Provision Skills
Personal Traits� Integrity, Motivation, Capacity, Humility, Risk Taking Traits
Foundational simulation-based high-stakes assessments (sample)
Evaluation and InterpretationFor certification/employment - Mandatory Completion of life support courses (70s-80s)� BLS, ACLS, ATLS, PALS, NRP, ERC etc.� 3rd generation simulators now incorporated into courses
For graduate-level licensure � Clinical Skills Assessments (CSA - OSCEs)� MCCQE Part 2 (1992) ; ECFMG�s CSA (1998) ; NBME�s CS (2004)
� Hong Kong (2005), Japan (2005), South Korea (2008), etc.� UK - Public Health Examination (2006), Netherland etc.
For selection / screening for medical school� OSCEs for Non-cognitive attributes (empathy, maturity etc.)
MMI (McMaster inspired�), MOR (Israel) etc�
Little published in other healthcare professions:� Nursing � increase in mandatory sim-based curriculum� Dentistry � low fidelity simulation for selection / graduation� Applied Health � Institutional high-stakes (e.g. Michener�s
MMI, Sim-based readiness for clinical)
Recent Advances� Incorporation of advanced simulators into OSCE stations
(e.g. Harvey station in the IM RC Certification in Canada)� Surgery - Simulation being incorporated into Assessment
and Mastery Surgical Requirements (e.g. FLS, National Skills Curricula)
� Anesthesiology (US) - Use of simulation for part 4 MOC �Maintenance of Certification
Foundational simulation-based high-stakes assessments (Cont.)
The Good News - Driving Forces of Simulation-based Assessment
Patient safety movement� Growing public demand for safer health care � Recognition of link between safety and simulation
Accountability of Medical Education (BMJ 2009)
� Readiness Concept / Competency-based education� MOC - Maintenance of Certification (re-certification) a growing
movement looking beyond just knowledge assessment
Accreditation bodies / professional boards� Performance Assessment
Global migration of health professionals� Proficiency Gate-keeping needs
Simulation industry � more mature / secure � Built-in proficiency Metrics (often- not validated�)
Growing collaboration (in Training & Assessment) � Low-Tech (High Touch) - OSCE and High-Tech
Trends
14th OTTAWA CONFERENCE ON THE ASSESSMENT OF COMPETENCE IN MEDICINE AND THE HEALTHCARE PROFESSIONS
5 Theme Groups (Consensus Papers)Medical Teacher, Mar 2011, Vol. 33, No. 3
Criteria for a good assessmentTechnology-enabled assessment of health professions education (including simulation) Performance assessment Assessment of professionalismAssessment for selection for the healthcare professions and specialty trainingResearch in assessment
Other ThemesWork place assessment & Portfolio-based AssessmentAssessment Centers & Multimodality-based Assessment
The Utstein Meeting � Nov, 2010
Sim Healthcare Jr, June 2011, Vol. 6, No. 3
SBME as Translational Science
T1 science �laboratory discoveryclinical research
T2 science (clinical effectiveness) �comparative impact translated into practice guidelines
T3 science � health care delivery systems measureable impact in health of individuals and society
Dougherty & Conway: JAMA 299:2319, 2008
Contributions of medical education interventions to T1, T2 and T3 outcomes
T1 T2 T3
Increased or improved
Knowledge, skill, attitudes, and professionalism
Patient care practices Patient outcomes
Target Individuals and teams Individuals and teams
Individuals and public health
Setting Simulation lab Clinic and bedside Clinic and communityMcGaghie: Science Translational Medicine 17 February 2010 Vol 2 Issue 19
Disruption:Disruption:
ResearchResearch
Group 1 (0-1 CL) Group 2 (2-49 CL) Group 3 (>50 CL)
Composite score stratification by experience level
P < 0.01
Com
posi
te
Sco
re
(N=3) (N=17) (N=45)(N=15) (N=14)
Unpublished data, Dong & Dunn
Skill Acquisition CurveImpact of Zero-Risk Training
CP1345275-1
A
B
CD
Clinical competence
Safety standard
Metr
ic a
ssess
men
t (e
.g.,
co
mp
osi
te
sco
re)
Time
Traditional traininga
b
Simulation-based training
Experiential reinforcement
Dong et al: Chest 2010
Simulation-based Assessment and the
Regulation of Healthcare Professionals
Eric S. Holmboe, Ajit Sachdeva; Morton, Mary Rizzolo, Amitai Ziv
.�Soon to be published
The Simulation Summit Jan, 2011
SBME - Applications
�Hands-On� Skills Training (Basic & Advanced)
Simulation-Based CMEEnhancement of Cognitive Learning
Teamwork and �Human Factor� TrainingPerformance Assessment of Individuals and Teams� Readiness for Practice / Formative / Summative
Patient Safety / Risk Management / QAStructured Exposure to New Medical Technologies
Screening, Licensure, Certification
MSR � The Israeli VisionNational resource for comprehensive
inter-disciplinary, multimodality medical simulation center
dedicated to:
Patient safety and quality care
Hands-on trainingReadiness to clinical practiceHigh-Stakes Performance Assessment
Cultural Change Vehicle
MSR: Virtual Medical Environment (>2000 Sqm)
Broad-spectrum simulation modalities � Simulated patient section� High-tech simulators � Task trainers / Skills lab
Clinical environments � Home, Field, ER, OR, clinic� Customized - EMR
Debriefing capabilities� One-way mirrors� Digital A-V equipment � Debriefing software
Multidisciplinary staff
Guidelines and PrinciplesNational exposure / collaboration� Identify national needs � high impact factor
� Involve regulators (IMA, MOH, IDF, HMO) � certification Non for profit - fee for service � operationally balanced� Financial stability (autonomy) � Business Model� Long term contracts � ongoing budget items� Partnership with industry / foundations / grants / donations
Focus on debriefing / assessment� �Train the Trainer / Rater� - instructors / raters� courses� Bottom-up & top-down approach
Assessment - Expertise in testing and evaluation� Strategic partnership with the �Israeli ETS�� NITE
MSR � 9 Years of Activity> 100,000 Trainees / Examinees ( >30% as teams)� >150 SPs / > 100 simulators / models� Dedicated Multidisciplinary staff (>50 = >30 FTEs)
Multiple Mandatory (Formative Training) Programs� In collaboration with prof. assoc., HMOs, IDF, RM, MOH�� E.g. - Interns � transition into hospitals (Driven by Job Analysis)
� 5 d �nightmare course� > 800 interns annually (> 4000 tot)� Multiple �Top-Down� soft skills - e.g. apology, disclosing
errors programs for senior management (safety/RM officers)
Mandatory �Train the Trainer / Rater� Policy� > 1500 trained Trainers� > 2500 trained Raters
National Change Agents
National Simulation-Based High-Stakes Assessment at MSR � In Collaboration with NITE
Anesthesiology board exams (since 2003)� > 70 examinees annually� In collaboration with the Israel Board of Anesthesia
Paramedics certification exams (since 2004)� >150 examinees annually� In collaboration with the Israel EMS (Magen David Adom)
MOR - Screening of medical school candidates (since 2005) � > 900 examinees annually� In collaboration with Technion, HU, TAU Medical/Dental Schools
Advanced nursing licensing exams (since 2008)
� >1000 examinees annually (15 different clinical domains)� In collaboration with the Israel MOH Nursing Authority
Initial Decision taken by the IBATesting Committee - 2003- 2003Simulation modalities to create realistic clinical scenarios
Complementary to existing traditional Exams (MCQs/Orals)
Task Driven Test - Minimal requirements in 5 domains:
� Trauma management
� Resuscitation
� Crisis management in the operating room
� Regional anesthesia
� Intensive care medicine
SBT � Blueprinting Development
Breathing Management
Management of Arrhythmia
Airway management
Chest drain insertion
Chest X-Ray Interpretation
ECG Analysis
Clinical conditions/Tasks
+++-++Resuscitation
+-+++-Trauma management
SBT � Tasks and Scoring Definitions
Tasks - sample
� Chest X-Ray interpretation
� Arterial blood gasses analysis
� Chest drain insertion
� Management of arrhythmias
Scoring - Checklist
� Total Checklist score / Critical items
� General holistic score
SBT - Preparation
Candidates Orientation
� List of tasks sent to examinees
� Standardized and structured exposure to the examination
environment, the simulation devices, and scenarios
� Frequent errors during test periods reported nationally
Examiners -"Train the Raters�� Standardized and structured mandatory exposure to the
examination environment, simulation devices, scenarios, assessment tools
� Additional �refresher� on the morning of the exam
�Evolution� of the Examination
2003 � 2004 - very cautious launch
� Gradual incorporation of the SBT scores as a component of the
Oral Board Exams resulting in SBT becoming a prerequisite for
applying to the oral boards
Since 2004 � Ongoing improvements in Exam
� Upgrades of Examiners� TTR workshops
� Increased Realism of the Exam � e.g. - �Standardized Nurses� ,
improved simulators and setting
� Additional clinical domains � e.g. - Ped & Neonatal Anesthesia
Since 2009
� Full scale scenario followed by debriefing by Raters
Anesthesiology Board ExamsSummary & Insights: 2003 - 2010
#s: 15 exams thus far � 75-80% ave. pass rate
> 500 examinees � (10 Independent Scorings for each)
> 100 Trained RatersInfluence on Anesth� Oral Board exam transformed into a much more structured exam � Train the Rater Workshop for Oral Board Ratings� Workshops for Test Case Writing at large � including Orals, MCQs
Research & Publications� H Berkenstadt, A Ziv et al, Anesth Analg, 2006� H Berkenstadt, A Ziv et al, IMAJ, 2006� E Ben-Menachem, H Berkenstadt, Anesth Analg, 2011� More�
Total # of Registered Israeli Anesthesiologists < 1000
3 Paramedic Schools in Israel
Graduate the course successfully
Practicing and tutoring in The intensive care ambulance units
Final written exam
Internal �Mega code�based exam
Paramedics in Israel
Licensing: OSCE at MSRSince 2004
Simulation based training day at MSR (1 month before)‾ Acquaintance with the simulators / SPs��/ exam structure ‾ 6 simulation based stations � hands on practice (Formative Training)
Train the Raters (examiners)‾ Acquaintance with the simulators / SPs��/ exam structure / tations ‾ Training and practicing the rating scales and criteria
The exam day‾ Each student is tested on 4 �hybrid� stations / Team of 3 perform‾ The leader is the examinee / teams� members are paramedic students‾ 4 stations - 15 minutes each
- Cardiology ; Trauma ; Pediatrics ; OB or Respiratory disorders� 2 Independent Trained Raters at each station � Evaluation � combination of check lists / Holistic evaluation
Structured Exam � Similar Model
#s - 2004-2010� # Examinees > 730� # Trained Raters > 180
� Exposure of >50% of Registered Paramedics in Israel
Summary & Feedback
Total # of Registered Israeli Paramedics< 1800
� (good)���1(poor)Questions
82%18%1. How well were the goals of the exam clarified?
65%31%4%2. How much were the scenarios like real cases in the�EMS field? � ��
69%27%4%3. The exam was fair
40%47%13%4. Evaluate the difficulty of the exam
45%47%4%4%5. How much did the orientation of the exam help you?
Examinees� Perspective
Would you recommend the use of simulators as a future component
in EMS exam?
Desired Personal Attributes (MOR)
Service awarenessHonestyEthical attitude
IntegrityEmpathyAttitude towards authority
Self confidenceSensitivitySelf awareness
MotivationResponsibilityMaturity
Inter-personal communication skills
The ability to identify a need for help, search for help and accept help
The ability to function under stress
��������OpennessSelf confidence
Structure of Assessment Center (MOR)Non-Cognitive Screening for Admission to Med Schools
SimulationsBiographical Questionnaire
Judgment and Decision-Making
Communication skills
Handling of stress
Initiative & responsibility
Consciousness & self-awareness
21 Questions 3 Dilemmas
120 minutes 90 minutes 45 minutes
9 Behavioral stations
Essay questions related to candidate�s past experiences
Short descriptions of dilemmas that require candidate to make decisions
Behavioral Stations � Examples
ExamplesStation Type
Problematic request. You are a family doctor. A patient asks you to give her a letter for work saying she needs to rest because of back pain, and she wants to go away on vacation �to rest and get better�
A challenging encounter with an actorSimulation
�Describe the situation��What would you have done differently?�
A structured interview regarding the candidate�s performance in the simulation
Debriefing
�Why do you want to be a doctor?��What is your opinion on
abortions?�Interview
1.Divide a budget among 3 different sectors in a hospital2. A devoted son has been taking good care of his mother. But, he is abusive to the staff.
Group
Description
A structured mini-interview
Groups of 3 or 6 candidates perform a task together
Score Calculation
Combination of 3 Scores� based on agreed-upon weights of the 3
independent scores:� Behavioral stations (60%)� Judgment / decision-making quest.(20%)� Biographical questionnaire (20%)
Alpha coefficients of MOR
Cronbach's alpha coefficients
Assessment Center 2006 2007 2008 2009Mean across years
MOR (Behavioral stations)
0.73 0.76 0.79 0.77 0.76
MOR (Full Assessment Center)
0.83 0.83 0.84 0.82 0.83
Cronbach Alpha coefficients, indicating internal consistency, ranges from 0.82 to 0.84
Inter-rater reliability, ranges from 0.62 to 0.95 for the different components
Test-retest scores correlation for 405 candidates who participated in successive years is 0.72
Overall (MOR) Reliability
Predictive validity � to be evaluated in an up-coming longitudinal research�� Good predictive data from McMaster�
Construct Validity - Correlation between the assessment center scores and candidates' cognitive scores approached zero
Face Validity - Feedback questionnaires from candidates and raters indicate that the assessment center is perceived as highly fair and appropriate for the screening to medical school
Results � Validity
The make-up of the student body changed by 20% each year (stable)Dramatic change in atmosphere at the facultyVery high national (and international) interest - two other faculties joined the process in 2006/7� Publication: A Ziv, O Rubin, et al , Medical Education; 2008
#s (2004-2010)� > 4300 candidates� > 1500 Faculty trained as Raters
Moral message to candidates, faculty & public
Consequences
Background - Nursing
In Nursing Education (Globally�) � SBME is common� Certification and licensing are based almost
exclusively on knowledge-based written exams
In Israel � to become a certified nurse specialist (ICU, EM, diabetes etc.) one must undertake a year-long specialty course and a test. � 2008 - In recognition of the need to add performance
measures to the registration process, the Israeli MOH Nursing Authority has decided that an SBT will be developed to �partially replace� the traditional MCQ.
MINISTRY OF HEALTHJERUSALEM
Test Structure & 2008-2011 #s
Test materials - Developed based on pre-defined test blueprints for each specialty by teams of NS, sim-experts and psychometricians
15 Clinical Domains = 15 Different ExaminationsAcute Care - ICCU, PICU, OR, ER etc.Chronic Care � Psychiatry. Geriatrics, Nephrology, etc.
11 stations (mix) - in each Exam� Simulator Stations, SP stations, Debriefing stations,� Video/PPT case-based analysis, Short MCQ test
N (Examinees - total in 3 years >2300)� 2009 � 632 nurses in 13 areas of expertise� 2010 � 746 nurses in 14 areas of expertise� 2011 > 1020 nurses in 15 areas of expertise
N (Raters) > 600MINISTRY OF HEALTHJERUSALEM
<60,000RNs inIsrael
Psychometrics (overview) � 2009-1052 testing days in two years (up to 44 examinees per day) Passing Score 60 (�Standard Setting�)
MINISTRY OF HEALTHJERUSALEM
MaxMinStd DevMeanN2009
92497.074632Total
85565.771308Acute care
92497.376272Chronic care
MaxMinStd DevMeanN2010
������������Total
������������������Acute Care
������������Chronic Care
������������������Midwifes
Reliability � Internal Consistency
Cronbach Alpha - for tests with large numbers of examinees:� Incorporated across different days (test versions)
MINISTRY OF HEALTHJERUSALEM
20102009
0.80N=163
0.58N=161
Intensive care
0.68N=78
0.58N=52
Midwifery
0.70N=38
0.72N=52
Primary care in the community
Reliability � Rater Agreement
Percent of raters disagreement
MINISTRY OF HEALTHJERUSALEM
20102009Station Type
4.64.7Simulator station (yes/no items)
30.425.3SP stations (1-4 scale)1 point diff
2.41.5SP stations (1-4 scale)2 point diff
National Impact of High-Stakes SBT
Penetration� Vertical: MS Candidates / Anesthesiology / Faculty Raters� Horizontal: Specialty RNs / Paramedics / Faculty Raters
Large proportion of each sector is involved in the assessment process (full cohorts of new generations)� Anesthesiology > 50% (including faculty) � Paramedics > 50% (including faculty) � MS national candidates > 80% � MS Faculty for admission > 50%
Significant increase in SBME in those professions� Forces revisions in educational curricula � Motivates schools to incorporate institutional SBTs
National Level SBT - Lessons Learned
Needs to be an Explicit Goal of SB educators� From Formative Assessment to Summative Testing and back
Requires on going Courageous (Political) Leadership on behalf of the professional / regulatory boards� Readiness to fight for the cause / defend the move in public
in the profession....Probably unfounded legal concerns�Dive into the water � don�t wait for the �ultimate� exam� Work-in-progress: Ongoing Improvement Process/Evolution� Improve in Small Increments � Maintain Structure
SBT as a Complementary Exam to the traditional ones� Not a Replacement
Lessons Learned (cont.)
Feedback loop back to the training programs on Identified Skills� Deficiencies� Revisions in Curriculum towards more competency-based
training & more focus on deficient skills� Raters� / Faculty more aware of the suboptimal �Product
Line� - Enhances Accountability
Collaboration between Content Experts / Simulation Experts and Measurement Experts (Psychometricians) - Crucial for Success� Emphasis on Validity � Content, Construct, etc� Aim at High Reliability (secondary in priority�)
� Given logistical / costs / �N� limitations
Large Scale (National) SBT - Insights
Huge investment on behalf of the all parties involved� Test Development Teams - Ongoing Commitment� Tough on sim-center personnel - Not as much fun�� 25% of MSR�s Activity (>60 days a year) in SBT
High-Cost ($s, Manpower, Resources�) � Top-down budget Vs Bottom-up funding� Fees diverted to examinees (MOR�)
Raters recruitment & Ongoing Commitment� Retention - Pay Raters?� Aim to establish High Reputation (Prestige) to rating mission
Risks �SBME may be perceived as a threat�� Balance between SB training & Testing
Research � Next Steps
Investigate psychometric qualities � Including predictability of Simulation-Based Testing
Impact of high-stakes exams on the transferability of simulation-based assessment to practice� Patient safety / curriculum / faculty / motivation
Improve assessment metrics & tools� Measuring the �un-measurable�/ �the important�� higher order competencies / skills (professionalism)� �Professionalism� of the Assessment
Standard Setting� Where & How to set the bar
*2 PhD students at MSR on SBT (Nursing & Dentistry)
Research � Next Steps
Measurement issues - individuals within group� Assessment of isolated skills Vs integrated skills� Assessment in different contexts
Role of simulation in multi-modal assessments� Simulation is only one of many tools�.
Role of simulation in maintenance of certification and licensure and Re-entry of health professionalsGeneralizability of SBT across disciplines & cultures
� International Collaboration re-assessment models� Cross fertilization across professions
BUT � Keep in mind that �Simulated Exams� won�t answer many of the research questions* (HB 1st Prize at IMSH)
High-Stakes SBT � Challenges
Incorporation of Simulator Driven Metrics� Need to develop / improve metrics � validity / reliability � Opportunity: to free-up human rating for high-order skills
Convincing other Professional Boards / Regulators� Moving into SBT requires a mental �Leap frog�
transformational / strategic / Value decision
Never forget the reference point �the absence of performance assessment in current
testing paradigm
Conclusion
OSCE took 30-40 years to mature enough for acceptance in high stakes assessmentSociety will not wait another 30 years for us to apply High-Stakes Performance Assessment and Simulation-based Testing�Across professional disciplines�At more advanced levels of practice�As part of lifelong professional development�As part of health care safety/quality culture
And so�
And learn as we go�
Contact Info
Amitai Ziv, MD, [email protected]
MSR � The Israel Center for Medical Simulation
www.msr.org.il
