Aligning InstituAligning InstituOSCEs with CLLISA DILLON PH D TSVETI MARKOV

Focus Areas LISA DILLON, PH.D, TSVETI MARKOVMARIA KOKAS, PH.D, S IMONE BRENNAN, M

utional utional LER VA MD FAAFP VA, MD, FAAFP, MA, PORCERELL I , JOHN, PH.D

Obj tiAddress institution-wide needs for r

ObjectivesAddress institution wide needs for r

Demonstrate adaptations to existinmeaningful data

Map OSCE cases with ACGME com

Develop OSCE faculty championsDevelop OSCE faculty champions

Provide feedback for residents, procontinuous improvementp

resident assessmentresident assessment

ng evaluation tools to yield

mpetencies and CLER focus areas

ograms, and the institution for

OSCE Data-Literature SOSCE Data-Literature SSearch Results1-7Search Results

NAS: Continuous improveement Cycle

Seeing CLER-lyg y

Patient Safety

QI and HeDisparit

Patient Safety

Transitions in Care

Duty Hand FatMitiga

Pr

ealth ies

Supervision

ours tigue tion

rofessionalism

Patient Safety

Profes

Patient Safety

Transitions in Transitions in Care

QI andDispa

I t l d CInterpersonal and C

ssionalism

Duty Hours and Fatigue Fatigue

Mitigation

Health arities

Supervision

i ti Skillommunication Skills

Capturing Interpersonal Cp g p

Non-spespecific

Direct

specific

Patient Safety

Transitions in Care

QI and Health

Disparities

Duty Hours and

Fatigue Mitigation

Supervision

Professionalism

Direct Observ

Mitigation

Interpersonal and Communication Skills

Multi-soassessm

OSCE Faculty Develop

Communication and Skil

ecialty cases cases

vation

ource ment

pment

Aligning OSCE with CLERGoal 1: Demonstrate adaptability

Aligning OSCE with CLER

• Recruit content matter exper• Simulation Center Director,• Learning Systems Director from HFH• GME Director of Research and Educ

• Analyze cases for CLER elem

G i i i l f• Generate training materials f

R of OSCE for CLER Process:R

rts

cation

ents

f f lfor faculty

OSCE Materials- Cas• Four non-specialty-specific ca

• Bad News E Di l • Error Disclosure

• Informed Consent • Surgical Disparity or Medical DispSurgical Disparity or Medical Disp

• One assessment of InterpersoOne assessment of Interperso• Kalamazoo Essential Elements Co

ses and Assessmentases:

parity in Healthcare parity in Healthcare

onal and Communication Skills: onal and Communication Skills: ommunication Checklist (KEECC)

alamazoo Essential Elementts Communication Checklis

MAPPIKEECCKEECC NG

CLERCLER

KEECC & CLER FOCUS AREA CODING AGREEMENTS

KEECC WITH ACGME COMPEETENCY CODING AGREEMEN

Terry Philips-Health CClinical Scenario:Terry Phillips is a middle aged ov

Terry Philips-Health C

Terry Phillips is a middle--‐aged, ovpatient. Terry is under a great deaa community health screening whblood pressure. Terry Phillips is in th

k h i l Th ti t twork physical. The patient recentscreening and was told they had

Today, the nurse took the patienty pelevated BP of 170/108.

Instructions:Please discuss with the patient themedication.

Do not perform any part of the PhDo not perform any part of the Ph

Care Disparity

verweight African American

Care Disparity

verweight, African American al of stress and recently attended here she was told she had high he outpatient clinic today for a l tt d d it h lth ly attended a community health HBP.

’s BP and patient again has p g

e need for taking HBP

hysical Examinationhysical Examination.

CULTY DEBRIEFING MATECULTY DEBRIEFING MATEERIALSERIALS

Results & Outcomes Ove

Resident level

Program levelg

I tit ti l lInstitution level

erview

Resident Level• KEECC Scores for eac

• Standardized Patient• Self-evaluation• Percentile Rank agai• Percentile Rank agai

Resident Program PGY BTrapper John Surgery 3 00Trapper, John Surgery 3.00

McDreamy, Derek Surgery 5.00

h caset

inst participantsinst participantsBad News Overall Percentile Rank

31 00 64 0031.00 64.0035.00 94.60

Program Level• KEECC averages and S• KEECC element averagg• Resources for program

B topic• By topic• By KEECC element

D by casege scoresgimprovement

Dermatology

KEECC Average

gy

31

27.8 27

3129.

Bad News Error Disclosure Informed Consent

Surgery Disp

Total number of participants:

N Residents who are 2 SD .6

N Residents who are 2 SD from case mean:

Below AboveBad News: 2 2

Error Disclosure: 3 2

Informed Consent: 3 2

parity

Surgery Disparity: 1 1

Dermatology—Average

Reaches Agreement

Provides closure

Understands the patient's perspective

Shares information

g

Opens the discussion

Gathers information 3

3.83.9

Builds relationship

score on each element of KEECC

4 04

4.13

4.21

4.04

4 08

3.96

4.21

4

4.25

4.08

44.1

4.24.3

ecommendations to proeview videos with residents (g

ecommendations to pro

Use as part of remediation plaUse as part of remediation plaesidents

Challenge the known OSCE ba

ogramsgroup/individual)

ograms

an for low scoring an for low scoring

arriers

Institutional Level• Overall program performan• Case Improvements by

Institutional Level

• Case Improvements by• Program• Case• Case and Year • PGYId tifi ti f f lt d• Identification of faculty devOSCE Retreat

• Data management (NI)Data management (NI)• Overall satisfaction• Enhancement of OSCE for 2Enhancement of OSCE for 2

nce

l t dvelopment needs

2014-20152014 2015

Overall Program Perfo

28 928.9

27.6 27.9 27.227.6

28.5

26.526

Dermatology Family Medicine Internal Medicine Orthopaedic Surgery

2012 DATA

ormance

29.8

27.6

26 0626 4 26 2

27.7

26.8

25.4626.0626.4 26.2

Otolaryngology Phys. Medicine & Rehabilitation

Transitional Year Urology

2013 DATA

Case Improvements

Bad News2012 Mean 2013 M

DermatologyInternalMedicine

25.32 28.89*

Orthopaedic Surgery

26.0 30.0†

Otolaryngologyy g gy

* p < .05† p < .001

by Program

Informed ConsentMean 2012 Mean 2013 Mean

24.33 26.67*

22.0 28.86*

2012

2012 & 2013 Institutional Com2012

Bad News 26.23

Error Disclosure 25.84

Informed Consent 27 36Informed Consent 27.36

Surgery Health Disparity 28.12

Medical Health Disparity 30.22

Case average 27 18Case average 27.18

PGY 1 Improvements2012 (N = 49) 0 ( 9)

Bad News 25.80

Error Disclosure 26.18

Informed Consent 27 75Informed Consent 27.75

Surgery Health Disparity 28.73

Medical Health Disparity 29.97

C 27 32Case average 27.32

2013

mparison201328.44**

24.76

28 87*28.87

27.11

29.85

27 70 * p < .0527.70** p < .001

2013 (N = 46)0 3 ( 6)

28.78**

24.69

29 4629.46

25.58

30.85

28 10* p < .05** < 00128.10 ** p < .001

OutcomesOutcomes

Program & faculty engage

Increased resident prepare

Higher rating of OSCE as a g g

ment

edness

learning toolg

New Directions for 2014• Institution-wide OSCE prep

New Directions for 2014

• Early commitment• Communicate CLER-lyy• Establish predictive validity• Redesign casesRedesign cases• OSHE

L it di l i• Longitudinal comparisons

4 2015paration

4-2015

y

Th k Thank ! you!

ReferencesReferences. Casey, P. M. , A. R. Goepfert , and E. L. Espey . To the pointructured Clinical Examination. Am J Obstet Gynecol 2009. 2

Simon, S. R. , A. Bui , S. Day , D. Berti , and K. Volkan . The reudents' OSCE scores and USMLE Step 2 scores. J Eval Clin Pr

Tervo, R. C. , E. Dimitrievich , A. L. Trujillo , K. Whittle , P. Redli i l E i ti (OSCE) i th li i l l k hi linical Examination (OSCE) in the clinical clerkship: an overv

Short, M. W., Jorgensen, J. E., Edwards, J. A., Blankenship, Rompetencies with an objective structured clinical examinat6.

Lypson, M. L. , J. G. Frohna , L. D. Gruppen , and J. O. Woodentifying the gaps. Acad Med 2004. 79 (6):564–570

Fuller R Homer M & Pell G (2013) Longitudinal interrela Fuller, R., Homer, M., & Pell, G. (2013). Longitudinal interrelamprovement and overall reliability. Medical teacher, 35(6), 5

Wallenstein, J., Heron, S., Santen, S., Shayne, P., & Ander, Dtructured Clinical Examination (OSCE) Can Predict Future Re

Medicine, 17(s2), S67-S71.

t: reviews in medical education–the Objective 200 (1):25–34.

elationship between second-year medical ract 2007. 13 (6):901–905.

inius , and L. Wellman . The Objective Structured i S D J M d 1997 50 (5) 153 156view. S D J Med 1997. 50 (5):153–156.

R. B., & Roth, B. J. (2009). Assessing intern core tion. Journal of graduate medical education, 1(1), 30-

olliscroft . Assessing residents' competencies at baseline:

ationships of OSCE station level analyses quality ationships of OSCE station level analyses, quality 515-517.

D. (2010). A Core Competency–based Objective esident Performance. Academic Emergency