Transform your clinical competency committee with learner coaching

Matthew Kelleher, MD, MEd (@kelzj3)

Benjamin Kinnear, MD (@Midwest_MedPeds)

Eric Warm, MD (@CincyIM)

Date: 10/21/17

I do not have an affiliation (financial or otherwise) with a

pharmaceutical, medical device or communications organization.

Je n’ai aucune affiliation (financière ou autre) avec une

entreprise pharmaceutique, un fabricant d’appareils

médicaux ou un cabinet de communication.

2

Learning Objectives

By the end of this workshop, we hope you are able to:

1. Evaluate your own competency committee as operating under

a problem identification model or a developmental model.

2. Use coaching techniques with your learners.

3. Use your competency committee to guide coaching strategies

for learner improvement.

3

Think-pair-share

What does your CCC do to help residents:

• Improve

• Learn

• Grow

4

Observable Practice Activities (OPAs)

Process-Based (POPA)

Mid/End of Rotation Feedback (formative)

Entrustment (1-5)

Content-Based (COPA)

Reporting Milestones

External Reporting Longitudinal Assessment Formative > Summative

Mapping Mapping

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

Average of 3,987 milestone assessments, 4325 narrative words per resident!!!!

6

Our CCC

Mountain of milestone data

Month of Residency

AggregateEntrustment

Blue = Resident Performance

Red = Predicted Score

N= 3320Number of

Assessments

• Trends from start of residency to present• Comparison to the predicted score

Blue = Resident Performance

Red = Predicted Score

Milestones Assessed in past 6

months

• Snapshot of the past 6 months• Gives us an idea of what we will be reporting

to ACGME and ABIM

EntrustmentScale

• Z-score = number of standard deviations Observed (blue line) is from Predicted (red Line)

Blue = Resident Performance

Red = Predicted Score

X-Bar Chart

SD Chart

"AUC"= 3.06

Z-Score (plus 3)

Standard Deviation within

the Subgroup (Competencies)

“Area under the curve”

3rd Year Resident at 36 Months

N= N= "AUC"=2470 815 3.06

There’s more!!!

12

1

6

11

16

21

26360 Patient Care

360 Teamwork360 Professionalism

360 Efficiency

Patient Evaluations

Testing Overall

DM Overall

Diabetes Process

Diabetes Outcome

Diabetes Change

Prevention Quality

Prevention Change

Absolute Work UnitsWork Intensity

CitizenshipSelf Professionalism

Self Patient Care

Self Patient Care

Self Systems Based…

Self Systems Based…

Self Systems Based…

Self Systems Based…

Self Knowledge

Self Communication

Self Efficiency

Self Professionalism

Self TeamworkSelf Patient Care

1

6

11

16

21

26360 Patient Care

360 Teamwork

360 Professionalism

360 Efficiency

360 Overall

Patient Evaluations

Testing Overall

DM Quality

DM Change

Prevention Quality

Prevention Change

Absolute Work Units

Work Intensity

Citizenship

AND MORE!!!!!!

13

Patient Evaluations Your Score Your Rank Class Average Stand. Dev Range

Number of Evaluations Total 22.00 14 21.77 6.79 8.00-37.00

Average Visits per Patient 2.36 22 2.65 0.31 2.17-3.41

Physician Explains 5.59 19 5.66 0.17 5.30-6.00

Physician Listens 5.36 24 5.68 0.19 5.30-5.94

Physician Gives Instructions 5.59 21 5.73 0.18 5.30-6.00

Physician Knows History 5.59 15 5.59 0.17 5.00-5.82

Physician Respects Patient 5.52 23 5.76 0.17 5.39-6.00

Physician Spends Enough Time 5.45 21 5.62 0.20 5.17-5.94

Physician Calls With Results 5.17 18 5.30 0.33 4.32-5.78

*Rate on Scale of 0-10 9.14 19 9.23 0.34 8.30-9.79

Would Recommend To Others (scale 1-4) 3.90 9 3.85 0.09 3.63-4.00

Average/Overall Rank 22

Testing Score Rank Average Stand. Dev Range

ITE1 73 3 63.13 6.92 48-77

ITE2 80 4 68.00 9.09 45-84

GIM 1 73 2 59.00 9.68 38-76

Cardiology 62 16 66.00 7.06 44-76

Rheumatology 71 4 62.00 8.74 44-76

GI Liver 80 4 70.00 9.75 51-89

Endocrinology 57.00 10.00 40-78

Mathis Testing Overall 72 3 62.00 6.00 54-77

Number Average Stand. Dev Range

Number of notes closed > 24 hours 33 32.69 33.86 4 - 136

Number of notes closed > 7 days 0 2.88 6.24 0-26

Number of refills completed in 48 hours 90% 99% 3.91% 90%-107%

Number of completed patient evaluations 22 21.77 6.79 13728

Percentage of eligble AAP Attendance 100% 96% 10% 60%-100%

Citzenship Rank 15

Patient assessments!

Knowledge! Citizenship!

Clinical Competency Committee

CollectPre-

reviewReview Deep dive

Reporting

Feedback

15

If dashboard looks good.No red flags in narrative.

If underperforming on dashboard.If red flags in narrative.

Assessment ecosystem

Direct observation

Entrustment/OPAs Milestones data

CCC

ACGME

Semi-annual PD meeting

Two CCC paradigms

17Hauer KE, Chesluk B, Iobst W, et al. Reviewing Residents' Competence: A Qualitative Study of the Role of Clinical Competency Committees in Performance Assessment. Academic medicine : journal of the Association of American Medical Colleges. 2015.

Problem identification Developmental

“Red flags” and “low-flyers” Helping all residents to improve

Focus on global performance Examine specific strengths/weaknesses

Competence is implicit, just avoid failure

Competence based on comparison to milestones

Feedback delivered by report with minimal follow-up

Feedback delivered by longitudinal mentor/coach

18Hauer KE, Chesluk B, Iobst W, et al. Reviewing Residents' Competence: A Qualitative Study of the Role of Clinical Competency Committees in Performance Assessment. Academic medicine : journal of the Association of American Medical Colleges. 2015.

CCC questions to discuss Answer + Explanation Why

What assessment data does your

program collect that can be used for

specific learner feedback?

What challenges do you foresee in

shifting your CCC to a

Developmental Model? (time,

training, etc) What challenges have

you already faced?

What output/feedback does your

CCC provide to learners? Is it

specific and actionable for learners?

If not, how could that be changed?

Who delivers the feedback? Is this

the right person?

19

Group sharing

Name some coaches you’ve had in life, what

made them great?

20

Draw the following?

21

Asking Questions

Giving Answers

Learner is Expert

You’re the

Expert

1. Mentor2. Coach3. Friend4. Facilitator5. Counselor6. Consultant7. Advisor

Draw the following?

22

Asking Questions

Giving Answers

Learner is Expert

You’re the

Expert

1. Coach2. Mentor3. Friend4. Counselor

Ben’s hockey coach

Law

yer

23

What is coaching?

“helping someone get from point A to point B”

24

“Coaching is the art of facilitating

the performance, learning, and

development of another.”

Many domains can be coached

• Resiliency

• Self-directed learning

• Academic Pursuits

• Leadership

Coaching

competence

Why is Coaching so Important?

Norman G. The adult learner: a mythical species. Acad Med. 1999;74(8):886–889.Davis D, Mazmanian P, Fordis M, Harrison R, Thorpe K, Perrier L.Accuracy of physician self-assessment compared with observed measures of competence. JAMA.2006;296(9):1094–

1102.

27

• Feedback should be a reflective dialogue • Performance monitored, reflections shared and validated,

activities planned, and follow-up negotiated and monitored.

“In conventional assessment programmes, adherence to minimum standards can suffice for promotion and graduation. In programmatic assessment individual excellence is the goal and the mentor (coach) is the key person to promote such excellence.”

28

Kerri PalamaraMcGrath, MD

Coaching in Medical Education

Consistent, Longitudinal

Built on a relationship of trust

Not a supervisor or evaluator

Understand the system and curriculum

Do not need to be experts

Facilitate reflection and prioritize goals

Active Listening

Relentlessly Positive

29Telio S, Ajjawi R, Regehr G. The “educational alliance” as a framework for reconceptualizing feedback in medical education. Acad Med. 2015; 90(5): 609-614.Gawande A. Personal best. The New Yorker, 3 October 2011.

Coaching Sessions

Check In

Get Curious: Reflect and Challenge

Explore: Build Motivation and

Confidence

Accountability: Narrow and Commit

30

Kerri Palamara McGrath, MD

31

Let’s Review and Reflect

32

Practice Coaching

33

Unsupervised Practice

Indirect Supervision

Practice Coaching

34

ICS3 Appropriate utilization and completionof health records.PC-5 Manages patients with progressiveresponsibility and independence.PC-3Requests and provides consultative care

Practice Coaching

35

15 - Escalate care in a critical or unstable patient10 - Demonstrate communication to healthcare team members in timely manner24 - Respond to pages in timely manner45 - Takes leadership role of teaching healthcare team66 - Role model effective communication in challenging situations 168 – Demonstrate empathy and compassion in relieving pain and suffering

Practice Coaching

Narrative is so important

36

What we are doing now

1. Changing the data we review in CCC

(more specific)

2. Planning faculty development in our CCC

3. Planning to recruit coaches

4. Strategizing incentives for coaches

5. Brainstorming on how to measure

change

6. Coaching a handful of residents to pilot

the process

37

38

Coaching questions to discuss Answer + Explanation Why

How many coaches will you need? Ratio to learner?

Who can be a coach? Who cannot?

How often should coach and learners meet? Will it be mandatory?

What training will coaches

need? How often? Other

requirements?

How will you recruit and retain

coaches? What incentive can

you offer?

How will you match learners to

coaches?

How do you know if coaches are

effective?

“Feedback cultivates insight. Coaching promotes

performance change”

- Dr. Marygrace Zetkulic (yesterday)

39

Team

If you want peer mentorship as you transform your CCC into a

developmental model that fuels coaching, email us:

- kellehmw@ucmail.uc.edu

- kinneabn@ucmail.uc.edu

- warmej@ucmail.uc.edu

40