40
Transform your clinical competency committee with learner coaching Matthew Kelleher, MD, MEd (@kelzj3) Benjamin Kinnear, MD (@Midwest_MedPeds) Eric Warm, MD (@CincyIM) Date: 10/21/17
Transform your clinical competency committee with learner coaching
Matthew Kelleher, MD, MEd (@kelzj3)
Benjamin Kinnear, MD (@Midwest_MedPeds)
Eric Warm, MD (@CincyIM)
Date: 10/21/17
I do not have an affiliation (financial or otherwise) with a
pharmaceutical, medical device or communications organization.
Je n’ai aucune affiliation (financière ou autre) avec une
entreprise pharmaceutique, un fabricant d’appareils
médicaux ou un cabinet de communication.
2
Learning Objectives
By the end of this workshop, we hope you are able to:
1. Evaluate your own competency committee as operating under
a problem identification model or a developmental model.
2. Use coaching techniques with your learners.
3. Use your competency committee to guide coaching strategies
for learner improvement.
3
Think-pair-share
What does your CCC do to help residents:
• Improve
• Learn
• Grow
4
Observable Practice Activities (OPAs)
Process-Based (POPA)
Mid/End of Rotation Feedback (formative)
Entrustment (1-5)
Content-Based (COPA)
Reporting Milestones
External Reporting Longitudinal Assessment Formative > Summative
Mapping Mapping
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518
Average of 3,987 milestone assessments, 4325 narrative words per resident!!!!
6
Our CCC
Mountain of milestone data
Month of Residency
AggregateEntrustment
Blue = Resident Performance
Red = Predicted Score
N= 3320Number of
Assessments
• Trends from start of residency to present• Comparison to the predicted score
Blue = Resident Performance
Red = Predicted Score
Milestones Assessed in past 6
months
• Snapshot of the past 6 months• Gives us an idea of what we will be reporting
to ACGME and ABIM
EntrustmentScale
• Z-score = number of standard deviations Observed (blue line) is from Predicted (red Line)
Blue = Resident Performance
Red = Predicted Score
X-Bar Chart
SD Chart
"AUC"= 3.06
Z-Score (plus 3)
Standard Deviation within
the Subgroup (Competencies)
“Area under the curve”
3rd Year Resident at 36 Months
N= N= "AUC"=2470 815 3.06
There’s more!!!
12
1
6
11
16
21
26360 Patient Care
360 Teamwork360 Professionalism
360 Efficiency
Patient Evaluations
Testing Overall
DM Overall
Diabetes Process
Diabetes Outcome
Diabetes Change
Prevention Quality
Prevention Change
Absolute Work UnitsWork Intensity
CitizenshipSelf Professionalism
Self Patient Care
Self Patient Care
Self Systems Based…
Self Systems Based…
Self Systems Based…
Self Systems Based…
Self Knowledge
Self Communication
Self Efficiency
Self Professionalism
Self TeamworkSelf Patient Care
1
6
11
16
21
26360 Patient Care
360 Teamwork
360 Professionalism
360 Efficiency
360 Overall
Patient Evaluations
Testing Overall
DM Quality
DM Change
Prevention Quality
Prevention Change
Absolute Work Units
Work Intensity
Citizenship
AND MORE!!!!!!
13
Patient Evaluations Your Score Your Rank Class Average Stand. Dev Range
Number of Evaluations Total 22.00 14 21.77 6.79 8.00-37.00
Average Visits per Patient 2.36 22 2.65 0.31 2.17-3.41
Physician Explains 5.59 19 5.66 0.17 5.30-6.00
Physician Listens 5.36 24 5.68 0.19 5.30-5.94
Physician Gives Instructions 5.59 21 5.73 0.18 5.30-6.00
Physician Knows History 5.59 15 5.59 0.17 5.00-5.82
Physician Respects Patient 5.52 23 5.76 0.17 5.39-6.00
Physician Spends Enough Time 5.45 21 5.62 0.20 5.17-5.94
Physician Calls With Results 5.17 18 5.30 0.33 4.32-5.78
*Rate on Scale of 0-10 9.14 19 9.23 0.34 8.30-9.79
Would Recommend To Others (scale 1-4) 3.90 9 3.85 0.09 3.63-4.00
Average/Overall Rank 22
Testing Score Rank Average Stand. Dev Range
ITE1 73 3 63.13 6.92 48-77
ITE2 80 4 68.00 9.09 45-84
GIM 1 73 2 59.00 9.68 38-76
Cardiology 62 16 66.00 7.06 44-76
Rheumatology 71 4 62.00 8.74 44-76
GI Liver 80 4 70.00 9.75 51-89
Endocrinology 57.00 10.00 40-78
Mathis Testing Overall 72 3 62.00 6.00 54-77
Number Average Stand. Dev Range
Number of notes closed > 24 hours 33 32.69 33.86 4 - 136
Number of notes closed > 7 days 0 2.88 6.24 0-26
Number of refills completed in 48 hours 90% 99% 3.91% 90%-107%
Number of completed patient evaluations 22 21.77 6.79 13728
Percentage of eligble AAP Attendance 100% 96% 10% 60%-100%
Citzenship Rank 15
Patient assessments!
Knowledge! Citizenship!
Clinical Competency Committee
CollectPre-
reviewReview Deep dive
Reporting
Feedback
15
If dashboard looks good.No red flags in narrative.
If underperforming on dashboard.If red flags in narrative.
Assessment ecosystem
Direct observation
Entrustment/OPAs Milestones data
CCC
ACGME
Semi-annual PD meeting
Two CCC paradigms
17Hauer KE, Chesluk B, Iobst W, et al. Reviewing Residents' Competence: A Qualitative Study of the Role of Clinical Competency Committees in Performance Assessment. Academic medicine : journal of the Association of American Medical Colleges. 2015.
Problem identification Developmental
“Red flags” and “low-flyers” Helping all residents to improve
Focus on global performance Examine specific strengths/weaknesses
Competence is implicit, just avoid failure
Competence based on comparison to milestones
Feedback delivered by report with minimal follow-up
Feedback delivered by longitudinal mentor/coach
18Hauer KE, Chesluk B, Iobst W, et al. Reviewing Residents' Competence: A Qualitative Study of the Role of Clinical Competency Committees in Performance Assessment. Academic medicine : journal of the Association of American Medical Colleges. 2015.
CCC questions to discuss Answer + Explanation Why
What assessment data does your
program collect that can be used for
specific learner feedback?
What challenges do you foresee in
shifting your CCC to a
Developmental Model? (time,
training, etc) What challenges have
you already faced?
What output/feedback does your
CCC provide to learners? Is it
specific and actionable for learners?
If not, how could that be changed?
Who delivers the feedback? Is this
the right person?
19
Group sharing
Name some coaches you’ve had in life, what
made them great?
20
Draw the following?
21
Asking Questions
Giving Answers
Learner is Expert
You’re the
Expert
1. Mentor2. Coach3. Friend4. Facilitator5. Counselor6. Consultant7. Advisor
Draw the following?
22
Asking Questions
Giving Answers
Learner is Expert
You’re the
Expert
1. Coach2. Mentor3. Friend4. Counselor
Ben’s hockey coach
Law
yer
23
What is coaching?
“helping someone get from point A to point B”
24
“Coaching is the art of facilitating
the performance, learning, and
development of another.”
Many domains can be coached
• Resiliency
• Self-directed learning
• Academic Pursuits
• Leadership
Coaching
competence
Why is Coaching so Important?
Norman G. The adult learner: a mythical species. Acad Med. 1999;74(8):886–889.Davis D, Mazmanian P, Fordis M, Harrison R, Thorpe K, Perrier L.Accuracy of physician self-assessment compared with observed measures of competence. JAMA.2006;296(9):1094–
1102.
27
• Feedback should be a reflective dialogue • Performance monitored, reflections shared and validated,
activities planned, and follow-up negotiated and monitored.
“In conventional assessment programmes, adherence to minimum standards can suffice for promotion and graduation. In programmatic assessment individual excellence is the goal and the mentor (coach) is the key person to promote such excellence.”
28
Kerri PalamaraMcGrath, MD
Coaching in Medical Education
Consistent, Longitudinal
Built on a relationship of trust
Not a supervisor or evaluator
Understand the system and curriculum
Do not need to be experts
Facilitate reflection and prioritize goals
Active Listening
Relentlessly Positive
29Telio S, Ajjawi R, Regehr G. The “educational alliance” as a framework for reconceptualizing feedback in medical education. Acad Med. 2015; 90(5): 609-614.Gawande A. Personal best. The New Yorker, 3 October 2011.
Coaching Sessions
Check In
Get Curious: Reflect and Challenge
Explore: Build Motivation and
Confidence
Accountability: Narrow and Commit
30
Kerri Palamara McGrath, MD
31
Let’s Review and Reflect
32
Practice Coaching
33
Unsupervised Practice
Indirect Supervision
Practice Coaching
34
ICS3 Appropriate utilization and completionof health records.PC-5 Manages patients with progressiveresponsibility and independence.PC-3Requests and provides consultative care
Practice Coaching
35
15 - Escalate care in a critical or unstable patient10 - Demonstrate communication to healthcare team members in timely manner24 - Respond to pages in timely manner45 - Takes leadership role of teaching healthcare team66 - Role model effective communication in challenging situations 168 – Demonstrate empathy and compassion in relieving pain and suffering
Practice Coaching
Narrative is so important
36
What we are doing now
1. Changing the data we review in CCC
(more specific)
2. Planning faculty development in our CCC
3. Planning to recruit coaches
4. Strategizing incentives for coaches
5. Brainstorming on how to measure
change
6. Coaching a handful of residents to pilot
the process
37
38
Coaching questions to discuss Answer + Explanation Why
How many coaches will you need? Ratio to learner?
Who can be a coach? Who cannot?
How often should coach and learners meet? Will it be mandatory?
What training will coaches
need? How often? Other
requirements?
How will you recruit and retain
coaches? What incentive can
you offer?
How will you match learners to
coaches?
How do you know if coaches are
effective?
“Feedback cultivates insight. Coaching promotes
performance change”
- Dr. Marygrace Zetkulic (yesterday)
39
Team
If you want peer mentorship as you transform your CCC into a
developmental model that fuels coaching, email us:
40
