Jennifer Kogan, MD
Associate Professor of Medicine
Director of Undergraduate Education
Department of Medicine
Assessment in Medical Education:
Evidence Based Clinical Skills
Assessment in the Competency Era
2
Who Watched You?
3
Objectives
1. Describe theories supporting importance of direct
observation (DO) of trainees’ clinical skills
2. Articulate factors that impact the quality and
accuracy of rater assessments
3. List strategies to improve assessment of clinical
skills
4
Objective 1: Theories Supporting DO Importance
Importance of and current state of clinical skills
Development of expertise
Role in competency based medical education
Necessity as part of effective supervision
5
Objective 1: Theories Supporting DO Importance
Importance of and state of clinical skills
Development of expertise
Role in competency based medical education
Necessity as part of effective supervision
6
History and Exam Skills Importance
Leads to diagnosis > 80% of the time
Even in era of technology
Required to avoid unnecessary testing
Faulty data gathering common source of diagnostic
errors
Hampton JR et al. BMJ 1975; 2(5969):486-9
Peterson MC et al. West J Med. 1992; 156(2):163-5
Graber, M et al. Acad Med. 2002;77(10):981-92
7
State of History & Exam Skills
Trainees
Wide variability in students’ clinical skills as MS4s
or starting internship
Practicing physicians
Variability in history taking skills
Variability in exam skills
Stillman PL et al. Ann Intern Med.1990; 65(5):320-6
Sachdeva AK. Surg.1995;118(2):300-8
Lypson M. Acad Med.2004; 79(6):564-70
Mangione S et al. JAMA;1997; 278(9):717-22
Paauw DS et al. JAMA 1995;274(17);1380-2
8
High Quality Care
Timely
Efficient
Equitable
Safe
Effective
Patient Centered
Crossing the Quality Chasm:
A New Health System for the 21st Century 2001
Crossing
the
Quality
Chasm
9
Patient Centered Care
‘‘A partnership among practitioners, patients, and
their families (when appropriate) to ensure that
decisions respect patient’s wants, needs, and
preferences and that patients have the education
and support they need to make decisions and
participate in their own care.”
IOM 2001
10
Importance of Effective Communication
Patient involvement in care
Patient knowledge and self-efficacy
Adherence to treatment
Patient well-being
Patient satisfaction
Improved outcomes
Decreases costs Levinson W et al. 2010; Health Aff 29: 1310-18
Williams S et al. Fam Prac.1998;15:480-92
DiMatteo M. Patient Educ Counsel. 2004;55:339-44
Stewart M . CMAJ. 1995; 152:1423-33.
AHRQ 2005.
11
State of Patient Centered Care
Practicing physicians
Missing elements of informed decision making
Why the gap?
Communication is sophisticated procedure
Needs to be taught/ honed
Skills are rarely taught or practiced
Braddock CH et al. J Gen Intern Med. 1997;12(6):339-45
Levinson W. BMJ Qual Saf 2011;20:823-5
12
LEARNING
ASSESSMENT
Direct Observation to Assess Core Skills
Legitimizes the subject
Sends message skills are important
Ensures assessment of essential skills
13
Objective 1: Theories Supporting DO Importance
Importance of and state of clinical skills
Development of expertise
Role in competency based medical education
Necessity as a part of supervision
14
Dreyfus and Dreyfus Model
Dreyfus SE and Dreyfus HL. 1980
Carraccio CL et al. Acad Med 2008;83:761-7
Instruction, Time, Practice, Experience
Novice
Advanced Beginner
Competent
Proficient
Expert/
Master S
K
I
L
L
15
Expert Performance vs. Everyday Skills
Ericsson KA. Acad Med. 2004;79(10):S70-81
16
Deliberate practice
Working on well defined tasks
Informative feedback
Repetition
Self-reflection
Motivation
Endurance
Ericsson KA et al. Psych Rev.1993.100(3):363-406.
Davis D et al. JAMA 2006; 296:1094-1102
Eva KW et al. Acad Med. 2005;80:S46-54
How Do People Become Experts?
Self-assessment is inaccurate
17
What Do They Have in Common?
Meryl Streep
Babe Ruth
Bobby Fischer
Isaac Stern
Nadia Comaneci
18
Role of the Coach
“They observe, they judge, and they
guide”
“That one twenty-minute discussion
gave me more to consider and work
on than I’d had in the past five
years”
“Medical practice is largely unseen
by anyone who might raise one’s
sights. I’d had no outside ears and
eyes.”
Atul Gawande, New Yorker 10/3/2011
19
DIRECT
OBSERVATION
Direct Observation and Expertise
EXPERTISE
Deliberate
Practice
20
Objective 1:Theories Supporting DO Importance
Importance of and state of clinical skills
Development of expertise
Role in competency based medical education
Necessity in supervision
21
Structure and Process and Time
Outcome (Competency)
Societal Needs
Competency Based Medical Education
22
Defining Competency Based Education
Frank JR et al. Med Teach. 2010;32:631-7
Defined outcome
Competence
Observable
&
Assessed
Milestone
Milestone
Milestone
Milestone
23
Miller’s Assessment Pyramid
Van der vleuten CPM et al. Best Practice & Research Clinical Obstetrics and Gynaecology . 2010(24) :703–19
24
Evidence Based Strategies to Assess Does
Direct Performance
Measures
Aggregated
Methods
Individual
Encounter *Mini-cex
Longer term
methods * Multi-source
feedback
*End of rotation
evaluations
* In-training
evaluations
Sampling
over time * Logbook
*Portfolios
Van der vleuten CPM et al. Best Practice & Research Clinical Obstetrics and Gynaecology . 2010(24) :703–19
25
In-Training Performance Assessment
Assessment in authentic situations
Learners’ ability to combine knowledge, skills, judgments, attitudes in dealing with realistic problems of professional practice
Assessment in day to day practice
Enables assessment of a range of essential competencies, some of which cannot be validly assessed otherwise
Govaerts MJB et al. Adv Health Sci Edu. 2007;12:239-60
26
Objective 1: Theories Supporting DO Importance
Importance of and state of clinical skills
Development of expertise
Role in competency based medical education
Necessity in supervision
27
IOM Report 2008: More Than Just Duty Hours
Resident Duty Hours:
Enhancing Sleep, Supervision and Safety
28
Assessing Does
THE PATIENT
29
Entrustment
“A practitioner has demonstrated the
necessary knowledge, skills, and attitudes to
be trusted to independently perform this
activity.”
Ten Cate O, Scheele F. Acad Med 2007;82:542-7
30
Objectives
1. Describe theories supporting importance of direct
observation (DO) of trainees’ clinical skills
2. Articulate factors that impact the quality and
accuracy of rater assessments
3. List strategies to improve assessment of clinical
skills
31
High Inter-Assessor Variability of Scores
Leniency effect
Stringency effect
Halo effect
Central tendency effect
Anchoring bias
Contrast bias
Poor accuracy
32
Objective 2: Rating Quality & Accuracy
Sources of high inter-rater variability
Frame of reference
Faculty clinical skills
Inference
Contextual factors
Emotions
33
Objective 2: Rating Quality & Accuracy
Sources of high inter-rater variability
Frame of reference
Faculty clinical skills
Inference
Contextual factors
Emotions
34
Variable Frames of Reference
Different basis for judgments/ratings
Self (predominant)
Normative (trainee level)
Absolute standard
Practicing physicians
Kogan JR, et al. Med Educ. 2011. 45(10):1048-60
Yeates P et al. Adv in Heath Sci Educ. In Press
Govaerts M Adv Health Sci Educ. 2007.12(2):239-60.
35
Objective 2: Rating Quality & Accuracy
Sources of high inter-rater variability
Frame of reference
Faculty clinical skills
Inference
Contextual factors
Emotions
36
Faculty Characteristics
Minimal impact of demographics
Age, gender, clinical and teaching experience
Faculty’s own clinical skills may matter
Faculty with higher history and patient
satisfaction performance scores provide more
stringent ratings
Kogan JR. et al. Acad Med. 2010;85(10 S):S25-8
37
Objective 2: Rating Quality & Accuracy
Sources of high inter-rater variability
Frame of reference
Faculty clinical skills
Inference
Contextual factors
Emotions
38
High Level Inference
Feelings
Comfort/Confidence
Intentions
Ownership/work-ethic
Personality
Culture
Skills
Knowledge
Competence
Prior experience/familiarity with
scenario
Kogan JR, et al. Med Educ. 2011. 45(10):1048-60
Govaerts M Adv Health Sci Educ. 2007.12(2):239-60.
39
Problems with Inference
Not recognized
Rarely validated for accuracy
Can be wrong
Kogan JR, et al. Med Educ. 2011. 45(10):1048-60
Govaerts M Adv Health Sci Educ. 2007.12(2):239-60.
40
Objective 2: Rating Quality & Accuracy
Sources of high inter-rater variability
Frame of reference
Faculty clinical skills
Inference
Contextual factors
Emotions
41
Contextual Factors and Emotion
Encounter complexity
Trainee characteristics
Trainee relationships
Institutional culture
Constructive feedback
42
Direct Observation: A Conceptual Model
Kogan JR, et al. Med Educ. 2011 . 45(10):1048-60
43
Objectives
1. Describe theories supporting importance of direct
observation (DO) of trainees’ clinical skills
2. Articulate factors that impact the quality and
accuracy of rater assessments
3. List strategies to improve assessment of clinical
skills
44
Objective 3: Strategies to Improve Assessment
Evidence based psychometric principles
Rater training
Meaningful feedback
Culture change
45
Objective 3: Strategies to Improve Assessment
Evidence based psychometric principles
Rater training
Meaningful feedback
Culture change
46
Evidence Based Principles from 1st Three Layers
Van der vleuten CPM et al. Best Practice & Research Clinical Obstetrics and Gynaecology . 2010(24) :703–19
1. Competence is
specific, not generic
2. Objectivity reliability
3. Need for programs of
assessment
47
Evidence Based Principles about “Does”
Govaerts MJB et al. Adv Health Sci Edu. 2007;12:239-60
Van der vleuten CPM et al. Best Practice & Research Clinical Obstetrics and Gynaecology . 2010(24) :703–19
1) Reliable assessment
requires sampling (8-10)
across contexts &
assessors
2) Bias inherent in expert
judgment
3) Qualitative/narrative
information important
4) Validity resides in
“instrument user”
48
Objective 3: Strategies to Improve Assessment
Evidence based psychometric principles
Rater training
Develop a shared mental model
(Re)define frame of reference and scale anchor
Re-think assessment goals
Align rater training with clinical skills education
Build in meaningful feedback
Transform culture
49
Objective 3: Strategies to Improve Assessment
Evidence based psychometric principles
Rater training
Develop a shared mental model
(Re)define frame of reference and scale anchor
Re-think assessment goals
Align rater training with clinical skills education
Build in meaningful feedback
Transform culture
50
Performance Dimension Training
51
Performance Dimension Training
Identify specific dimensions of a competency
in behavioral terms
Discuss the criteria and qualifications required for
each dimension of that competency
Develop a SHARED MENTAL MODEL
Achieve evidence-based standardization
and calibration
Holmboe ES ABIM 2010
52
Objective 3: Strategies to Improve Assessment
Evidence based psychometric principles
Rater training
Develop a shared mental model
(Re)define frame of reference and scale anchor
Re-think assessment goals
Align rater training with clinical skills education
Build in meaningful feedback
Transform culture
53
Re-define the Frame of Reference
Competency Based
Education
Criterion Based defined by
High Quality Care
Milestones
Entrustment
Self, Normative, Gestalt
54
What is Needed by the Patient
Dreyfus SE and Dreyfus HL. A 1980
Carraccio CL et al. Acad Med 2008;83:761-7
Time, Practice, Experience
Novice Advanced Beginner
Competent
Proficient
Expert/
Master
55
Re-define Scale Anchor
Scale Midpoint
=
Satisfactory
=
Competent
=
Safe, Effective,
Patient Centered Care
56
A Foot In Two Worlds
Normative Competent for
Entrustment
57
Implications
Many trainees considered “unsatisfactory”
Educational culture shift in meaning of
unsatisfactory
Non-aspirational
Defining superior
58
Objective 3: Strategies to Improve Assessment
Evidence based psychometric principles principles
Rater training
Develop a shared mental model
Re-define frame of reference and scale anchor
Re-think assessment goals
Align rater training with clinical skills education
Meaningful feedback
Transform culture
59
Utility Elements of Assessment
Validity
Reliability
Educational impact
Practicability
Acceptability
Cost effectiveness
Patient care outcome
Trainee
Learning Process
Assessment
Inform Supervision
What can trainee do
independently?
What does supervisor
need to contribute?
Van der Vleuten. Adv Health Sci Educ. 1996;1:42-67
60
Accountable Supervision and Quality Index
Safe,
Effective,
Patient
Centered
Care
Trainee
Performance
* Function of
level of
trainee
competence
in context
Appropriate
Supervision
Level
** Function
of level of
attending
competence
in context
x Must
=
61
Objective 3: Strategies to Improve Assessment
Evidence based psychometric principles
Rater training
Develop shared mental model
Re-define frame of reference and scale anchor
Re-think assessment goals
Align rater training with clinical skills education
Build in meaningful feedback
Transform culture
62
Leverage Rater Assessment Problem
Faculty development on skills being assessed
Patient centered communication
Evidence based physical exam
Effective use of EHR
Dual benefit
Clinical skills
Educator Skills
63
Objective 3: Strategies to improve assessment
Evidence based psychometric principles
Rater training
Develop shared mental model
Re-define frame of reference and scale anchor
Re-think assessment goals
Align rater training with clinical skills education
Build in meaningful feedback
Transform culture
64
Build in Meaningful Feedback
65
Transform Culture
1) Buy-in about DO importance
2) Faculty champions
3) Systems to accommodate DO
4) Motivations/reward
66
Quality
improvement for
the trainee
Quality
improvement for
the patient
Conclusions: Direct Observation
67
Eric Holmboe, MD
Lisa Conforti, MPH
Elizabeth Bernabeo, MPH
Brian Hess, PhD
Krista Hirshman
Siddhartha Reddy, MPH
William Iobst, MD
American Board of Internal Medicine
Acknowledgements
68
Questions
