Portfolio assessment of behavioral competencies at the Feinberg School of Medicine Celia Laird O’Brien, PhD
Competency Based Medical Education
4
Does
Shows How
Knows How
Knows MCQ exam
Clinical vignette, essays
OSCE, simulation
Workplace Based Assessment
Assessment of behaviors • Many data points are necessary, collected from
multiple observers, in different contexts, across time.
6
Assessment of behaviors
• How do we organize and review this data to determine competence?
7
Result: Lots of data!
The portfolio approach
8
• The Feinberg electronic portfolio is used to organize this data: a “learning chart”
• Allows students to review and reflect on their performance across time
• Allows us to measure and enhance behavioral competencies in our students
The portfolio approach
Three goals:
1. Permit assessment of competencies not easily measured by traditional grading methods. 2. Develop students’ capacity to self-assess their skills and abilities using external feedback. 3. Develop students’ ability to engage in self-directed learning and improvement.
9
Assessment at Feinberg
10
Fail/Pass/High Pass/Honors Pass/Fail Fail/Pass/Honors
Traditional grading system
Portfolio review Portfolio review
The portfolio approach
11
Competencies subject to portfolio review:
• Effective Communication and Interpersonal Skills (ECIS)
• Patient Centered Medical Care (PCMC)
• Professional Behavior and Moral Reasoning (PBMR)
• Systems Awareness and Team Based Care (SATBC)
• Continuous Learning and Quality Improvement (CLQI)
The portfolio approach Examples of assessment data:
• Faculty and peer evaluations of small group work
• Clinical performance evaluations
• Feedback from standardized patients
• Feedback from nurses
•Narrative data is most informative. 12
Portfolio review process
• Portfolio review committees comprised of experienced clinician educators
• Each portfolio is reviewed by at least two clinicians, who judge competence by the following:
a) Contents of the portfolio (institutional assessment data, external uploads)
b) Quality of reflections and learning plans
17
Portfolio review process •Decisions:
• Progressing towards competence
• Progressing towards competence with some concern
• Progressing towards competence pending additional development
18
Portfolio review process •Assessment of competence isn’t about the numbers.
•Process depends on judgment from experienced educators.
• Involves analysis of qualitative data; uses qualitative methodology to collect evidence of validity
19
Portfolio review process •Reviewers work in pairs
• Independently score portfolios, then come together to reach consensus through discussion and debate
•Third reviewer used if consensus cannot be reached
•Final decisions reviewed by committee as a whole
20
Portfolio review process •Multiple training sessions are held, including group standard setting exercise
• Important to work from a shared frame of reference
•Reviewers look for patterns and themes in performance across time
21
Results – Phase 1
22
25
39
30 38
16
1
4
4
4
1
0
5
10
15
20
25
30
35
40
45
50
CLQI ECIS PCMC PBMR SATBC
PC PAD
Evaluation
• Statistical analyses show a significant association between results of the portfolio review in Phase 1 and clerkship performance in Phase 2, even after controlling for medical knowledge.
23
Evaluation
24
3
2
1 1
2
3
0
1
2
3
4
5
6
7
Honors High Pass Pass
Median number of clerkship grades awarded by group
No concerning behavior (n=111) Concerning behavior (n=24)
p<.01 p<.01
Evaluation
25
82.3
79.4
70
72
74
76
78
80
82
84
86
88
90
No concerning behavior Concerning behavior
Mean clerkship performance score
p<.001
27
“The last two decades have been characterized by happy
portfolio developers and grumpy portfolio users.”
- Erik Driessen, 2016
Lessons learned
• Many students don’t see value of mandated reflection or self-improvement efforts
28
29
4.4
3.2
1
2
3
4
5
My mentor is able to helpme understand my
evaluation data as it relatesto the competency
standards.
Writing reflections about myprogress in the
competencies has proved tobe a useful exercise.
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
Lessons learned
• Numbered ratings proved to be misleading and often contradicted narrative comments.
• Vast majority of students are rated “above average”… even when they’re not.
30
Lessons learned
• There is inherent overlap between these competency domains.
31
Communication Professionalism Teamwork
Lessons learned
• We need to re-examine our remediation efforts and determine how to best support students with deficiencies.
33
Lessons learned
• We had to pull back plans to transmit results of the portfolio review to residency directors
34
Conclusion
• Portfolio assessment is a feasible way to assess behavioral competencies but requires investment of dedicated time and resources.
35
References 1. O’Brien C.L., Sanguino S.M., Thomas J.X., Green M.M. Feasibility and outcomes of implementing a portfolio
assessment system alongside a traditional grading system [published online ahead of print March 29, 2016]. Academic Medicine.
2. Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Academic Medicine. 2007;82(5):493-502.
3. Driessen E, van der Vleuten C, Schuwirth L, van Tartwijk J, Vermunt J. The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study. Medical Education. 2005;39(2):214-220.
4. Johnston B. Summative Assessment of Portfolios: An Examination of Different Approaches to Agreement over Outcomes. Studies in Higher Education. 2004;29(3):395-412.
5. Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Academic Medicine. 2010;85(7):1212-1220.
6. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Medical Teacher. 2012;34(3):205-214.
36