Competency-based assessment Competency-based assessment and e-portfolio in residency trainingand e-portfolio in residency training

Maastricht University Maastricht University Manchester UniversityManchester University

Marjan GovaertsJ. Donkers; H.Brackel; B. Verhoeven;

C. Van der Vleuten; E. Driessen; T. Dornan

Residency Training NetherlandsResidency Training Netherlands

Nationwide reformNationwide reform

Competency-based: CanMEDS rolesCompetency-based: CanMEDS rolesWorkplace-based assessmentWorkplace-based assessmentPortfolioPortfolio– Guidance of competence developmentGuidance of competence development– Evidence-based decision making Evidence-based decision making

Implementation ProblemsImplementation Problems

Workplace-based assessment: time Workplace-based assessment: time consuming and administrative overloadconsuming and administrative overload– Meaningful feedback is lackingMeaningful feedback is lacking– Documentation of feedback/performance is Documentation of feedback/performance is

lackinglacking

Portfolio: limited value Portfolio: limited value – Information in portfolio non-transparentInformation in portfolio non-transparent– Credible (evidence-based) norms are lackingCredible (evidence-based) norms are lacking

Maastricht Manchester TSS Maastricht Manchester TSS

E-portfolio with built-in online support E-portfolio with built-in online support system for Workplace Assessmentsystem for Workplace Assessment

Rating forms attuned to educational goalsRating forms attuned to educational goals– Competency-based (CanMEDS)Competency-based (CanMEDS)

Limited number of rating instrumentsLimited number of rating instruments– Mini-CEX; OSATS; MSF (incl. self assessment) Mini-CEX; OSATS; MSF (incl. self assessment) – Numeric scores for behavioral objectives + narrative Numeric scores for behavioral objectives + narrative

feedbackfeedback

Maastricht Manchester TSSMaastricht Manchester TSS

1.1: 3.51.1: 3.51.2: 4.21.2: 4.2

2.1: 5.02.1: 5.02.2: 4.32.2: 4.3

7.4: 3.4 7.4: 3.4

ScoresScores

Rating formRating form

CCAANNMMEEDDSS

LongitudinalLongitudinaldevelopmentdevelopment

Communicator

Cohort scoresCohort scores

Performance peers

MMTSS feedbackMMTSS feedbackMedical expert

Professional

Manager

Communicator

Collaborator

Scholar Health advocate

Individual versus peer group

Medical expertCommunicatorCollaboratorScholarHealth advocateManagerProfessional

Individual over time

MMTSS feedbackMMTSS feedback

Narrative feedback

Performance ratings(others vs self)

Maastricht Manchester TSSMaastricht Manchester TSS

Rating forms attuned to educational goalsRating forms attuned to educational goals– Competency-based template (CanMEDS)Competency-based template (CanMEDS)

Aggregation of scores across rating instrumentsAggregation of scores across rating instruments– Automated feedback reports on competence Automated feedback reports on competence

development development

Collection of performance rating data in Collection of performance rating data in databasedatabase– Evidence-based norms and decision makingEvidence-based norms and decision making

Comparison of individual performance with aggregated Comparison of individual performance with aggregated performance data of peer groupperformance data of peer group

MMTSS E-portfolioMMTSS E-portfolio

Documentation (surgical) procedures / Documentation (surgical) procedures / learning activitieslearning activitiesReflective practice and self-directed Reflective practice and self-directed learninglearning– Critical incident reportsCritical incident reports– Self assessment and analysisSelf assessment and analysis– Personal Development PlansPersonal Development Plans…….other artefacts/’evidence’ illustrating .other artefacts/’evidence’ illustrating achievement of professional competenceachievement of professional competence

Results small scale pilot Results small scale pilot

Promotes workplace based assessmentPromotes workplace based assessmentProvides meaningful feedbackProvides meaningful feedback– Insight in strengths and weaknessesInsight in strengths and weaknesses– AcceptableAcceptable– TransparentTransparent

User-friendlyUser-friendly

ChallengesChallenges

How to encourage meaningful use of the How to encourage meaningful use of the portfolio after portfolio after large scale implementationlarge scale implementation??Organizational support requiredOrganizational support required– Purposeful use by residents, teachersPurposeful use by residents, teachers

Reflection and self-directed learningReflection and self-directed learning– Quality improvement of residency trainingQuality improvement of residency training

Flexible, tailor-made, cost-efficientFlexible, tailor-made, cost-efficient

Thank you for your attention.Thank you for your attention.

Questions?Questions?