Validation of the Manitoba Practice Assessment ProgramCME Congress: June 1, 2012

MPAP Program TeamJose Francois MD, Associate DeanMarilyn Singer MD, DirectorJillian Horton MD, Assistant DirectorBrenda Stutsky PhD, Program AdvisorDebbie Lemkey, Program AssistantRobert Renaud PhD, Dept. Ed. Admin., Foundations & Psychology

Conflict of Interest: Nothing to disclose: Pilot study self-funded

Background/Purpose• MPAP developed based on need from the College of

Physicians and Surgeons of Manitoba (CPSM)• Involves the assessment of practicing physicians on the

CPSM conditional register

• The purpose of the pilot study was to test selected MPAP processes and examine the preliminary reliability and validity of the practice assessment tools

Method• Mixed methods design• 6 Stages:

1. Training of assessors2. Selection of family physicians and 1st year family

medicine residents3. Self-assessment process and 360 degree surveys4. Onsite assessment5. Collation of data and feedback to the physician

candidates6. Final debriefing session

Results: Assessor Online Training (N = 8)Length of time to complete the self-directed online learning activities:

– Mean 7.13 hrs. (Range of 4-9 hrs.)

Enjoyed online format:– Yes 75%– It was okay 25%– No 0%

After completing the online activities, assessors believed they had a good understanding of the various assessment strategies/processes:

– Yes 100%– No 0%

Results: Assessor Face-to-Face Training (N = 12)I am ready to begin to assess physicians:

– Yes, and I do not require additional training: 50%– Yes, but I will need to shadow/buddy an experienced

assessor or participate in a pilot/mock assessment: 50%– No, I will need additional guidance and training: 0%

I recommend that the Assessor Workshop be modified:– Yes: 0%– No: 100%

Results: Comparison of Mean Scores between Family Physicians and Residents & Correlation to PAR

Note: Higher scores reflect more positive ratings

CanMEDS Roles Overall Onsite Visit Scores (range 1-2)

360 Degree Survey Scores Self-Assessment Scores

(range 1-4) Physician Colleagues (range 1-4)

Interprofessional Colleagues (range 1-4)

Patients (range 1-4)

FP (5) R (1) FP (5) R (4) FP (5) R (3) FP (5) R (1) FP (5) R (4) Medical Expert 1.99 2.00 3.39 2.71 3.60 3.23 3.70 3.36 2.74 2.65 Communicator 2.00 2.00 3.39 2.62 3.55 3.38 3.60 3.21 2.97 2.83 Collaborator 2.00 2.00 3.45 2.68 3.47 3.31 3.64 3.75 2.69 2.78 Manager 2.00 -- 3.31 2.28 3.52 3.22 3.20 2.94 2.47 2.38 Health Advocate 2.00 2.00 2.95 2.74 3.53 3.21 3.50 3.20 2.44 2.61 Scholar 2.00 -- 3.31 2.95 3.60 2.80 3.44 3.00 2.64 2.19 Professional 2.00 2.00 3.23 3.02 3.60 3.39 3.72 3.58 2.68 2.97 Correlation between Mean total MPAP scores and Mean total Alberta scores

0.96 0.72 0.61

Results: Debriefing Session1. Overall, what did you like about the MPAP Process?

– Interprofessional collaboration, training process, usability and variety of tools

2. Overall, what could improve?– Office scheduling, patient selection, access to EMR/charts, use of

EMR for data, space, length of time between training and onsite assessment

3. How did you feel during the assessment process?– Assessors: uncomfortable, anxious → confident over time– Candidates: nervous, pre-occupied with other roles, “like a resident

again”4. If you were to provide guidance to physicians

undergoing the assessment process, what would you tell them?– Relax, process allows your strengths to shine, encouraging

comments - want them to succeed

Conclusions• Blended learning was effective• Provided valuable training experience for

assessors in a low-stakes environment• Limited generalizability given small sample

size• Validity and reliability and of tools is promising

– Continue to work on validity and reliability • Need only minor changes to the processCopy of Presentation (Follow MPAP links):

www.umanitoba.ca/cpd