A Novel Approach for Developing Rating Scales for a Practice Ready OSCE

Bruce HolmesDivision of Medical Education, Dalhousie University

Saad ChahineMount Saint Vincent University

Clinician Assessment for Practice Program (CAPP)

• A program of the College of Physicians and Surgeons of Nova Scotia (CPSNS)

• A 3-part program assessing IMG physicians (residents in Canada, without formal Canadian training) for practice readiness.

The CAPP ProgramPart A: Initial assessment• Assessment of competence via OSCE & therapeutics

exam

Part B: 1 year mentorship with a family physician• Defined license for 1 year with performance assessment

Part C: Additional 3 years of defined license untilcertified by The College of Family Physicians

(CCFP)

Purpose of the CAPP OSCE

1. To assess the clinical competence of IMG candidates for readiness to practice.

2. To provide feedback on candidates' performance for their continuing professional development.

• The OSCE is not a “pass or fail” examination, but rather one factor in the decision to grant a defined license to practice in Nova Scotia.

Examining the Constructs of Rating Form?

• Re-design rating form

• Provide rater guidelines on how to be an examiner

• Two online practice videos for raters– Results submitted– Group data reported to compare with individual rating– Personal feedback available

OSCE Competencies

• History Taking• Physical Exam• Communication Skills• Quality of Spoken English• Counselling• Professional Behaviour• Problem Definition & Diagnosis*• Investigation & Management*

• Overall Global*

Example of Elements for Competency

Purpose of Elements“Guided Conceptualization”

• The elements act as a guide to forming a conceptualization of the competency

• Theory the elements help the examiner in self regulatory process to build an

“awareness and management of one’s own thought” (Kuhn & Dean, 2004)

• Provide “standardized” comments as feedback for candidates and Credentials Committee of College

How are Elements Related to Competencies?

• Reviewed the ratings in 2012 OSCE– 36 candidates

– Total of 432 observations

• Each set of elements correlated with competency– e.g., History; Investigation & Management

• Spearman Correlation between the element ratings and the respective competency rating was conducted one at a time

Results• History Taking; Counselling; Investigation &

Management:– High positive correlation for all element ratings

with the respective competency rating (r-values 0.74-0.82)

• Problem Definition and Diagnosis: – High positive correlation (r-values 0.70-0.81) except for

alternate diagnosis (r=0.61)

• Physical Exam – High positive correlation (r-values 0.70-0.75) except for

sensitivity (r=0.61)

Future Analysis for Competencies – CFA

History Taking

Breadth

Relevance

Focus

Sequencing

Chronology

• Include data from 2013 & 2014

• Confirmatory Factor Analysis conducted through Structural Equation Modeling

Take Home Messages

• Strong positive associations between elements and competency rating– Appears to be helpful to guide examiner ratings– Helpful feedback on competencies

• Follow-up with refinement of elements using examiner feedback

• Follow-up with data from June 2013 & 2014 OSCE

Thank You!

www.capprogram.ca

Saad Chahine

saad.chahine@msvu.ca

Bruce Holmes

bruce.holmes@dal.ca