Establishing Content Validity of a Novel Written Examination to Assess Medical
Students on the Surgery ClerkshipA Berlin MD, A Reinert BA, A Swan-Sein MEd PhD,
R Nowygrod MD, A Fingeret MD
Agenda
• Needs assessment• Exam Development• Exam Design• Methodology• Results• Discussion
Needs Assessment• Historical assessment of students– Clinical evaluations– Unstructured oral exam – NBME surgery subject exam
• Goals of new evaluation tool– Objective and reliable– Congruent with educational objectives– Based on ACGME core competencies
Exam Design
• Novel Exam: Clinical Skills Exam (CSE)– 20+ clinical scenarios• 5 randomly selected per exam
– Short answer or essay response– Administered over 3 hours– ExamSoft computer software
• Grading – Rubric– Senior residents
Example 1
• Identify operation
• Obtain surgical consent
• Determine order
• Describe steps shown
Example 2
• Generate differential based on history
• Describe risk factors• Interpret radiographic
study• Identify next step in
management
Methodology• July, 2011 - February, 2013– n = 262
• Analysis– Descriptive statistics– Multivariate analysis
• Comparison data for validity assessment – USMLE Step 1 and 2CK– NBME Subject Exams in Medicine and Surgery– Clerkship clinical evaluations – Clerkship order
Results: Descriptive Statistics – CSE Content and Score Distribution
Results: Trend Analysis – CSE and NBME Surgery Subject Exam
Results: Multivariate Statistics - Validity Analysis
CSE NBME
CorrelationLinear
Regression CorrelationLinear
Regressionr p value r p value
NBME Surgery 0.36* 0.82 NBME Medicine 0.01 0.54 0.08 0.20Step 1 0.34* 0.82 0.55* 0.02Step 2 0.44* 0.007 0.56* 0.004Clerk - Clinical Reasoning 0.48* 0.001 0.43* 0.09
Clerk - Fund of Knowledge 0.56* 0.004 0.63* < 0.001
* = p < 0.001
Results: Multivariate Statistics – Validity Analysis
CSE NBMEBivariate Analysis
Bivariate Analysis
Logistic Regression
p value p value p value
Clerkship order ANOVA 0.30 < 0.001 0.01
OB/Gyn prior X2 0.34 < 0.001 0.02
Medicine prior X2 0.69 < 0.001 0.03
Results: Student Feedback on CSE
Results: Student Feedback - Word Cloud
Limitations
–Study•Generalizability• Statistical analysis
Future Directions
• Future directions– New content– Open access– Computer algorithm based grading– Use as formative assessment mid-clerkship
Conclusions
• CSE is a useful adjunct to NBME surgery subject exam– Independent of clerkship order– Course goals and objectives– ACGME core competencies
•Questions / Comments