Post on 21-Dec-2015
transcript
Standard setting Determining the pass mark
- OSCEs
The Old Way (1)
……..I think that ..I think that the pass mark the pass mark in this is exam in this is exam is probably is probably about here…..about here…..
there is a there is a natural break I natural break I can see can see dividing the dividing the bottom from bottom from the restthe rest
The Old Way (2)
……..The pass ..The pass mark is 60%.mark is 60%.
Assessments
Assessments
Match OutcomeObjectives
Assessments
Match OutcomeObjectives
Integrated
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
From approvedlist
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
From approvedlist
Studentinformation
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
From approvedlist
Studentinformation
External Examiners
Standard setting
Knowledge Skills
Angoff
Hofstee
Borderline method
Knowledge Skills
Standard setting
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
Examiners score the students’s performance at the station e.g 17/20
Examiners judge the overall performance
clear pass / borderline / clear fail
Mark sheets rated borderline identified and the scores of borderline students averaged
Process repeated for each station
Calculate the median borderline score for all stations
Item I Clinical Problem
Type of station/question
Max marks available
Median Borderline score
Station1 Cranial nerves Examination 40 28.75
Station2b Patient education
Communication 20 13.38
Station3 Breathlessness History 20 12.00 Station4 chest Examination 20 11.00
Overall borderline score
510 297.89
Overall borderline score %
100
58.41
Borderline Method
1 2 3
Pass/Fail
Station 15 (static) Oral Lesions
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00
Score
0
10
20
30
40
50
Co
un
t
Borderlinegroup
pass
fail
Median Borderline score 58.41
Borderlinegroup
pass
fail
Median Borderline score 58.41 How wide should
this band be?
Borderlinegroup
pass
fail
Median Borderline score 58.41 How wide should
this band be?
+/- 1 - standard deviation- standard error- or what?
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
R = Variance Variance x Error
Borderlinegroup
Pass
Fail
Median Borderline score 58.41
Pass Score60.93
} - 1 SEM
Fail score 55.88
} + 1 SEM
Number of students
Mean Score (%)
Overall Borderline Score (%)
Standard deviation
Standard Error of Measurement
Pass mark (%)
Reliability
04 195 73.25
58.28 5.50 2.40 60.68
0.81
03 215 74.4 58.41 5.77 2.52 60.93
0.81
02 214 73. 0 59. 81 4.95 2.98 62.79
0.64
01 195 71.85
58.90 5.86 3.21 62.11
0.70
Internal Reliability of the ExamCronbach’s Alpha
Cronbach’s alpha
Shows whether randomly split halves of the exam by item vary together
Split halfCorrelations
(SH)
Item / Stations
Population
(students)
Borderline
Grading System
Borderline
Fail
Pass
Good Pass
Top Mark
Borderline Mark
Distinction
Pass Mark
Pass
Fail
Reducing measurement errorReducing measurement error
1. Increase reliability of exam
2. Compromise with feasibility/cost
* Content specificity
* More items in the exam
* Blueprinting
* Quality assurance of item writing
* Training examiners/standardised patients
* Feedback from exam performance