Standard setting Determining the pass mark
The Old Way
……..I think that ..I think that the cut score in the cut score in this is exam is this is exam is probably about probably about here…..here…..
there is a there is a natural lacuna I natural lacuna I can see can see dividing the dividing the bottom from bottom from the restthe rest
Assessment
• Blueprinting Assessments
• Pass or Fail? Methods of Standard Setting
- Angoff - Borderline- Hofstee
………………………………………………………….
• Assessment of Professional Behaviours
Vision StatementThe University of Sheffield strives to produce excellent medical graduates.
The medical curriculum will be outcome focussed where the aim is to produce graduates who are able to fulfil their role as junior doctors in the NHS and who also possess the generic skills expected of students attending a research-led university.
The course will feature increased opportunities to see patients in the community; a high degree of integration; an emphasis on facilitating student learning; and an increase in student choice.
The course will be organised on a body system basis with a progressive emphasis on learning around undifferentiated patient problems.
The instructional approach will consist of a spine of problem, case and patient-based integrated learning activities complemented by a range of other teaching and learning activities. There will be an increase in systematic teaching of some components to ensure competence in key areas.
Students will be expected to become progressively more self-directed, aided by increasing reliance on IT-based and distance learning materials and activities.
Assessment, both formative and summative, will be closely matched to defined outcomes.
The curriculum will be managed centrally by a multidisciplinary team, including those with a stake in the outcome of medical education. The Department of Medical Education, the Administration and an IT-based curriculum management system will provide support.
A monitoring system will be established to evaluate the implementation of the curriculum and to support a process of continuous improvement.
Excellent graduates
Outcome focussed
Integrated programmeStudent choiceUndifferentiated problems
Integrated Learning Activities
Able learners
Assessment matches outcomes
Curriculum management
Quality enhancement
Align
Objectives
Teaching & learning
Assessment
Summary
• Assessments should match
local/national outcomes for Validity• Multiple observations
required for Reliability• Electronic data collection
for optimal Feasibility• Good information to students
and staff to enhance Acceptance
Pass
Competent
Fail
Incompetent
measurementerror
Pass
Fail
Standard setting
Knowledge Skills
Angoff
Hofstee
Borderline method
Knowledge Skills
Standard setting
Angoff (modified)What is the mark that a student who just passes would score for each item?
DISCUSS+/-MODIFY
Mean mark for all items is the pass mark
Written and clinical examinations
The pass mark for this exam using a modified Angoff is 63.7%
Item Number Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 61 70 75 80 70 70 802 50 70 70 60 50 603 66 80 70 75 70 80
40 60 70 70 60 60 70
overall estimate 57.7 65.7 69.7 64.4 60.2 64.6
mean of all judges 63.7
Angoff
Angoff standard setting
• Determine the pass mark for the ‘just-competent’ student and write it in the box
• Do not assume that the pass mark is 50% or 60% or constant
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
Examiners score the students’s performance at the station e.g 17/20
Examiners judge the overall performance
clear pass / borderline / clear fail
Mark sheets rated borderline identified and the scores of borderline students averaged
Process repeated for each station
Calculate the median borderline score for all stations
Item I Clinical Problem
Type of station/question
Max marks available
Median Borderline score
Station1 Cranial nerves Examination 40 28.75
Station2b Patient education
Communication 20 13.38
Station3 Breathlessness History 20 12.00 Station4 chest Examination 20 11.00
Overall borderline score
510 297.89
Overall borderline score %
100
58.41
Borderline Method
1 2 3
Pass/Fail
Station 15 (static) Oral Lesions
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00
Score
0
10
20
30
40
50
Co
un
t
Borderlinegroup
pass
fail
Borderline score 58.41
Borderlinegroup
pass
fail
Borderline score 58.41
How wide should this band be?
Borderlinegroup
pass
fail
Borderline score 58.41
How wide should this band be?
+/- 1 - standard deviation- standard error- or what?
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
R = Variance Variance x Error
Borderlinegroup
Pass
Fail
Borderline score 58.41
Pass Score60.93
} - 1 SEM
Fail score 55.88
} + 1 SEM
Station/MEQ ID Competence System Problem Max marksavailable
Median Borderline score
Station1 Phys Exam CNS Abnormal Gait 40 23.50
Station2 (static) Data (x-ray) RS Wheeze 10 6.00
Station3 Phys Exam GI Abdominal Mass 20 11.75
Station4 History RS Haemoptysis 20 12.00
Station6 Pat ed/comms Skin/Misc Rash 20 13.00
Station7 History Endo Weight loss 20 13.50
Station8 (static) Pract skills Onc Pain 10 5.00
Station9 History Eyes/ENT Visual Disturbance 20 12.00
Station10 Phys Exam MSS/CNS Laceration 20 11.00
Station11 Pract Skills CVS Collapse 20 10.50
Station12 Phys exam Onc Breast Lump 20 13.00
Station14b Pat ed/Comms Onc Dying Patient 20 12.50
Station15 (static) Data Skin/Misc Itch 10 6.00
Written1 Prob solving/Dx/Manage CVS Claudication 10 6.00
Written2 Prob solving/Dx/Manage CVS/Endo Palpitations 10 6.00
Written3 Prob solving/Dx/Manage CVS Heart failure 10 6.00
Written4 Prob solving/Dx/Manage Resp Haemoptysis 10 7.00
Written5 Prob solving/Dx/Manage Mental Mania 10 5.00
Written6 Prob solving/Dx/Manage ENT Hearing Difficulty 10 4.00
Written7 Prob solving/Dx/Manage CNS Coma 10 7.00
Written8 Prob solving/Dx/Manage MSS Joint Pain 10 5.00
Written9 Prob solving/Dx/Manage MSS Fracture 10 5.00
Written10 Prob solving/Dx/Manage GI Diarrhoea 10 7.00
Written11 Prob solving/Dx/Manage GI Haematemesis 10 6.00
Written12 Prob solving/Dx/Manage Mental Overdose 10 5.00
Written13 Prob solving/Dx/Manage Endo Weight loss 10 7.00
Written14 Prob solving/Dx/Manage Onc Breast Lump 10 6.00
Written15 Prob solving/Dx/Manage Haem Bleeding 10 5.00
Written16 Prob solving/Dx/Manage Onc Testicular Mass 10 4.00
Written17 Prob solving/Dx/Manage Eyes Red Eye 10 5.00
Written18 Prob solving/Dx/Manage Endo Hairy 10 5.00
Written19 Prob solving/Dx/Manage CNS Weakness 10 5.50
Written20 Prob solving/Dx/Manage GI Vomiting 10 7.00
Static1 Investig /Interpretation CNS Headache 10 6.00
Static2 Investig /Interpretation GI Juandice 10 5.00
Static3 Investig /Interpretation Resp SOB 10 6.00
Static4 Investig /Interpretation Endo Dehydration 10 5.00
Static5 Investig /Interpretation CNS Consciuoness 10 6.00
Static6 Investig /Interpretation ENT Abnormal Hearing 10 6.00
Overall borderline score
510
297.25
Overall borderline score %
100
58.28
Written14 Prob solving/Dx/Manage Onc Breast Lump 10 6.00
Written15 Prob solving/Dx/Manage Haem Bleeding 10 5.00
Written16 Prob solving/Dx/Manage Onc Testicular Mass 10 4.00
Written17 Prob solving/Dx/Manage Eyes Red Eye 10 5.00
Written18 Prob solving/Dx/Manage Endo Hairy 10 5.00
Written19 Prob solving/Dx/Manage CNS Weakness 10 5.50
Written20 Prob solving/Dx/Manage GI Vomiting 10 7.00
Static1 Investig /Interpretation CNS Headache 10 6.00
Static2 Investig /Interpretation GI Juandice 10 5.00
Static3 Investig /Interpretation Resp SOB 10 6.00
Static4 Investig /Interpretation Endo Dehydration 10 5.00
Static5 Investig /Interpretation CNS Consciuoness 10 6.00
Static6 Investig /Interpretation ENT Abnormal Hearing 10 6.00
Overall borderline score
510
297.25
Overall borderline score %
100
58.28
Number of students
Mean Score (%)
Overall Borderline Score (%)
Standard deviation
Standard Error of Measurement
Pass mark (%)
Reliability
04 195 73.25
58.28 5.50 2.40 60.68
0.81
03 215 74.4 58.41 5.77 2.52 60.93
0.81
02 214 73. 0 59. 81 4.95 2.98 62.79
0.64
01 195 71.85
58.90 5.86 3.21 62.11
0.70
Internal Reliability of the ExamCronbach’s Alpha
Cronbach’s alpha
Shows whether randomly split halves of the exam by item vary together
Split halfCorrelations
(SH)
Item / Stations
Population
(students)
Borderline
Grading System
Borderline
Fail
Pass
Good Pass
Top Mark
Borderline Mark
Distinction
Pass Mark
Pass
Fail
Reducing measurement errorReducing measurement error
1. Increase reliability of exam
2. Compromise with feasibility/cost
* Content specificity
* More items in the exam
* Blueprinting
* Quality assurance of item writing
* Training examiners/standardised patients
* Feedback from exam performance
Hofstee procedureHofstee procedure
Used for written examinations
Judges review a copy of the whole exam
Judges indicate:
minimum % failure rate
maximum % failure rate
minimum acceptable cut percentage
maximum acceptable cut percentage
Administer test, plot curve and read off standard
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s