Standard setting Determining the pass mark

The Old Way

……..I think that ..I think that the cut score in the cut score in this is exam is this is exam is probably about probably about here…..here…..

there is a there is a natural lacuna I natural lacuna I can see can see dividing the dividing the bottom from bottom from the restthe rest

Assessment

• Blueprinting Assessments

• Pass or Fail? Methods of Standard Setting

- Angoff - Borderline- Hofstee

………………………………………………………….

• Assessment of Professional Behaviours

Vision StatementThe University of Sheffield strives to produce excellent medical graduates.

The medical curriculum will be outcome focussed where the aim is to produce graduates who are able to fulfil their role as junior doctors in the NHS and who also possess the generic skills expected of students attending a research-led university.

The course will feature increased opportunities to see patients in the community; a high degree of integration; an emphasis on facilitating student learning; and an increase in student choice.

The course will be organised on a body system basis with a progressive emphasis on learning around undifferentiated patient problems.

The instructional approach will consist of a spine of problem, case and patient-based integrated learning activities complemented by a range of other teaching and learning activities. There will be an increase in systematic teaching of some components to ensure competence in key areas.

Students will be expected to become progressively more self-directed, aided by increasing reliance on IT-based and distance learning materials and activities.

Assessment, both formative and summative, will be closely matched to defined outcomes.

The curriculum will be managed centrally by a multidisciplinary team, including those with a stake in the outcome of medical education. The Department of Medical Education, the Administration and an IT-based curriculum management system will provide support.

A monitoring system will be established to evaluate the implementation of the curriculum and to support a process of continuous improvement.

Excellent graduates

Outcome focussed

Integrated programmeStudent choiceUndifferentiated problems

Integrated Learning Activities

Able learners

Assessment matches outcomes

Curriculum management

Quality enhancement

Align

Objectives

Teaching & learning

Assessment

Summary

• Assessments should match

local/national outcomes for Validity• Multiple observations

required for Reliability• Electronic data collection

for optimal Feasibility• Good information to students

and staff to enhance Acceptance

Pass

Competent

Fail

Incompetent

measurementerror

Pass

Fail

Standard setting

Knowledge Skills

Angoff

Hofstee

Borderline method

Knowledge Skills

Standard setting

Angoff (modified)What is the mark that a student who just passes would score for each item?

DISCUSS+/-MODIFY

Mean mark for all items is the pass mark

Written and clinical examinations

The pass mark for this exam using a modified Angoff is 63.7%

Item Number Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 61 70 75 80 70 70 802 50 70 70 60 50 603 66 80 70 75 70 80

40 60 70 70 60 60 70

overall estimate 57.7 65.7 69.7 64.4 60.2 64.6

mean of all judges 63.7

Angoff

Angoff standard setting

• Determine the pass mark for the ‘just-competent’ student and write it in the box

• Do not assume that the pass mark is 50% or 60% or constant

Borderline procedureBorderline procedure

Used for clinical (OSCE) examinations

Borderline procedureBorderline procedure

Used for clinical (OSCE) examinations

Examiners score the students’s performance at the station e.g 17/20

Examiners judge the overall performance

clear pass / borderline / clear fail

Mark sheets rated borderline identified and the scores of borderline students averaged

Process repeated for each station

Calculate the median borderline score for all stations

Item I Clinical Problem

Type of station/question

Max marks available

Median Borderline score

Station1 Cranial nerves Examination 40 28.75

Station2b Patient education

Communication 20 13.38

Station3 Breathlessness History 20 12.00 Station4 chest Examination 20 11.00

Overall borderline score

510 297.89

Overall borderline score %

100

58.41

Borderline Method

1 2 3

Pass/Fail

Station 15 (static) Oral Lesions

0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00

Score

0

10

20

30

40

50

Co

un

t

Borderlinegroup

pass

fail

Borderline score 58.41

Borderlinegroup

pass

fail

Borderline score 58.41

How wide should this band be?

Borderlinegroup

pass

fail

Borderline score 58.41

How wide should this band be?

+/- 1 - standard deviation- standard error- or what?

The Standard Error of MeasurementThe Standard Error of Measurement

• depends on the reliability of the test (R)

• depends on the standard deviation of the test (SD)

• SEM = SD 1 - R• acts as a confidence interval in high stakes situations

The Standard Error of MeasurementThe Standard Error of Measurement

• depends on the reliability of the test (R)

• depends on the standard deviation of the test (SD)

• SEM = SD 1 - R• acts as a confidence interval in high stakes situations

R = Variance Variance x Error

Borderlinegroup

Pass

Fail

Borderline score 58.41

Pass Score60.93

} - 1 SEM

Fail score 55.88

} + 1 SEM

Station/MEQ ID Competence System Problem Max marksavailable

Median Borderline score

Station1 Phys Exam CNS Abnormal Gait 40 23.50

Station2 (static) Data (x-ray) RS Wheeze 10 6.00

Station3 Phys Exam GI Abdominal Mass 20 11.75

Station4 History RS Haemoptysis 20 12.00

Station6 Pat ed/comms Skin/Misc Rash 20 13.00

Station7 History Endo Weight loss 20 13.50

Station8 (static) Pract skills Onc Pain 10 5.00

Station9 History Eyes/ENT Visual Disturbance 20 12.00

Station10 Phys Exam MSS/CNS Laceration 20 11.00

Station11 Pract Skills CVS Collapse 20 10.50

Station12 Phys exam Onc Breast Lump 20 13.00

Station14b Pat ed/Comms Onc Dying Patient 20 12.50

Station15 (static) Data Skin/Misc Itch 10 6.00

Written1 Prob solving/Dx/Manage CVS Claudication 10 6.00

Written2 Prob solving/Dx/Manage CVS/Endo Palpitations 10 6.00

Written3 Prob solving/Dx/Manage CVS Heart failure 10 6.00

Written4 Prob solving/Dx/Manage Resp Haemoptysis 10 7.00

Written5 Prob solving/Dx/Manage Mental Mania 10 5.00

Written6 Prob solving/Dx/Manage ENT Hearing Difficulty 10 4.00

Written7 Prob solving/Dx/Manage CNS Coma 10 7.00

Written8 Prob solving/Dx/Manage MSS Joint Pain 10 5.00

Written9 Prob solving/Dx/Manage MSS Fracture 10 5.00

Written10 Prob solving/Dx/Manage GI Diarrhoea 10 7.00

Written11 Prob solving/Dx/Manage GI Haematemesis 10 6.00

Written12 Prob solving/Dx/Manage Mental Overdose 10 5.00

Written13 Prob solving/Dx/Manage Endo Weight loss 10 7.00

Written14 Prob solving/Dx/Manage Onc Breast Lump 10 6.00

Written15 Prob solving/Dx/Manage Haem Bleeding 10 5.00

Written16 Prob solving/Dx/Manage Onc Testicular Mass 10 4.00

Written17 Prob solving/Dx/Manage Eyes Red Eye 10 5.00

Written18 Prob solving/Dx/Manage Endo Hairy 10 5.00

Written19 Prob solving/Dx/Manage CNS Weakness 10 5.50

Written20 Prob solving/Dx/Manage GI Vomiting 10 7.00

Static1 Investig /Interpretation CNS Headache 10 6.00

Static2 Investig /Interpretation GI Juandice 10 5.00

Static3 Investig /Interpretation Resp SOB 10 6.00

Static4 Investig /Interpretation Endo Dehydration 10 5.00

Static5 Investig /Interpretation CNS Consciuoness 10 6.00

Static6 Investig /Interpretation ENT Abnormal Hearing 10 6.00

Overall borderline score

510

297.25

Overall borderline score %

100

58.28

Written14 Prob solving/Dx/Manage Onc Breast Lump 10 6.00

Written15 Prob solving/Dx/Manage Haem Bleeding 10 5.00

Written16 Prob solving/Dx/Manage Onc Testicular Mass 10 4.00

Written17 Prob solving/Dx/Manage Eyes Red Eye 10 5.00

Written18 Prob solving/Dx/Manage Endo Hairy 10 5.00

Written19 Prob solving/Dx/Manage CNS Weakness 10 5.50

Written20 Prob solving/Dx/Manage GI Vomiting 10 7.00

Static1 Investig /Interpretation CNS Headache 10 6.00

Static2 Investig /Interpretation GI Juandice 10 5.00

Static3 Investig /Interpretation Resp SOB 10 6.00

Static4 Investig /Interpretation Endo Dehydration 10 5.00

Static5 Investig /Interpretation CNS Consciuoness 10 6.00

Static6 Investig /Interpretation ENT Abnormal Hearing 10 6.00

Overall borderline score

510

297.25

Overall borderline score %

100

58.28

Number of students

Mean Score (%)

Overall Borderline Score (%)

Standard deviation

Standard Error of Measurement

Pass mark (%)

Reliability

04 195 73.25

58.28 5.50 2.40 60.68

0.81

03 215 74.4 58.41 5.77 2.52 60.93

0.81

02 214 73. 0 59. 81 4.95 2.98 62.79

0.64

01 195 71.85

58.90 5.86 3.21 62.11

0.70

Internal Reliability of the ExamCronbach’s Alpha

Cronbach’s alpha

Shows whether randomly split halves of the exam by item vary together

Split halfCorrelations

(SH)

Item / Stations

Population

(students)

Borderline

Grading System

Borderline

Fail

Pass

Good Pass

Top Mark

Borderline Mark

Distinction

Pass Mark

Pass

Fail

Reducing measurement errorReducing measurement error

1. Increase reliability of exam

2. Compromise with feasibility/cost

* Content specificity

* More items in the exam

* Blueprinting

* Quality assurance of item writing

* Training examiners/standardised patients

* Feedback from exam performance

Hofstee procedureHofstee procedure

Used for written examinations

Judges review a copy of the whole exam

Judges indicate:

minimum % failure rate

maximum % failure rate

minimum acceptable cut percentage

maximum acceptable cut percentage

Administer test, plot curve and read off standard

0102030405060708090

100

50 60 70 80

Student score %

% c

andid

ate

s

0102030405060708090

100

50 60 70 80

Student score %

% c

andid

ate

s

0102030405060708090

100

50 60 70 80

Student score %

% c

andid

ate

s

0102030405060708090

100

50 60 70 80

Student score %

% c

andid

ate

s

0102030405060708090

100

50 60 70 80

Student score %

% c

andid

ate

s

0102030405060708090

100

50 60 70 80

Student score %

% c

andid

ate

s