Kevin Fox

Lead Task Force Certification at ESC

ESC Education Committee

How to Evaluate Clinical Performance

October 2017

Introduction

• Why evaluate clinical performance

• Performance v Competence

• Knowledge - exams

• Skills and Professionalism – multiple methods

Embedding the importance of going beyond knowledge to train well performing doctors

Testing knowledge - exams still best

An Effective Exam Process – a 1 year cycle - The EEGC (a UEMS / ESC collaboration)

• Write Questions - SBA MCQs - group review

• Select Questions

• Set Standard – score each item individually, then as a group (2d)

• Final Review

• Candidates take test

• Analysis of test results

• Agree a pass mark – different techniques (Angoff v Hofstee)

• Distribution of exam results

Analysis of pass mark – Hofstee method

Comments

The validity of any exam is influenced by

• Candidates perception of the exam

• Candidates level of training

• Wider stakeholder perceptions eg National and Trainer

• Familiarity with exam format (trainers and candidates)

• The challenge

• Contextualising for individual countries (eg NICE in the UK)

Skills and Professionalism

Methods to assess performance

Skills and Professionalism

• The development of DOPS

Skills and Professionalism

• Introduced (in UK) 2005

• There is scanty psychometric data on DOPS perhaps due to the fact that direct

observation is carried out informally. However intrinsically, in terms of

“competency level” it is seen as a high quality instrument as it tests at the “does”

level. Authors have commented on the lack of studies accessing the validity and

reliability of DOPS, despite it being fairly widely used to assess competency of

surgical trainees

DOPS

• Train the trainers

• Standardise the assessment framework

• Use lots of items

Increasing reliability

• The OSCE can be a reasonably reliable, valid and objective method of

assessment , but its main drawback is that it is resource intensive

OSCE assessment - assesses competence

Increasing reliability

Train the assessors

Increasing the number of stations

Standardise assessment tool

OSCE assessment

Simulation – for individuals and teams

Recommendation: The BCS, SAC and Deaneries should work together to

develop simulator based assessment of practical and professional

skills for cardiology trainees. Initially this will probably be for

formative assessment but could progress to summative assessment if

robust data on reliability, validity and utility can be provided

Relationship Between Peer Assessment and Later PDs’ Reports

• There were 43 students from the class of 2004 for whom we received

internship directors’ ratings (response rate = 44%). Overall ratings were

significantly correlated with both second- and third-year WH scores

(r = .32 [P = .015] and r = .43 [P = .004], respectively). Interpersonal

attributes scores were not correlated with later PDs’ ratings (r = .15 and

−.09, respectively).

360 degree assessment

Skills tracking (longitudinal performance)

– use of portfolios /logbooks

General Logbook Process – EACVI TTE

Grader can mark the cases and can chose to: Send the logbook to back office agent Fail Pass

Process re-started

Logbook review by graders

Team Performance – the wider context

Conclusion

Assessing performance is vital

But accurate reliable, reproducible assessment is extremely challenging

Exams remain effective for knowledge

A variety of reasonably validated techniques exist for (competence) and performance

These methods themselves require quality assessment

Thank you

Standard Setting (2-day) Meeting

• Scored each item difficulty in advance

• Discussed each item

• Rescored each item difficulty at meeting

• This data used to derive the boundaries for the acceptable pass mark

Set the pass mark – Hofstee method

54.1% 66.4%

Graders activity and performance

Appeals against assessments

• Appeals and complaints will happen

• They are difficult stressful and time consuming and potentially expensive

Examples of challenges

I always wear my noise reduction earphones for exams but they wouldn’t let me

How clear are the rules?

I spoke to a woman at the Heart House and she said that it wouldn’t matter if my log book was a few weeks late

Is every contact documented?

EEGC