PROSEDUR

OSCE, MINI-CEX DAN PORTOFOLIO

Susanto Nugroho

Laboratorium Ilmu Kesehatan Anak

Fakultas Kedokteran Universitas Brawijaya

Sir William Osler (1849-1919)

Father of Modern Medicine

“He who studies medicine without

books sails an uncharted sea, but he

who studies medicine without

patients does not go to sea at all”

“No teaching without the patient for

a text and the best teaching is often

that taught by the patient himself”

…....in Clinical Teaching/Training

How should we know if students

competence?How should we know if

students have

good performance?

“CLINICAL ASSESSMENT”

Patient is very IMPORTANT !!!

OBJECTIVE

After this lecture, all participants will be able

to describe and apply these clinical

assessments following in medical

education/training:

• Objective Structured Clinical

Examination (OSCE)

• Mini-Clinical Evaluation Exercise (Mini-

CEX)

• Portfolio

Why should we

assess the medical

students in clinics ?

Clinical

Assessment

• Evidence of competence/performance &

inform progression (Tomorrow’s doctors -

GMC, 2003)

• To drive learning (van der Vleuten, 2000)

• To improve trainee confidence (van der

Vleuten, 2000)

• Public confidence:

- Scepticism of profession to self-

regulate (Smith, 1998)

- Better measures of quality of practice (Scally, 1998)

• To rebuke legal challenges (Tweed and

Why to assess the medical

students ?

What should we

assess the medical

students in clinics ?

Clinical

Assessment

COMPETENCE

~ “able to do”

PERFORMANCE

~ “actually does”

Clinical

Assessment

How should we

assess the medical

students in clinics ?

Clinical

Assessment

• OSCE ~ Objective structure clinical examination

• OSLER ~ Objective structured long case

examination record

• OSATS ~ Objective structures assessment of

technical skills

How to assess the clinical

competency?

...... undertaken ”outside the real” clinical environment

...... have many aspects of realism of workplace

How to assess the clinical

performance?• Mini-CEX ~ Mini-clinical evaluation exercise

• DOPS ~ Direct observation of procedural skill

• CbD ~ Case-based discussion

• Mini-PAT ~ Mini peer assessment tool

...... undertaken ”on the real” patient & workplace

...... have many aspects of realism of workplace

Assessment ~ Miller’s Pyramid (Miller, 1990)

Does

Shows how

Knows how

Knows

Knowledge

Performance

A framework for assessing clinical competence and performance

Mini-CEX, DOPS, CbD,

Mini-PAT, Portfolio

OSCE, short case,

long case

MEQ, EMQ, PMPs,

SAQ, SEQ

MCQ, Essay, Oral

Competence

Does

Shows how

Knows how

Knows

Knowledge

Performance

A framework for assessing clinical competence and performance

OSCE

Competence

Assessment ~ Miller’s Pyramid (Miller, 1990)

UU Praktek Kedokteran Konsil Kedokteran

Indonesia (KKI) SKDI & Standar Pendidikan

Profesi Dokter pada tahun 2006

(Direvisi tahun 2012)

KBK

Kemampuan

Institusi Berbeda

Kualitas

BerbedaSTANDARISASI

Latar Belakang

Standar Pendidikan Dokter

INPUT PROSES OUTPUT OUTCOME

Kualitas

penerimaan

Kualitas

pendidikan &

pembelajaran

Kualitas

lulusanKualitas

profesionalisme

ASESSMENT

OSCE

Affandi, 2008

Objective: semua peserta diuji dengan

ujian yang samaO

SStructured: penilaian di setiap stasion

terstruktur & yang diujikan adalah

ketrampilan klinik tertentu (anamnesis,

PF, prosedur tindakan, dll)

CEClinical Examination: penilaian

terhadap kemampuan ketrampilan klinik

(bukan pengetahuan) & mahasiswa

harus mendemonstrasikan

Pengertian OSCE

Tujuan OSCE

Menilai kompetensi dan ketrampilan klinis

mahasiswa secara objektif dan terstruktur.

Komponen dalam Pelaksanaan

OSCE• Blue print & soal

ujian

• Penguji

• Pasien standar

(PS)

• Pelatih PS

• Peserta/kandidat

• Manekin & peralatan

ujian

• Sarana & prasarana

• Supporting team/staf

Rancangan OSCE

Station 1

Station 2

Station 3

Station 4

Station 5

Station 6

Station 7

Station 8

Station 9

Station 10

Station 11

Station 12

Rancangan OSCE

Heteroanamnesisibu dengan anak

sakit

PemeriksaanAbdomen

PemeriksaanRefleks

Anamnesis Penyakit

Kronis

PemeriksaanGinekologis

Bedah Minor

MelakukanProsedurAseptik

Blue Print

Blue Print OSCE

• Kategori kompetensi: kemampuan

anamnesis, pemeriksaan fisik, penunjang &

interpretasinya, prosedur tindakan, edukasi

& profesionalisme sesuai standar

kompetensi

• Sistem organ/divisi ~ sesuai dengan yang

akan diujikan

• Kasus: untuk memberikan situasi klinik

yang diharapkan

• Distribusi & proporsi pencapaian

kompetensi: perilaku profesional harus

100%

Katagori Kompetensi

Min

ima

l

Sis

tem

Muskulo

skele

tal

Sis

tem

Muskulo

skele

tal

Sis

tem

Muskulo

skele

tal

Sis

tem

Hem

ato

poie

tk

Sis

tem

Hem

ato

poie

tk

Sis

tem

Kulit

&

jaringan

Ikat

Sis

tem

Kulit

&

jaringan

Ikat

Sis

tem

En

do

kri

n&

Meta

bo

lik

Kasus

1. Anamnesis 1 +

2. Pemeriksaan fisik 1 + +

3. Melakukan

tes/prosedur

klinik/interpretasi

data

3 + + +

4. Menentukan

diagnosis atau

diagnosis banding

5 + + + + +

5. Penatalaksanaan:

a. Non

Farmakoterapi

1 +

b. Farmakoterapi 4 + + + +

6. Komunikasi &

Template Stasion OSCE

Standard Setting OSCE

Absolute Methods

• Anggoff (modified)

• Ebel

Compromise Methods

• The Hofstee

Method

• Borderline Group

Method

• Borderline

Regression

Method

Borderline Regression Method

• Metode standar setting yang sering digunakan

pada OSCE

• Penilaian meliputi: “Actual Mark” dan “Global

Rating”

• Actual Mark: deskripsi skor (0 s/d 3) di daftar

tilik (rubrik) harus jelas agar penguji tepat

dalam memberikan skor

• Global Rating: persepsi penguji terhadap

“overall performance” (meminimalisasi

subyektivitas penilaian)

1 = tidak lulus

2 = borderline (minimally competence)

3 = lulus

Penilaian (Skor) yang

DimasukkanTotal nilai :

Actual Mark : ..............

Global Rating :

1. tidak lulus 2. borderline

3. lulus 4. outstanding

Penentuan “Minimum Passing

Level ” (MPL)

10

5

15

20

25

30

1 2 3 4

NO GR AM

1 1 10

2 3 20

3 2 14

4 2 18

5 3 22

6 4 28

7 4 30

8 3 26

9 2 16

10 3 24

Global Rating

Actual Marck

1 4 242 1 123 1 94 2 155 2 206 3 227 2 168 3 179 1 10

10 2 1211 3 1412 4 2213 4 2114 3 1615 1 816 4 2617 3 2018 2 1119 3 1620 1 8

0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

Act

ual

Mar

ck

Global Rating

Actual Marck

Linear (Actual Marck)

14

≥14

Lulus

Kelebihan OSCE

1. Valid

2. Reliabel

3. Setting klinik yang nyata dan menarik

4. Berbagai ketrampilan dengan variasi dapat

diujikan dalam waktu yang relatif singkat

5. Setting standar kompetensi tertentu dapat

ditentukan

6. Obyektif: variasi pasien dan penguji dikurangi

7. Format OSCE bersifat fleksibel

8. Pengamatan langsung pada setiap peserta

9. Terstruktur dan terencana

10.Feasibel

Kekurangan OSCE

1. Kompartementalisasi

2. “High cost”

3. “High human resources”:

- Pasien standar (PS)

- Pelatih PS

- Penguji yang terlatih: perlu pelatihan

4. “High time consuming”: untuk persiapan,

pelaksanaan dan evaluasi

5. Perlu organisasi dan koordinasi yang baik

Does

Shows how

Knows how

Knows

Knowledge

Performance

A framework for assessing clinical competence and performance

MINI-CEX

Competence

Assessment ~ Miller’s Pyramid (Miller, 1990)

Method to assess clinical performance of students in

workplace – “DOES” level (Miller’s pyramid)

Key features – real patients, clinical conditions, work

settings, clinical tasks and constrains

Relies on – multiple encounters, assessors, settings,

occasions and judgements

Involves – clinicians to score, short focused patient-trainee

interaction, and clinician offering ± 5 minutes of

developmental feedback to trainee to improve quality of

clinical skills

Features of Mini-CEX

Can assess the clinical skills – based on standard of

competencies

Can assess the clinical performance – clinical skills,

student – patient interaction (attitudes & behaviour)

Direct observation & immediate feedback – not only

helps student to identify strength & weakness, but can helps

to improve skills

Need few minutes – 10-15 minutes of observation & 5-10

minutes of feedback

Advantages of Mini-CEX

Mini-CEX Forms

Student’s information

Assessment

Feedback & action plan

Assessor ID

Competencies assessed &

descriptors

Step 1. Preparation

Step 2. Observation &

Assessment

Step 3. Verbal & Written

Feedback

The Validity of Mini-CEX

The Reliability of Mini-CEX

The mini-CEX is a reliable tool for performance

assessment, and is acceptable to and well received

by both learners and supervisors.

Nair, et al., 2008

Changes in reliability as a function of

the number of encounters

Norcini & Burch, 2007

Students should be observed at

least four times by different

assessors to get a reliable

assessment of competence.

Mini-CEX is a good example of workplace-

based assessment method that fulfils three

requirements for facilitating learning:

1. The course content, expected competencies &

assessment practices are aligned

2. Feedback is provided either during or

immediately after the assessment

3. The assessment is used to direct learning

towards desired outcomes

The Reliability of Mini-CEX

Norcini & Burch, 2007

The Educational Impacts of

Mini-CEX

Liao, et al., 2013

• Knowledge acquisition and enhancement of

giving feedback when the faculty members

used the tool.

• Providing effective feedback should be

conducted to increase the impact of the mini-

CEX as a formative assessment.

Mini-CEX ~ a valid, reliable & feasible

Validity – it is important instrument for the direct observation

of trainee’s clinical performance

Reliability

• more encounters & assessors - more reliable

• expected competency, feedback & used to direct learning

Educational impacts

• can correct the weakness & mature professionally

• can monitor progress & identify educational needs

• reassures student’s satisfactory performance

• increases the interaction student & teacher

Conclusions

Assessment ~ Miller’s Pyramid

Does

Shows how

Knows how

Knows

Knowledge

Performance

A framework for assessing clinical competence and performance

Portfolio

Competence

Portfolio

EXAMINATIONtoward broader methods of

assessment

• To encourage closer links between assessment &

learning using assessment & feedbackby

learning improvement

• To enhance the assessment of areas that are

difficult to assess by traditional methods: attitudes,

personal attributes, reflection & professionalismHaldane, 2014

A purposeful collection of work(Stecher, 2001)

A collection of papers and other

forms of evidence that learning has

taken place

(Davis et al.,

2001)A collection of student work that exhibits the

student’s efforts, progress and achievements in

one or more areas(Gisselle & Martin-Kniep, 2000)

Definition of Portfolio

1. Portfolios’ contribution to assessment

2. Focus on personal attributes

3. Enhances interactions between students &

teachers

4. Stimulates the use of reflective strategies

5. Expands understanding of professional

competence

Why Use Portfolio ?

Models of Portfolio

Portfolio Description Advantages Disadvantages

Shopping

trolley

Contains anything which

has been produced or

used during the learning

proces

Very inclusive Difficult to assess.

No analysis of

contents

Toast rack “Toast” for each period of

learning

Corresponds with

the curriculum

Can be marked

Includes

reflection

Each item is

discrete & does not

provide overall

assessment of

learning

No overall

reflection

Cake mix Integration of the parts

“Mixing” is reflection on

the

analytical components

Global

assessment

Individual

components may

not be clear

Spinal

column

Series of competency

statements are the

Each

competency has

• Portfolio model should be easily aligned with

the curriculum ~ “toast rack” or “spinal

column”

• Portfolio should provide opportunity to

demonstrate learning in many different ways

& should be a holistic record of learning

• The appropriate model depends on its

purpose will be used in assessment

process

• Structure should be decided based on the

format the evidence is required

Portfolio Contents, Alignment &

Assessment

Haldane, 2014

Five Steps in Portfolio

Assessment Process

Davis & Ponnamperuma, 2006

Step

I.

EvidenceDocumentation of experience by the learner

Step

II.

Reflection Commentary by the learner on experiences

and learning that has resulted

Step

III.

EvaluationStudying the evidence by examiners

Step

IV.

Defending the evidenceA dialogue between learner and examiner

Step

V.

Assessment decisionFormative and summative

Learning Outcome

A three-circle classification model of

learning outcome

A. Inner circle ~ “what the doctor is able to

do”

(1) Clinical skills

(2) Practical procedures

(3) Investigating a patient

(4) Patient management

(5) Health promotion & disease prevention

(6) Communication skills

(7) Information handling & retrieval Harden et al, 1999

B. Middle circle ~ “how the doctor

approach the task”

(8) Understanding of basic, clinical & social

sciences

(9) Appropriate attitudes, ethical understanding

& legal responsibilities

(10) Appropriate decision making, clinical

reasoning & judgementC. Outer circle ~ “doctor as a professional”

(11) The role of the doctor within the health

service

(12) Attitude for personal development

Harden et al, 1999

1. Defining the purpose

2. Determining competences to be assessed

3. Selection of portfolio material

4. Developing a marking system

5. Selection and training of examiners

6. Planning the examination process

7. Student orientation

8. Developing guidelines for decisions

9. Establishing reliability and validity evidence

10. Designing evaluation procedures

Steps in Developing Portfolio

AMEE Medical Education Guide, 2001

1. Defining the purpose

The Purposes of Portfolio

1. For the trainee:

• to record the training experience, education

supervision, professional development

plans, workshops attended, reflective entries

and assessment reports

• to identify deficiencies

• to plan for necessary remediation

2. For supervisors:

• to assess overall training & work with the

trainee

• to correct deficiencies

Joint Committee on Specialist Training, 2011

Expected achievement in the 6 core

competencies:

1. Patient care

2. Medical knowledge

3. Practice based learning & improvement

4. Interpersonal & communication skills

5. Professionalism

6. System based practice

Expected Achievement of

Competencies

2. Determining competences to be

assessed

Joint Committee on Specialist Training, 2011

Expected frequency of assessments

Other areas of curriculum & assessment

3. Selection of portfolio material

To assess: patient management skills, e.g.

patient education

• Written outline of a patient education

programme in community

• A video of individual patient education

session with patient discussing one topic

To assess: attitudes, ethical & legal

understanding & responsibility

• The elective report ~ the student shows

ethical understanding of issues inherent in the

elective

• Case discussion on ethics ~ the students will

provide evidence of ethical judgement & moral

reasoning & could be questioned about the

Record the cases

Reflective entries

DOPS for

Medical

Training

Record of Procedures

Name of procedure: Intraosseous needle insertion

Name of procedure: Percutaneous central line insertion

Practice Based Learning Assessment

Teaching of Communication Skill Evaluation

4. Developing a marking system

• Students’ work is judged by criteria which will

specify the level of their academic

achievements and will determine their progress

towards state standards.

• The portfolio material should direct the

examiner to consider student progress

according to the outcome specification & should

enable the examiner to identify strengths &

weaknesses.

• If one competence is assessed, highly specific

criteria could be employed; if multiple

competences, general standards should be

developed.

5. Selection & training of examiners

• According to its purpose of portfolio the

appropriate examiners, include: staffs,

teachers in the basic sciences & laboratory-

based diciplines, clinicians, faculty who

indicate special interest in education &

student development.

• Another selection issue: seniority of

examiners

• A key point of the success of programme:

the training of faculty examiners &

maintaining them should be preserved &

reinforced

6. Planning

the

examination

process

7. Student orientation

• Students must be informed at the beginning

of the course about the portfolio

examination, the guidelines & criteria for

judging performance clearly.

• Students can use their on-going work for

selection of material (if they demonstrate

good progress in their achievements & have

confidence in ability to pass the portfolio)

• The more information given to students the

more positive they become towards the

portfolio.

8. Developing guidelines for

decisionsFlow diagram for the decision-making process

9. Establishing reliability & validity

evidence• It is important to determine what will

constitute good reliable evidence & plan the

examination, e.g two independent

examiners, one examiners, independent

rating, consensus or both, minimum desired

reliability or generalisability co-efficient.

• Define desired correlations or absolute inter-

rater agreement and set the minimum

standards for tolerance of misclassification

error.

• Triangulation of portfolio results with other

forms of assessment will increase the

validity of the decision.

10. Designing evaluation

procedures Feedback: “student” & “examiners” opinion on the

portfolio’s strength & weakness changes &

improvements

• Questionnaires

• Focus group discussion

• Individual interview

• Request for written comments

Portfolio’s Assessment Features

1. Formative & summative

2. Qualitative & quantitative

3. Personalised

4. Standarised

5. Authentic

Thank You for Your Attention